16. Cheat sheet

Everything worth having to hand, in one place.

The five facts

1. AIRFLOW SCHEDULES WORK; IT SHOULD NOT DO THE WORK.
   Tasks tell other systems what to do.

2. DAG files are parsed every ~30s, on the processor
   AND on every worker. NO I/O AT MODULE LEVEL.

3. A run covers an INTERVAL and starts after it ENDS.
   `ds` is the interval START, not today.

4. `scheduled` = waiting for Airflow (a limit is full).
   `queued`    = waiting for a worker.

5. TASKS MUST BE IDEMPOTENT. Retries, zombie reruns,
   clears and backfills all guarantee re-execution.

A production DAG

from airflow.sdk import dag, task
import pendulum

@dag(
    schedule="0 2 * * *",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,                       # ← ALWAYS
    max_active_runs=1,
    tags=["sales", "daily"],
    default_args={
        "retries": 2,
        "retry_delay": pendulum.duration(minutes=5),
        "retry_exponential_backoff": True,
        "execution_timeout": pendulum.duration(hours=2),   # ← ALWAYS
        "on_failure_callback": alert_slack,
    },
)
def daily_sales():

    @task
    def extract(**context) -> str:
        path = f"s3://bucket/staging/{context['ds']}/orders.parquet"
        write(path, query(context["data_interval_start"],
                          context["data_interval_end"]))
        return path                       # ← a POINTER, not data

    @task
    def load(path: str):
        warehouse.execute("DELETE FROM t WHERE day = %s", ...)   # idempotent
        warehouse.copy_from(path)

    load(extract())

daily_sales()

Template variables

Variable	Value
`{{ ds }}`	interval start date, `2026-08-29`
`{{ ds_nodash }}`	`20260829`
`{{ data_interval_start }}`	interval start — use this
`{{ data_interval_end }}`	interval end — use this
`{{ logical_date }}`	the run’s logical timestamp
`{{ run_id }}`	unique run identifier
`{{ ti }}`	task instance
`{{ params.x }}`	a DAG param
`{{ var.value.x }}`	a Variable — run time, safe
`{{ conn.my_conn.host }}`	a connection field
`{{ macros.ds_add(ds, -7) }}`	date arithmetic

-- what nearly every Airflow query should look like
WHERE ts >= '{{ data_interval_start }}' AND ts < '{{ data_interval_end }}'

Task states

State	Meaning
`scheduled`	deps met — waiting for Airflow (a limit)
`queued`	sent to executor — waiting for a worker
`running`	executing
`success` / `failed`	terminal
`up_for_retry`	failed, waiting out `retry_delay`
`up_for_reschedule`	a `reschedule`-mode sensor sleeping
`deferred`	with the triggerer — holds nothing
`skipped`	branch or trigger rule excluded it
`upstream_failed`	not this task’s problem

Trigger rules

Rule	Runs when
`all_success`	default
`all_done`	upstream finished, whatever happened — cleanup
`none_failed_min_one_success`	the join-after-branch fix
`one_success`	first one wins, does not wait
`none_failed`	skipped is acceptable
`always`	unconditionally

Concurrency (most restrictive wins)

   parallelism                 whole installation      (32)
     max_active_tasks_per_dag  one DAG                 (16)
       pool slots              a named budget
         max_active_tis_per_dag  one task, all runs
   max_active_runs_per_dag     concurrent runs         (16)
   default_pool                everything unassigned   (128)
   worker_concurrency          per Celery worker       (16)

CLI

airflow dags list
airflow dags list-import-errors          # ← FIRST COMMAND, ALWAYS
airflow dags report                      # parse time per file
airflow dags next-execution my_dag
airflow dags unpause my_dag
airflow dags test my_dag 2026-08-30      # whole DAG, no DB state
airflow tasks test my_dag extract 2026-08-30   # one task, no DB state
airflow tasks states-for-dag-run my_dag <run_id>

airflow tasks clear my_dag --start-date 2026-08-01 \
  --end-date 2026-08-30 --task-regex "transform.*" --downstream

airflow backfill create --dag-id my_dag \
  --from-date 2026-08-01 --to-date 2026-08-30

airflow pools list
airflow pools set warehouse 5 "concurrent queries"
airflow connections add / list / delete
airflow variables set / get / export

airflow db clean --clean-before-timestamp 2026-01-01 --dry-run
airflow db clean --clean-before-timestamp 2026-01-01 --yes
airflow db export-archived --output-path /backups --drop-archives
airflow db migrate                        # BACK UP FIRST
airflow rotate-fernet-key

Settings that matter

[core]
executor = CeleryExecutor
parallelism = 32                    # = real worker capacity
max_active_tasks_per_dag = 16
max_active_runs_per_dag = 16
default_pool_task_slot_count = 128

[operators]
default_deferrable = true           # ← HIGHEST-VALUE ONE LINE

[scheduler]
min_file_process_interval = 30
parsing_processes = 2
max_tis_per_query = 512
scheduler_zombie_task_threshold = 300

[celery]
worker_concurrency = 16             # cores × 2 for I/O-bound

[triggerer]
default_capacity = 1000

[logging]
remote_logging = True
remote_base_log_folder = s3://acme-airflow-logs/
delete_local_logs = True

[secrets]
backend = ...VaultBackend
use_cache = True
cache_ttl_seconds = 900

[metrics]
statsd_on = True

Metrics and alerts

Metric	Threshold
`scheduler_scheduler_loop_duration`	> 1 s degraded, > 10 s broken
`dag_processing_import_errors`	> 0 — a DAG is silently absent
`pool_starving_tasks`	> 0 — an invisible bottleneck
`executor_queued_tasks`	rising and staying up
`dagrun_schedule_delay`	runs starting late
`dag_processing_total_parse_time`	rising = a slow DAG file
`triggers_blocked_main_thread`	a bad synchronous trigger
`triggerer_heartbeat`	stale = deferred tasks hang

The XCom rule

   ✓ a path, a URI, a job id, a row count, a flag
   ✗ DataFrames, query results, file contents, SECRETS

   Data goes to object storage. Airflow carries a string.

Sensor modes

Mode	Worker slot
`poke` (default)	held the whole wait
`reschedule`	released between pokes; a DB row per poke
`deferrable=True`	nothing held at all

Always set timeout. Prefer an Asset to a sensor when the producer is another DAG.

Idempotency patterns

   ✓ DELETE the interval, then INSERT
   ✓ overwrite a partition  (date=2026-08-29/)
   ✓ MERGE / INSERT ... ON CONFLICT DO UPDATE
   ✓ write to staging, then swap atomically

   ✗ append without a key
   ✗ increment a counter
   ✗ read current state and add to it

Diagnosis order

1. airflow dags list-import-errors     is the DAG there?
2. is it paused? next-execution        should it have run?
3. what state?
     scheduled → pools / concurrency
     queued    → executor / workers / wrong queue
     running   → your code
     upstream_failed → look upstream
4. task logs
5. scheduler logs      zombies, parse failures
6. database            table sizes, dead tuples, connections
7. worker health       OOMKilled?

Do not

✗ do I/O at DAG module level (HTTP, DB, Variable.get, datetime.now)
✗ leave catchup=True with an old start_date
✗ backfill without max_active_runs
✗ put data — or secrets — in XComs
✗ leave sensors in poke mode
✗ omit execution_timeout
✗ omit retries, or retry deterministic failures
✗ write non-idempotent tasks
✗ use all_success on a join after a branch
✗ use depends_on_past unless the data requires it
✗ split work into tasks finer than Airflow's per-task overhead
✗ skip db clean until something breaks
✗ run db migrate without a backup
✗ use SQLite, or a shared NFS volume for DAGs
✗ treat Airflow as a processing engine, a job queue,
  a CI system, or a microservice orchestrator

The health checklist

☐ default_deferrable = true
☐ catchup=False on every DAG
☐ execution_timeout in default_args everywhere
☐ DagBag import test in CI
☐ every DAG file parses in under a second
☐ db clean scheduled weekly, as a DAG
☐ autovacuum tuned on task_instance
☐ PgBouncer in front of PostgreSQL
☐ remote logging to S3 with a lifecycle policy
☐ secrets backend, not the metadata database
☐ DAGs baked into the image
☐ 2–3 schedulers, 2 triggerers
☐ alerts on loop_duration, import_errors, pool_starving
☐ DATA checks, not just task-failure alerts
☐ on Airflow 3 (workers hold no DB credentials)

That is Airflow. It usually sits on PostgreSQL — whose tuning chapter is the other half of chapter 10 — dispatches through Redis or RabbitMQ, increasingly runs on Kubernetes, and should be getting its credentials from Vault rather than from its own database.

Written by Sepuri Sai Krishna