5. A DAG run traced end to end

This is the chapter to return to when a task is stuck and you do not know which component to look at.

The setup: a daily DAG, extract → transform → load, CeleryExecutor with Redis, three workers, PostgreSQL metadata database. It is 02:00 on 30 August 2026 and the interval Aug 29 00:00 → Aug 30 00:00 has just ended.

Step 0 — the DAG is already parsed

Before anything, the DAG processor has done its work: it imported the file, found the DAG object, and wrote a serialised representation into the database. The scheduler never reads your .py file.

   dags/daily_sales.py  ──parse──►  serialized_dag table
                                        ▲
                                        │ the scheduler reads THIS

This is why a change to a DAG file can take up to min_file_process_interval seconds to appear, and why an import error means the DAG simply does not exist rather than failing loudly.

Step 1 — the scheduler creates the DAG run

The scheduler loop, running every few seconds, asks: has the next data interval ended?

   next interval end = Aug 30 00:00
   now               = Aug 30 02:00
   → yes. Create the run.

INSERT INTO dag_run (dag_id, run_id, logical_date,
                     data_interval_start, data_interval_end, state)
VALUES ('daily_sales', 'scheduled__2026-08-29T00:00:00+00:00',
        '2026-08-29 00:00', '2026-08-29 00:00', '2026-08-30 00:00', 'queued');

and one task_instance row per task, state NULL.

Note the run is created at 02:00 and its logical_date is Aug 29 00:00 — chapter 4’s whole point, visible in a database row.

Cost: milliseconds. But this happens under a row lock, and with several schedulers running, SELECT ... FOR UPDATE SKIP LOCKED is what keeps them from creating the same run twice.

Step 2 — dependencies are evaluated

The scheduler asks, for each task instance: are you ready?

   extract    upstream: none               → READY
   transform  upstream: extract (NULL)     → not ready
   load       upstream: transform (NULL)   → not ready

extract moves to scheduled.

   NULL → SCHEDULED
   "the scheduler agrees this task's dependencies are met"

Cost: milliseconds, but this is the query that scales badly. It touches task_instance for every active run. Chapter 10.

Step 3 — concurrency and pools

scheduled does not mean it will run. The scheduler now checks the limits, in order:

   parallelism                     ← global: all tasks, all DAGs
   max_active_tasks_per_dag        ← this DAG
   pool slots available            ← this task's pool
   max_active_tis_per_dag          ← this specific task across runs

If any is exhausted the task stays scheduled and is reconsidered next loop.

scheduled means “waiting for Airflow”. queued means “waiting for a worker”. Telling them apart is most of diagnosing a stuck task.

Assuming there is room:

   SCHEDULED → QUEUED

and the scheduler hands it to the executor.

Step 4 — the executor dispatches

With Celery, the executor publishes a message to Redis:

   scheduler ──► Redis ──► one of the three workers picks it up

The message is small — DAG id, task id, run id, try number — not your code. The worker must already have the DAG file, which is why chapter 2 insisted that workers and schedulers carry identical code.

   Cost: 10–100 ms with Celery
         1–30 s with KubernetesExecutor (pod creation,
           image pull, container start)

That difference is the main practical distinction between the two executors and chapter 7 quantifies it.

Step 5 — the worker starts the task

The worker:

   1. receives the message
   2. PARSES the DAG file to find the task object
      ← module-level code runs HERE too (chapter 3)
   3. forks a task-runner process
   4. sets state RUNNING and starts a heartbeat
   5. executes your code

   QUEUED → RUNNING

Step 2 is worth pausing on. Every task execution re-parses the DAG file on a worker. A DAG file that takes three seconds to import adds three seconds to every task in it. This is the second place the top-level code rule bites, and it is less obvious than the first.

The heartbeat in step 4 matters for step 8.

Step 6 — your code runs

@task
def extract(**context):
    start = context["data_interval_start"]
    end   = context["data_interval_end"]
    rows = warehouse.query(f"... WHERE ts >= '{start}' AND ts < '{end}'")
    return {"rows": len(rows)}

The return value is pushed to XCom — a row in the xcom table (chapter 9).

   RUNNING → SUCCESS

or, on an exception:

   RUNNING → UP_FOR_RETRY   (if attempts remain)
           → FAILED         (if not)

Cost: however long your code takes. This is the only step that should be measured in minutes, and if any other step is, something is wrong.

Step 7 — downstream unlocks

The scheduler’s next loop sees extract is success, evaluates transform’s dependencies, and the cycle repeats from step 2.

   scheduler loop interval = how long between a task
   finishing and its downstream starting

Typically a few seconds. On a loaded scheduler it can be tens of seconds, and that gap — multiplied across a deep DAG — is often what people are measuring when they say Airflow is slow. A 20-task chain with a 10-second loop spends over three minutes doing nothing at all.

Step 8 — the run completes

When every task is in a terminal state, the run is closed:

   all SUCCESS or SKIPPED    → run state SUCCESS
   any FAILED                → run state FAILED

The whole picture

   02:00:00  interval ended; scheduler creates the run     ms
   02:00:03  extract: NULL → SCHEDULED                     ms
   02:00:03  limits checked → QUEUED                       ms
   02:00:03  published to Redis                            ms
   02:00:04  worker picks it up, PARSES THE DAG           0.1–3 s ◄
   02:00:05  → RUNNING
   02:04:30  → SUCCESS                                     4.5 min (your code)
   02:04:35  scheduler loop: transform → SCHEDULED         ~5 s   ◄
   ...

   Airflow's own overhead: a few seconds per task
   Your code:              everything else

The two arrows are the only places Airflow adds time, and both are tunable: DAG parse cost (chapter 3) and scheduler loop latency (chapter 13).

The states, and what each means

State	Meaning
(none)	the run exists; the task has not been considered
scheduled	dependencies met, waiting for Airflow — a limit is full
queued	sent to the executor, waiting for a worker
running	executing, heartbeating
success	finished cleanly
failed	finished badly, no retries left
up_for_retry	failed, waiting out retry_delay
up_for_reschedule	a sensor in reschedule mode, sleeping
deferred	handed to the triggerer (chapter 11)
skipped	a branch or trigger rule excluded it
upstream_failed	an upstream task failed
removed	it was in the run but no longer exists in the DAG
restarting	asked to stop and start again

The three that answer most questions:

scheduled for a long time → a concurrency limit or a full pool. Nothing is broken; Airflow is throttling you deliberately.

queued for a long time → no worker took it. Workers are down, saturated, listening to a different queue, or the Kubernetes pod cannot schedule.

upstream_failed → not this task’s problem. Look upstream.

Zombies and the heartbeat

Step 5 started a heartbeat. If the worker dies — OOM killed, node terminated, network partition — the task is still running in the database and nothing is executing it.

The scheduler detects this:

   task_instance is RUNNING
   AND its last heartbeat is older than
       scheduler_zombie_task_threshold (default 300 s)
   → declare it a ZOMBIE
   → mark it failed, or retry it

This is Airflow’s answer to the general distributed-systems problem: you cannot tell a dead process from a slow one, so you use a timeout and accept the risk. A task that stops heartbeating for six minutes but is genuinely alive gets killed and retried — which is fine if your tasks are idempotent, and is a correctness problem if they are not.

up_for_retry after an OOM is the same story: the process vanished, the heartbeat stopped, the scheduler noticed.

Where the time actually goes

Symptom	Step	Fix
DAG does not appear at all	0	import error — airflow dags list-import-errors
DAG appears, no runs	1	paused; catchup; start_date in the future
task stuck scheduled	3	pool full, or a concurrency limit
task stuck queued	4	workers down, saturated, or wrong queue
every task slow to start	5	DAG file parse time
gaps between tasks	7	scheduler loop latency — chapter 13
tasks randomly up_for_retry	8	OOM, or zombie detection — check worker memory
everything slow, no errors	all	the metadata database — chapter 10

What to take from this chapter

• The scheduler reads serialised DAGs from the database, never your files. An import error means the DAG does not exist, silently.
• A run is created after its interval ends, with one task_instance row per task.
• scheduled = waiting for Airflow (a limit is full). queued = waiting for a worker. This distinction diagnoses most stuck tasks.
• The dispatch cost is 10–100 ms with Celery, 1–30 s with Kubernetes — pod creation and image pull.
• The worker re-parses the DAG file for every task. Slow module-level code costs you on every single execution, not just at parse time.
• The scheduler loop interval is the gap between a task finishing and its downstream starting — a few seconds each, which compounds through a deep DAG.
• Airflow’s own overhead is a few seconds per task; everything else is your code.
• Zombie detection kills tasks whose heartbeat stops for 300 seconds. It cannot distinguish dead from slow, which is why idempotent tasks are a correctness requirement, not a nicety.

Next: operators, sensors and hooks — the building blocks, which ones to use, and which ones to avoid.