14. Failure modes

A catalogue. Each entry: what you see, what is happening, what to do, how to prevent it.

The pattern worth noticing: Airflow’s failures are mostly silent. It does not crash. It stops doing something, or does something you did not intend, and the signal is absence.

---

1. The DAG that never appears

Symptoms. You wrote a DAG, deployed it, and it is not in the UI. No error anywhere.

Cause. An import error. The DAG processor could not import the file, so no DAG object exists, so there is nothing to show.

What to do.

airflow dags list-import-errors      # the answer, almost always
airflow dags list | grep my_dag
python dags/my_dag.py                # reproduce the import locally

Common causes: a missing provider package, a syntax error, a module-level exception (an API call at import time failing), a DAG created inside a function without being assigned to globals(), or the file not being in dags_folder at all.

Prevention. The DagBag import test in CI from chapter 3. This is a five-line test that catches most deployment breakage before it exists.

---

2. The DAG that appears but never runs

Symptoms. The DAG is listed. No runs are created. Nothing in the logs.

Cause. In order of likelihood:

   1. IT IS PAUSED. (Check first. It is usually this.)
   2. catchup=False and the current interval has not ENDED yet
   3. start_date is in the future
   4. end_date has passed
   5. schedule=None — it is manual-only
   6. max_active_runs already reached

What to do.

airflow dags unpause my_dag
airflow dags next-execution my_dag       # when will it run?
airflow dags details my_dag

Prevention. dags_are_paused_at_creation = False if your workflow is “deploy means live”; otherwise a deployment step that unpauses.

---

3. Tasks stuck in scheduled

Symptoms. Tasks sit in scheduled. Workers are idle. Nothing moves.

Cause. A concurrency limit. Airflow is deliberately throttling you (chapter 13).

What to do. Check them in order:

airflow pools list                        # any pool at capacity?
airflow config get-value core parallelism

   parallelism                 (whole installation)
   max_active_tasks_per_dag    (this DAG)
   pool slots                  (this task's pool)
   default_pool                (128 — everything unassigned)
   max_active_tis_per_dag      (this task across runs)

The metric airflow_pool_starving_tasks names this directly and is worth an alert, because a starved pool looks exactly like a healthy queue.

Prevention. Size parallelism to real worker capacity, and know that default_pool exists.

---

4. Tasks stuck in queued

Symptoms. Tasks sit in queued and never start.

Cause. The executor could not place them (chapter 7).

What to do.

   Celery:
     are workers running?           airflow celery worker alive?
     is the task's QUEUE served?    ← silent forever if not
     is Redis/RabbitMQ reachable?
     are all slots full of SENSORS? ← very common
   Kubernetes:
     kubectl get pods -n airflow — pending?
     insufficient CPU/memory? node not scaling?
     ImagePullBackOff?

The wrong queue case has no error message at all. A task routed to queue="gpu" with no worker listening to gpu waits forever, silently. Airflow does not warn that a queue has no consumers.

Prevention. Alert on airflow_executor_queued_tasks rising and staying up. Use deferrable operators so sensors are not consuming your pool.

---

5. The accidental catchup

Symptoms. You deploy a DAG and hundreds of runs appear at once. Your database, or the API your tasks call, falls over.

Cause. catchup=True (the Airflow 2 default) with a start_date far in the past.

What to do — immediately:

airflow dags pause my_dag
airflow dags delete my_dag        # removes runs; DESTRUCTIVE, be sure

Or mark the unwanted runs as failed in the UI and let it settle.

Prevention. catchup=False on every DAG, and max_active_runs set whenever you do want history. Airflow 3 changed the default, which helps considerably.

---

6. Zombie tasks

Symptoms. Tasks marked up_for_retry or failed with a log ending abruptly, or Detected zombie job in scheduler logs.

Cause. The worker process died — OOM killed, node terminated, spot instance reclaimed, network partition — so the heartbeat stopped and the scheduler reclaimed the task (chapter 5).

What to do. Find out why the worker died:

kubectl describe pod <worker>       # look for OOMKilled
dmesg | grep -i "killed process"

Almost always memory. A task loading a large DataFrame is chapter 1’s rule being violated.

Prevention. Give workers enough memory and set limits. Move heavy work into KubernetesPodOperator with its own resources. Set execution_timeout. And make tasks idempotent, because zombie recovery reruns them whether you like it or not.

---

7. Everything is slow and nothing is broken

Symptoms. Tasks take a long time to move between states. The UI is sluggish. No errors. Scheduler CPU is low.

Cause. The metadata database (chapter 10). This is the classic Airflow scaling failure and it arrives gradually.

What to do.

SELECT relname, n_live_tup, n_dead_tup
FROM pg_stat_user_tables ORDER BY n_live_tup DESC LIMIT 10;

SELECT count(*) FROM pg_stat_activity;

airflow db clean --clean-before-timestamp 2026-01-01 --dry-run

Then: clean, vacuum, check connections, add PgBouncer.

Prevention. Scheduled db clean from day one. Alert on scheduler_loop_duration. Fewer, chunkier tasks.

---

8. A DAG file that slows everything down

Symptoms. All DAGs are slow to update. dag_processing_total_parse_time is high. Scheduling lags across the board.

Cause. Module-level code in one or more DAG files (chapter 3) — an API call, a database query, a large import.

What to do.

airflow dags report      # parse duration per file — find the offender

Prevention. The top-level code rule, and a CI check that fails a build if any DAG file takes more than a second to import.

---

9. Sensors consuming the entire worker pool

Symptoms. All workers busy. Tasks queued. The running tasks are all sensors, doing nothing.

Cause. mode="poke" (the default), holding a slot for the whole wait (chapter 6).

What to do. Immediately: mode="reschedule". Properly: deferrable=True, or default_deferrable = true installation-wide.

Prevention. default_deferrable = true and a review habit of asking whether a sensor is needed at all — an asset is better when the producer is another DAG.

---

10. Duplicate or doubled data

Symptoms. Row counts twice what they should be. A metric that jumped.

Cause. A non-idempotent task ran more than once — a retry, a zombie rerun, a manual clear, or a backfill over data already processed.

What to do. Fix the data, then fix the task (chapter 8): delete-then-insert, partition overwrite, or upsert.

Prevention. Idempotency is not optional. You cannot prevent re-execution — zombie detection alone guarantees it will happen eventually.

---

11. Import errors on some workers only

Symptoms. The same task succeeds on one worker and fails with ModuleNotFoundError on another.

Cause. Workers do not all have the same code or dependencies. A rolling deployment mid-flight, git-sync partially applied, or one worker on an old image.

What to do. Check versions across workers; restart the stragglers.

Prevention. Bake DAGs into the image (chapter 13). This failure is largely a consequence of git-sync or shared volumes.

---

12. The depends_on_past deadlock

Symptoms. A DAG stops producing runs. No failures visible today. Every task is waiting.

Cause. depends_on_past=True and an old failure that nobody cleared. Every subsequent run is blocked on it, forever.

What to do. Find the earliest failure, clear or mark it success, and the chain unblocks.

Prevention. Use depends_on_past only where the data genuinely requires ordering, and alert on DAG runs not starting — a DAG that quietly stops is not visible in failure alerts, because nothing failed.

---

13. Diagnosis order

   1. Is the DAG there?
        airflow dags list-import-errors

   2. Is it paused? Should it have run?
        airflow dags next-execution my_dag

   3. What state are the tasks in?
        scheduled  → concurrency/pools
        queued     → executor/workers
        running    → your code
        upstream_failed → look upstream

   4. Task logs.
        airflow tasks logs, or the UI

   5. Scheduler logs.
        for zombies, parse failures, loop duration

   6. Database health.
        table sizes, dead tuples, connection count

   7. Worker health.
        memory, OOMKilled, restarts

Steps 1–3 resolve most incidents. The single most useful habit is knowing what scheduled and queued mean, because that one distinction routes you to the right half of the system immediately.

What to take from this chapter

• Airflow fails silently. A missing DAG, a paused DAG, a starved pool and a queue nobody listens to all look like nothing happening.
• DAG not appearing = import error. airflow dags list-import-errors, and a DagBag test in CI.
• DAG not running = it is paused, or catchup/start_date/max_active_runs.
• scheduled = a concurrency limit (including default_pool’s 128). queued = the executor could not place it — and a wrong queue waits forever with no error.
• The accidental catchup is the classic first-day incident. catchup=False, always max_active_runs when backfilling.
• Zombies are almost always OOM. Give workers memory, use KubernetesPodOperator for heavy tasks, set execution_timeout, and make tasks idempotent — re-execution is guaranteed.
• Slow-everything with low scheduler CPU is the metadata database. It degrades gradually over months if db clean never runs.
• One DAG file with module-level I/O slows the whole installation. airflow dags report finds it.
• Sensors in poke mode eat the worker pool. default_deferrable = true.
• depends_on_past deadlocks silently — nothing failed, so nothing alerted.

Next: when to use Airflow and when not — the comparison with Dagster, Prefect, Temporal, Argo Workflows and plain cron.