2. The architecture

Airflow is five or six processes arranged around one database. Learning which process does what shortens every troubleshooting session, because the first question is always which one is unhappy.

The whole thing on one page

   ┌────────────────────────────────────────────────────────────┐
   │                                                            │
   │  ┌──────────────┐   ┌───────────────┐   ┌───────────────┐ │
   │  │ DAG PROCESSOR│   │  SCHEDULER    │   │  API SERVER   │ │
   │  │ parse .py    │   │ decide what   │   │  UI + REST +  │ │
   │  │ files, write │   │ may run now   │   │  Task Exec API│ │
   │  │ serialised   │   │               │   │               │ │
   │  │ DAGs         │   │  + EXECUTOR   │   └───────┬───────┘ │
   │  └──────┬───────┘   └───┬───────┬───┘           │         │
   │         │               │       │               │         │
   │         ▼               ▼       │               ▼         │
   │  ┌──────────────────────────────────────────────────────┐ │
   │  │       METADATA DATABASE  (PostgreSQL)                │ │
   │  │  DAG runs, task instances, state, XComs,             │ │
   │  │  connections, variables, pools, logs of state        │ │
   │  │  ── THE SOURCE OF TRUTH, AND THE BOTTLENECK ──       │ │
   │  └──────────────────────────────────────────────────────┘ │
   │                          ▲                                 │
   │  ┌──────────────┐        │           ┌──────────────────┐ │
   │  │  TRIGGERER   │────────┘           │    WORKERS       │ │
   │  │ async waits  │                    │  run the tasks   │ │
   │  │ (deferrable) │                    │                  │ │
   │  └──────────────┘                    └──────────────────┘ │
   └────────────────────────────────────────────────────────────┘
                                    ▲
                        ┌───────────┴──────────┐
                        │  BROKER (Celery only)│
                        │  Redis or RabbitMQ   │
                        └──────────────────────┘

The scheduler

The heart of the system. It runs a loop, forever:

   forever:
     1. should any DAG have a new run? (its schedule fired)
        → create DAG runs
     2. for every running DAG run, which task instances
        have all their dependencies satisfied?
        → mark them SCHEDULED
     3. of those, which fit within concurrency limits,
        pool slots and priority?
        → mark them QUEUED and hand to the EXECUTOR
     4. reconcile: tasks whose worker died, tasks stuck
        in queued, timed-out runs

Two things worth knowing immediately.

The scheduler does not run your tasks. It decides what is eligible. Confusing this is the reason people size scheduler machines for compute; the scheduler is almost entirely database-bound.

The scheduler is highly available since 2.0. Run two or three; they coordinate through row-level locks in the metadata database (SELECT ... FOR UPDATE SKIP LOCKED) rather than through leader election. No node is primary. If you have read the PostgreSQL locking chapter, this is exactly the SKIP LOCKED job-queue pattern — several schedulers pull distinct work without blocking each other.

That design has a direct consequence: the metadata database is where scheduler coordination happens, so scheduler throughput is database throughput. Chapter 10 is about that.

The executor

Not a process — a strategy, configured in the scheduler, deciding where a queued task actually runs.

Executor	Where tasks run	Needs
`LocalExecutor`	subprocesses on the scheduler machine	nothing
`CeleryExecutor`	a fixed pool of worker processes	Redis or RabbitMQ
`KubernetesExecutor`	one pod per task	a cluster
`CeleryKubernetesExecutor`	either, per task queue	both
`EdgeExecutor` (3.x)	remote workers over HTTP	—

Chapter 7 compares them properly. The thing to hold now is that the executor is the boundary between “Airflow decided” and “something ran”, and most “my task is stuck in queued” problems live exactly there.

Workers

Where your code executes. With Celery they are long-running processes pulling from a broker; with Kubernetes they are pods created per task and destroyed after.

Workers need:

your DAG files — a worker parses the DAG to find the task it is meant to run
every dependency your tasks import
network access to whatever your tasks talk to

That first bullet is the source of a lot of pain. Every worker needs the same DAG code as the scheduler, and a mismatch produces tasks that fail with import errors on some workers and not others. Chapter 13 covers distribution: baking DAGs into the image (best), git-sync (common), or a shared volume (avoid).

The DAG processor

Parsing DAG files is unbounded work — it runs arbitrary Python — so it is done in its own process, separately from scheduling.

   for each .py file in dags/:
     import it in a subprocess
     find DAG objects
     SERIALISE them into the metadata database
     record any import error

The serialisation matters. Since 2.0 the scheduler and UI read DAGs from the database, not from the files. The processor is the only component that executes your DAG files, which is why a DAG can appear in the UI seconds after you save it and why the scheduler never blocks on a slow import.

In Airflow 3 the DAG processor is a separate, mandatory component, and this is a security improvement worth naming: the scheduler no longer executes user code at all. In 2.x it could run inside the scheduler process, and a DAG file with an infinite loop at import time could take scheduling down.

Two settings shape its behaviour:

[dag_processor]
dag_file_processor_timeout = 50    # kill a parse that takes longer
min_file_process_interval = 30     # re-parse each file at most this often
parsing_processes = 2

min_file_process_interval is the delay between saving a DAG and seeing it. Lowering it to 5 makes development pleasant and multiplies parsing load; 30 is a reasonable production value.

The API server

In Airflow 2 this was the webserver: the UI plus a REST API. In Airflow 3 it is the API server and it does something new and important.

The Airflow 3 execution change

This is the biggest architectural shift since 2.0.

   AIRFLOW 2                        AIRFLOW 3

   worker ──────► METADATA DB       worker ──► API SERVER ──► DB
   (a task had full database         (a task calls the Task
    credentials and could             Execution API; no DB
    read or write ANYTHING)           credentials at all)

In Airflow 2, every worker — and therefore every task, and therefore any code anyone put in a DAG — held credentials to the metadata database. A task could read every connection and every variable in the installation, including ones for systems it had nothing to do with. In a multi-team setup that is not a boundary at all.

Airflow 3’s Task Execution API ends it. Tasks talk to the API server over HTTP with a scoped, short-lived token; the API server does the database work. The consequences are broad:

Workers need no database credentials, so a compromised task cannot read the whole installation
Remote workers become viable — the Edge executor exists because of this
Tasks can be written in other languages in principle, since the interface is HTTP rather than SQLAlchemy
the database connection count drops, which matters more than it sounds (chapter 10)

If you are on Airflow 2, this is the strongest single argument for upgrading.

The triggerer

A process that runs asyncio, added in 2.2, and it solves a specific waste.

A sensor waiting for a file to appear occupies a whole worker slot doing nothing. A hundred sensors occupy a hundred slots. Teams routinely discovered their entire worker pool was asleep.

Deferrable operators hand the waiting to the triggerer:

   task starts → works out what it is waiting for
              → DEFERS: releases its worker slot entirely
              → the TRIGGERER polls, asynchronously,
                thousands of waits in one process
              → the condition is met
              → the task is rescheduled onto a worker
                and resumes

One triggerer handles thousands of concurrent waits, because each is an async coroutine rather than a process. Chapter 11 covers this properly. If your DAGs do a lot of waiting — for files, for external jobs, for other DAGs — this is the single largest efficiency win available in Airflow.

The metadata database

Everything durable:

Table	Holds
`dag` / `dag_version`	DAG definitions, schedule, pause state
`serialized_dag`	the parsed DAG structure the scheduler reads
`dag_run`	one row per run: interval, state, timestamps
`task_instance`	one row per task per run — the big one
`task_instance_history`	previous attempts (retries)
`xcom`	small values passed between tasks
`connection`, `variable`	credentials and config
`slot_pool`	concurrency pools
`log`, `job`	audit trail and component heartbeats

Use PostgreSQL. MySQL works and is less well trodden. SQLite is for a laptop only — it cannot support more than one scheduler or the LocalExecutor properly, for the reason its own chapter 11 gives: one writer per file, and Airflow wants several.

The reason this component gets its own chapter is arithmetic. task_instance grows as DAGs × tasks per DAG × runs per day, and the scheduler queries it constantly:

   200 DAGs × 30 tasks × 24 runs/day
     = 144,000 rows per day
     = 4.3 million rows per month

Every scheduler loop scans parts of this table. At a few million rows on an under-provisioned database, the loop slows, tasks queue, and everything feels broken — with no error anywhere. This is the most common Airflow scaling failure and it looks like nothing at all. Chapter 10 is about preventing it.

The broker — Celery only

With CeleryExecutor, a message broker sits between the scheduler and workers.

Redis is the usual choice: simple, fast, everywhere. RabbitMQ is more robust for the job. The trade is exactly the one the Redis and RabbitMQ guides describe — Redis can lose a message on failover because replication is asynchronous; RabbitMQ tracks per-message acknowledgement and is designed not to.

In practice Airflow tolerates a lost message better than you would expect, because the scheduler reconciles: a task queued but never started is eventually detected and rescheduled. That safety net is why most installations use Redis and are fine.

Where the failure modes live

A map for the rest of the guide:

Symptom	Component
DAG does not appear	DAG processor — import error
DAG appears, never runs	scheduler — schedule, `catchup`, or paused
tasks stuck in `queued`	executor / worker capacity
tasks stuck in `scheduled`	pools, concurrency limits
tasks fail with import errors	workers do not have the code or dependencies
everything slow, no errors	metadata database
UI slow but tasks fine	API server or database reads
sensors eating all workers	not using deferrable operators

What to take from this chapter

The scheduler decides what may run; it does not run anything. It is database-bound, and HA since 2.0 using SELECT ... FOR UPDATE SKIP LOCKED rather than leader election.
The executor is a strategy, not a process, and is the boundary where “stuck in queued” problems live.
Workers need your DAG code and every dependency. A mismatch between scheduler and worker code is a classic failure.
The DAG processor is the only component that executes your DAG files; it serialises DAGs into the database, and the scheduler and UI read from there. In Airflow 3 it is separate and mandatory.
Airflow 3’s Task Execution API means workers no longer hold database credentials — in Airflow 2 any task could read every connection in the installation. This is the strongest reason to upgrade.
The triggerer runs deferrable operators asynchronously, so a waiting task holds no worker slot. Thousands of waits, one process.
The metadata database is the source of truth and the bottleneck. task_instance grows as DAGs × tasks × runs; millions of rows makes everything slow with no error message.
Use PostgreSQL. SQLite is for a laptop.
With Celery, Redis is common and RabbitMQ is more robust — the scheduler’s reconciliation makes Redis survivable.

Next: DAG authoring — the TaskFlow API, dynamic DAGs, and the top-level code mistake that slows down every installation that makes it.

Written by Sepuri Sai Krishna