1. What Airflow is, and why it exists

Airflow exists because of a problem that starts small and gets bad quickly: you have jobs that must run in an order, and some of them fail.

That sentence sounds trivial. Most of this guide is about how much machinery it actually takes.

The problem

Start where everyone starts. You have a nightly report:

# crontab
0 2 * * *  /opt/jobs/extract.sh
0 3 * * *  /opt/jobs/transform.sh
0 4 * * *  /opt/jobs/load.sh

This works, right up until it doesn’t. Every failure below is one somebody has actually had:

The gap is a guess. transform.sh runs at 03:00 because extract “usually takes about forty minutes”. One day it takes seventy, and transform runs on half-written data. Nothing errors. The report is wrong and nobody notices for a week.

A failure is silent. extract.sh exits 1. Cron mails root, which nobody reads. Transform and load run anyway, on yesterday’s data.

Retrying is manual. The API was down for ninety seconds. Someone has to notice, find the script, work out which arguments it had, and run it by hand.

Backfilling is a hand-written loop. You add a column and need thirty days of history. Now you are writing a bash loop over dates, hoping nothing runs twice, hoping nothing runs concurrently that shouldn’t.

Nobody knows what ran. Did last Tuesday’s job succeed? How long did it take? What were its logs? Cron does not remember anything.

It runs on one machine. That machine is a single point of failure, and when the work outgrows it there is nowhere to go.

Every one of these is solvable with enough shell. Solving all of them is how you accidentally write a worse Airflow.

The answer: dependencies, not times

Airflow’s central move is to stop scheduling tasks by time and start scheduling a graph, by dependency.

   CRON                              AIRFLOW

   extract  at 02:00                 extract
   transform at 03:00  ← a guess       ↓  (when extract SUCCEEDS)
   load     at 04:00  ← a guess      transform
                                       ↓  (when transform SUCCEEDS)
                                     load

   the times are hope                the arrows are enforced

You declare the shape of the work. Airflow decides when each piece may start, which is: when everything it depends on has finished successfully. There is no gap to guess, because there is no gap.

Everything else follows from having that graph written down:

Retries are automatic, because Airflow knows a task failed and knows how to run it again.

Downstream work stops, because a failed task’s dependents never become eligible.

Backfilling is a first-class operation, because a run is parameterised by a date and Airflow can create as many as you ask for.

History exists, because every task attempt is a row in a database with a state, timestamps and logs.

Work distributes, because tasks are dispatched to workers rather than executed by the scheduler.

The DAG

The graph is a DAG — a Directed Acyclic Graph. Three words, each carrying weight:

   DIRECTED   edges have a direction: A must finish
              before B starts

   ACYCLIC    no loops. A→B→C→A is rejected, because
              nothing in it could ever start

   GRAPH      not a list. Tasks can fan out, fan in,
              and run in parallel

The acyclic part is the one with consequences. Airflow cannot express “keep doing this until X”. There is no loop construct, no while. Branching exists, and dynamic task generation exists, but the shape of the graph is fixed before it runs. If your problem is genuinely iterative, chapter 15 is honest about it being the wrong tool.

In Python:

from airflow.sdk import dag, task
import pendulum

@dag(
    schedule="0 2 * * *",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
)
def daily_report():

    @task
    def extract():
        return {"rows": 4211}

    @task
    def transform(data):
        return data["rows"] * 2

    @task
    def load(n):
        print(f"loaded {n}")

    load(transform(extract()))

daily_report()

Three tasks, dependencies inferred from how the functions are called. This is the TaskFlow API, and it is how new Airflow should be written.

Where it came from

Maxime Beauchemin built Airflow at Airbnb in 2014. He had worked on Facebook’s internal Dataswarm, and the problem was the same everywhere: data teams were accumulating hundreds of interdependent jobs with no way to express the interdependence.

The design decision that made Airflow spread was configuration as code. Its competitors at the time — Oozie, Azkaban, Luigi to a degree — described workflows in XML or config files. Airflow made a DAG a Python file. Pipelines could be generated, parameterised, tested, reviewed and imported like any other code, and that turned out to matter more than any feature.

It went to the Apache Incubator in 2016 and graduated in 2019. It is now the default answer for data orchestration by a very large margin — which is a mixed blessing, because it is also used for a great many things it is not good at.

Three version facts you need:

Airflow 1.x is gone and its advice is wrong. If you find a blog post using PythonOperator with provide_context=True, it predates everything.

Airflow 2.0 (December 2020) was a near-rewrite: a highly available scheduler, the TaskFlow API, providers split into separate packages, and a REST API. Everything modern starts here.

Airflow 3.0 (April 2025) changed the execution model. Tasks no longer connect to the metadata database directly — they go through a Task Execution API — which is the single most important architectural change since 2.0 and the subject of much of chapter 2. It also brought DAG versioning, scheduler-managed backfills, a rewritten UI, and the removal of SubDAGs and SLAs.

This guide is written against Airflow 2.9–3.x, and says which is which wherever they differ.

The one thing to understand early

Airflow is a scheduler, not a data processor.

This is the mistake that ruins Airflow installations, and it is worth stating before anything else.

Airflow’s job is to decide what runs, when, and in what order, and to record what happened. It is not built to process your data. A task that loads ten million rows into a pandas DataFrame inside an Airflow worker is using a scheduler as a compute cluster, and it will go badly: workers OOM, the scheduler’s heartbeat is starved, and a machine sized for orchestration is being asked to do analytics.

The shape that works:

   ✗ WRONG                          ✓ RIGHT

   task: read 10M rows into         task: submit a query to
         pandas, transform,               ClickHouse / Snowflake
         write back                task: wait for it
                                    task: submit the next one

   Airflow is doing the work        Airflow is TELLING SOMETHING
                                    ELSE to do the work

A well-behaved Airflow task issues a command to a system built for the job — a warehouse, a Spark cluster, a Kubernetes Job — and waits. Airflow moves control, not data. Chapters 9 and 15 come back to this repeatedly, because almost every serious Airflow performance problem is a violation of it.

The four jobs people use it for

1. ETL and ELT. Move data from sources into a warehouse, then transform it there. This is the original purpose and still the dominant one.

2. Orchestrating other systems. Kick off a Spark job, wait for it, then trigger a dbt run, then refresh a dashboard. Airflow as the conductor rather than the orchestra.

3. Scheduled maintenance and reporting. Anything cron did, with retries, history and alerting attached.

4. Machine-learning pipelines. Feature extraction, training, evaluation, deployment — with the caveat that ML-specific tools exist and chapter 15 discusses whether you want one.

What it is not

It is not a streaming system. Airflow’s granularity is a scheduled run. Nothing here is for events arriving continuously — that is Kafka and a stream processor. Airflow 3’s asset-triggered DAGs blur this slightly and do not change it.

It is not a low-latency job queue. Scheduling latency is seconds at best. If you need “run this now, in 50 ms”, use RabbitMQ or a task queue.

It is not a data processing engine. Stated above; it bears repeating.

It is not a CI/CD system. People do use it that way. Argo CD and a pipeline are better at it.

It is not for sub-minute schedules. The scheduler loop, database round trips and worker dispatch mean a task every ten seconds is fighting the architecture.

The mental model to carry

Five things. The rest of this guide is detail:

1. A DAG is a Python file that DESCRIBES a graph. It is
   parsed constantly, in a separate process, and code at
   the top level of the file runs on EVERY parse.

2. A DAG RUN is one execution of that graph for one
   data interval. Tasks belong to runs.

3. The SCHEDULER decides what may run. The EXECUTOR
   decides where. WORKERS run it. They are different
   things and the failure modes differ.

4. The METADATA DATABASE is the source of truth for
   every state transition — and it is the bottleneck
   in almost every large installation.

5. Airflow SCHEDULES work; it should not DO the work.
   Tasks should tell other systems what to do.

Points 1 and 4 are where nearly all real-world Airflow trouble lives, which is why they are chapters 3 and 10 rather than an appendix.

Why Airflow in 2026

Fair question — Dagster, Prefect, Temporal and Argo Workflows all exist and all improve on something.

The ecosystem is the argument. Ninety-plus provider packages: every cloud, every warehouse, every database, dbt, Spark, Kubernetes. Whatever you need to talk to, an operator exists, and it has been used by thousands of people before you.

Everyone already knows it. Hiring someone who has run Airflow is easy. That is not an engineering argument and it is a real one.

It is genuinely good at the boring middle. A few hundred DAGs of scheduled batch work with retries, alerting, backfills and an audit trail is exactly its shape, and it does that reliably.

The honest counterpoint, so chapter 15 is not a surprise: Airflow’s data-passing model is weak, its local development experience is poor, and its scheduling abstractions are historically confusing — the execution_date naming (chapter 4) has confused every person who has ever used it. Newer tools fixed these deliberately. Whether that is worth leaving the ecosystem is a real decision, and chapter 15 argues both sides.

What to take from this chapter

Cron breaks because the gap between jobs is a guess, failures are silent, retries are manual, backfills are hand-written, and nothing is remembered.
Airflow schedules a graph by dependency, not tasks by time. A task starts when its upstream succeeds — so retries, stopping downstream work, backfills and history all follow for free.
A DAG is directed and acyclic: no loops, and the shape is fixed before the run starts.
Airflow’s original advantage was configuration as code — a DAG is a Python file you can generate, test and review.
Airflow 2.0 brought the HA scheduler and TaskFlow API; Airflow 3.0 stopped tasks talking to the metadata database directly and added DAG versioning. Anything about 1.x is wrong.
Airflow is a scheduler, not a data processor. A task should tell another system to do the work. Almost every serious performance problem is a violation of this.
It is not streaming, not a low-latency queue, not a processing engine, not CI/CD, and not for sub-minute schedules.
Carry five ideas: DAG files are parsed constantly, a DAG run is one execution for one interval, scheduler / executor / worker are different, the metadata database is the bottleneck, and schedule work rather than doing it.

Next: the architecture — the five components, what changed in Airflow 3, and why the metadata database sits in the middle of everything.