15. When to use Airflow, and when not

Airflow is the default choice for data orchestration, which means it is chosen a great deal without a decision being made. This chapter is the decision.

What Airflow is genuinely good at

Scheduled batch work with dependencies. The boring middle: a few hundred DAGs, retries, alerting, backfills, an audit trail of what ran. It does this reliably and has for a decade.

The ecosystem. Ninety-plus provider packages. Every cloud, every warehouse, every database, dbt, Spark, Kubernetes, Slack. Whatever you need to talk to, someone has already written the operator and thousands of people have used it.

Backfills. A first-class, well-understood operation. Most competitors treat historical reruns as an afterthought; Airflow’s whole model is built around a run being parameterised by an interval.

Being known. Hiring people who have run Airflow is easy. Every data engineer has debugged one. That is not an engineering argument and it is a real one.

Where its age shows

Stated plainly, because this is what the newer tools were built to fix.

Data passing is weak. XComs are database rows with a practical size limit, so passing anything real means writing to storage and passing a path (chapter 9). Dagster and Prefect treat data flow as the primary abstraction.

Local development is poor. There is no good way to run a DAG on your laptop with real inputs and iterate quickly. airflow tasks test helps; it is not a development experience.

The scheduling abstractions confused everyone. execution_date (chapter 4) is the most-explained concept in data engineering, and it should never have needed explaining. Airflow 3 fixed the naming; the model remains interval-centric in a world where much work is event-driven.

Dependencies are installation-wide. One image, one set of Python packages, negotiated across every team using the installation. KubernetesPodOperator works around it rather than solving it.

Testing is awkward. Testing a DAG’s logic means either extracting all of it into plain functions (which you should do anyway) or accepting that the orchestration layer is untested.

It is heavy. Scheduler, workers, triggerer, API server, PostgreSQL, a broker. For twenty scheduled jobs, that is a lot of machinery.

The comparisons

Airflow vs Dagster

The most substantive competitor, and the one that disagrees with Airflow philosophically rather than incrementally.

	Airflow	Dagster
Central concept	a task graph	assets — the data produced
Data passing	XComs, or do it yourself	first class, typed IO managers
Local development	poor	very good
Testing	awkward	designed for it
Type checking	none	types on inputs and outputs
Backfills	excellent	good
Ecosystem	enormous	growing
Maturity	a decade	since 2019
Hiring	easy	harder

Dagster’s core idea is that you should declare the assets you want to exist — this table, that model, that report — and let the system work out the tasks. Airflow’s assets (chapter 4) are a partial adoption of the same insight, which is a fair indication of who was right.

Choose Dagster if you are starting fresh, data lineage matters, you want real local development and testing, and typed data passing appeals. Choose Airflow if the ecosystem or the hiring pool decides it, or you already have one.

Airflow vs Prefect

	Airflow	Prefect
Definition	a DAG, fixed before it runs	plain Python, dynamic at runtime
Loops and conditionals	expand/branch only	native — it is just Python
Local development	poor	good
Hybrid execution	—	agents run anywhere; control plane is hosted
Ecosystem	enormous	smaller

Prefect’s argument is that a workflow is just Python and you should not have to express it as a graph in advance. If your pipeline has genuine loops or runtime-determined structure that expand cannot express, Prefect is a considerably better fit.

Airflow vs Temporal

Different category, frequently confused.

Temporal is for durable application workflows — a payment that must complete across five services, a multi-day user onboarding, a saga with compensations. It gives you exactly-once semantics, durable state, and code that can be paused for weeks and resume on a different machine.

Airflow is for scheduled batch data work.

   "run this nightly ETL"                → Airflow
   "this order must complete or roll
    back across four services, and it
    might take three days"               → Temporal

They rarely compete. When someone reaches for Airflow to orchestrate a long-lived business process, Temporal is what they wanted — and its chapter 15 draws the same line from the other side, where the giveaway is a DAG per order.

Airflow vs Argo Workflows

	Airflow	Argo Workflows
Definition	Python	Kubernetes YAML
Runs on	anything	Kubernetes only
Each step	a task	a container
Scheduling	rich	cron only
Backfills	first class	do it yourself
Data ecosystem	enormous	none

Argo Workflows is container-native and Kubernetes-native. If every step is already a container and you want no Python layer at all, it is lighter and cleaner. If you want warehouse operators, backfills and data-aware scheduling, it has none of that. It pairs naturally with Argo CD for teams already in that ecosystem.

Airflow vs dbt

Not a competitor. A complement, and one people confuse constantly.

dbt transforms data inside a warehouse with SQL, and it has its own dependency graph. It does not extract, does not load, and has no scheduler.

   Airflow:  extract → load → [ trigger dbt ] → notify
   dbt:                          the SQL DAG inside the warehouse

The common architecture is Airflow orchestrating everything around dbt and invoking dbt build as one or a few tasks. Cosmos renders a dbt project as Airflow tasks if you want per-model visibility, at the cost of many more task instances (chapter 10). dbt’s chapter 13 argues for starting with one task and splitting by tag before splitting by model.

Airflow vs cron

Worth taking seriously rather than dismissing.

Cron is fine when: the jobs are independent, there are few of them, failures are visible another way, and nobody backfills.

Cron stops being fine when: jobs depend on each other, you need retries, you need to know what ran last Tuesday, or you need to reprocess history.

Chapter 1’s list. Ten independent cron jobs do not need Airflow. Ten dependent ones do, and the transition is usually one incident where a job ran on incomplete data.

The decision

   Independent scheduled jobs, no dependencies,
   fewer than ~10?
     → cron, or your cloud's scheduler. Seriously.

   Batch data pipelines with dependencies, retries,
   backfills, and a big ecosystem requirement?
     → AIRFLOW

   Starting fresh, care about lineage, testing and
   local development?
     → Dagster

   Genuine loops or runtime-determined structure?
     → Prefect

   Long-lived business processes with durable state
   and compensations?
     → Temporal

   Everything is a container, Kubernetes-only,
   no data ecosystem needed?
     → Argo Workflows

   Transformations inside a warehouse?
     → dbt — with Airflow around it

   Sub-minute or event-driven at high volume?
     → Kafka and a stream processor, not any of these

Where Airflow is used and should not be

Four common misuses:

As a data processing engine. Chapter 1’s rule. Tasks loading gigabytes into worker memory. Use a warehouse, Spark, or a pod with real resources — and note that Spark’s own chapter 15 is equally blunt in the other direction: under about a hundred gigabytes, DuckDB on one machine beats a cluster comfortably.

As a low-latency job queue. “Trigger this DAG on every user signup” at thousands per minute. Scheduling latency is seconds and each run is database rows. Use RabbitMQ or a task queue.

As a CI/CD system. It runs commands in order, so people build deployment pipelines in it. It has no artefact management, no environment promotion, no approvals. Use a CI system and Argo CD.

As a microservice orchestrator. Coordinating a request across services with retries and compensation. That is Temporal, and the mismatch shows up as a DAG per request and a metadata database full of runs.

The argument in one paragraph

Airflow is a decade-old scheduler with a design that shows its age in data passing, local development and its interval-centric model — and it remains the right answer for a great many teams, because the ecosystem is enormous, backfills genuinely work, everyone knows it, and the boring middle it occupies is where most data work actually lives. Its newer competitors are better along the axes they were built to be better along, and choosing one means trading an ecosystem for an abstraction. If you already run Airflow, the honest advice is usually to run it well — upgrade to 3, use deferrable operators, keep the database small, stop passing data through XComs — rather than to migrate. If you are starting from nothing and the ecosystem is not your constraint, look properly at Dagster before defaulting.

What to take from this chapter

• Good at: scheduled batch work with dependencies, an enormous ecosystem, first-class backfills, and being universally known.
• Its age shows in: weak data passing, poor local development, confusing interval semantics, installation-wide dependencies, awkward testing, and operational weight.
• Dagster is the substantive competitor — assets rather than tasks, typed data passing, real local development and testing. Airflow’s own Assets are a partial adoption of the same idea.
• Prefect wins when the workflow has genuine loops or runtime-determined structure.
• Temporal is a different category — durable business processes, not batch data.
• Argo Workflows is container-native and Kubernetes-only, with no data ecosystem.
• dbt is a complement, not a competitor — Airflow orchestrates around it.
• Ten independent cron jobs do not need Airflow. Ten dependent ones do.
• Do not use Airflow as a processing engine, a low-latency queue, a CI/CD system, or a microservice orchestrator.
• If you already run it, run it well rather than migrating. If starting fresh and the ecosystem is not your constraint, look at Dagster first.

Next: the cheat sheet — commands, settings, states, metrics and procedures on one page.