13. Tuning and operations

Airflow has a lot of settings and about a dozen that matter. This chapter is ordered by how often each one is the answer.

The concurrency knobs

They form a hierarchy, and the most restrictive wins. Almost every “my tasks are stuck in scheduled” question is one of these.

   ┌─────────────────────────────────────────────────┐
   │ parallelism = 32                                │
   │   every running task, across the whole install  │
   │                                                 │
   │  ┌───────────────────────────────────────────┐  │
   │  │ max_active_tasks_per_dag = 16             │  │
   │  │   running tasks in ONE DAG                │  │
   │  │                                           │  │
   │  │  ┌─────────────────────────────────────┐  │  │
   │  │  │ pool slots                          │  │  │
   │  │  │   a named budget across DAGs        │  │  │
   │  │  │                                     │  │  │
   │  │  │  ┌───────────────────────────────┐  │  │  │
   │  │  │  │ max_active_tis_per_dag        │  │  │  │
   │  │  │  │   ONE task across all runs    │  │  │  │
   │  │  │  └───────────────────────────────┘  │  │  │
   │  │  └─────────────────────────────────────┘  │  │
   │  └───────────────────────────────────────────┘  │
   └─────────────────────────────────────────────────┘

   plus:  max_active_runs_per_dag = 16
            concurrent RUNS of one DAG

Setting	Scope	Default	Set it to
`parallelism`	whole installation	32	total worker capacity
`max_active_tasks_per_dag`	one DAG	16	stop one DAG monopolising
`max_active_runs_per_dag`	one DAG	16	1 for anything sequential
`max_active_tis_per_dag`	one task, all runs	none	rate-limit one task
`worker_concurrency`	one Celery worker	16	cores × 2 for I/O-bound
pool `slots`	a named resource	—	protect external systems

Two of these deserve attention.

parallelism must not exceed your actual worker capacity. Setting it to 256 with 32 worker slots means 224 tasks sit in queued — which looks broken and is just a number being wrong.

max_active_runs=1 is the correct setting for any DAG whose runs must not overlap. Without it, a slow run and the next scheduled run execute simultaneously against the same tables, which is how people discover their pipeline was never safe to run concurrently.

Pools

A named budget, usually protecting something outside Airflow.

airflow pools set warehouse 5 "concurrent warehouse queries"
airflow pools set legacy_api 2 "the old API falls over above 2"

@task(pool="warehouse", pool_slots=1, priority_weight=10)
def heavy_query():
    ...

Pools are the right tool for “this external system cannot take more than N of these at once”. A warehouse with limited concurrency, an API with a rate limit, a database with a connection budget.

pool_slots lets one expensive task consume several:

@task(pool="warehouse", pool_slots=3)     # counts as three
def enormous_query(): ...

Everything with no pool goes in default_pool, which defaults to 128 slots. That is effectively a second global parallelism limit, and people occasionally hit it without realising it exists.

priority_weight orders tasks competing for the same pool — higher first. It only matters when there is contention, and by default it accumulates downstream, so a task deep in a DAG has higher weight than one at the top. weight_rule="absolute" turns that off if it surprises you.

Scheduler tuning

[scheduler]
max_tis_per_query = 512          # task instances examined per loop query
scheduler_idle_sleep_time = 1    # pause when there is nothing to do
min_file_process_interval = 30   # DAG re-parse interval
parsing_processes = 2            # parallel DAG parsing
scheduler_heartbeat_sec = 5

min_file_process_interval is the one to think about. Lower means changes appear faster and parsing load multiplies; 30 is a good production value, 5 is pleasant in development.

Run 2–3 schedulers for availability. More than that rarely helps, because they contend for the same database rows — chapter 10’s point that the bottleneck is the database, not scheduler CPU.

The metric that tells you whether scheduling is healthy:

   airflow_scheduler_scheduler_loop_duration
     < 1 s     healthy
     1–5 s     tasks wait that long between transitions
     > 10 s    the installation feels broken

If it is high and scheduler CPU is low, the database is the problem and adding schedulers makes it worse.

Getting DAGs to workers

Every scheduler, worker and triggerer needs identical DAG code. Three approaches:

Approach	How	Verdict
Baked into the image	DAGs in the container image; deploy = new image	best — immutable, versioned, atomic
git-sync	a sidecar polls a repository into a shared volume	common; a window of inconsistency during sync
Shared volume	NFS/EFS mounted everywhere	avoid — slow parsing, a single point of failure

Baking DAGs into the image is the right default. It makes the deployment atomic, ties DAG code to its dependencies, and means a rollback is an image tag. It also removes the failure where the scheduler has a new DAG and a worker does not.

git-sync is popular because it decouples DAG changes from image builds, which teams like. The cost is a genuine window where components disagree, and a task that starts on a worker mid-sync can fail with an import error that never reproduces.

The shared-volume option is worth naming as a trap: DAG parsing does a lot of small file reads, and NFS makes that slow enough to affect scheduling. It also makes one filesystem a dependency of everything.

Logging

By default each task writes a log file on whichever worker ran it. That machine may not exist tomorrow, which is why remote logging is not optional in production:

[logging]
remote_logging = True
remote_base_log_folder = s3://acme-airflow-logs/
remote_log_conn_id = aws_logs
delete_local_logs = True

Supported: S3, GCS, Azure Blob, Elasticsearch, OpenSearch, Stackdriver.

The behaviour worth knowing: logs are uploaded when the task finishes. While a task is running, the UI streams from the worker directly; if the worker dies mid-task, those logs may be lost entirely. That is the single most annoying gap when debugging an OOM — the task that would tell you what happened is the one whose logs did not survive.

Mitigations: ship logs from the container with a sidecar or a node agent as well, or send to Elasticsearch which supports live streaming.

Set a retention policy on the log bucket. Airflow does not manage it, and a busy installation produces a lot of small objects.

Monitoring

Airflow emits StatsD metrics; the usual path is the StatsD exporter into Prometheus.

[metrics]
statsd_on = True
statsd_host = statsd-exporter
statsd_port = 9125
statsd_prefix = airflow

The ones that matter:

   airflow_scheduler_scheduler_loop_duration    ← scheduling health
   airflow_dagrun_schedule_delay                ← runs starting late
   airflow_executor_queued_tasks                ← rising = no capacity
   airflow_executor_running_tasks
   airflow_dag_processing_total_parse_time      ← DAG file cost
   airflow_dag_processing_import_errors         ← should be ZERO
   airflow_pool_starving_tasks                  ← a pool is a bottleneck
   airflow_triggerer_heartbeat
   airflow_ti_failures / airflow_ti_successes

Two deserve alerts on their own:

import_errors > 0 means a DAG is not in Airflow at all. Silent absence is worse than a red task, and this is the only signal.

airflow_pool_starving_tasks tells you a pool is the constraint — which is otherwise invisible, since starved tasks just sit in scheduled looking normal.

Alerting on data, not tasks

The position this chapter builds to.

default_args = {
    "on_failure_callback": alert_slack,
    "on_retry_callback": None,          # do not alert on retries
}

Task failure alerts are necessary and insufficient. A DAG can succeed and be wrong: an extract that returned zero rows because an upstream API changed, a job that processed yesterday’s file again, a transform that silently dropped a column. Every task is green.

So add tasks that check the data:

from airflow.providers.common.sql.operators.sql import SQLCheckOperator

SQLCheckOperator(
    task_id="row_count_sane",
    conn_id="warehouse",
    sql="""
      SELECT COUNT(*) > 1000
      FROM daily_sales WHERE day = '{{ ds }}'
    """,
)

SQLCheckOperator, SQLValueCheckOperator and SQLIntervalCheckOperator cover most of it; Great Expectations or dbt tests do more.

A pipeline without data checks has task monitoring, not data monitoring, and the failures that actually hurt — quietly wrong numbers on a dashboard for three weeks — are invisible to the first and caught by the second.

Airflow 2’s SLAs were the built-in attempt at “this should have finished by now” and were unreliable enough that Airflow 3 removed them. Until deadline alerting lands, the practical substitute is an external check: a Prometheus alert on dagrun_schedule_delay, or a separate DAG that verifies yesterday’s outputs exist.

Upgrades

   1. read the release notes — Airflow's are thorough
   2. run `airflow db check-migrations` and the upgrade
      check tooling for your version
   3. BACK UP THE METADATA DATABASE. Migrations are
      not reversible.
   4. `airflow db migrate` — on a large database this
      can take a long time; test the duration on a copy
   5. upgrade one minor version at a time
   6. upgrade provider packages separately, pinned

Step 3 is not a formality. Schema migrations on a task_instance table with tens of millions of rows can take hours, during which Airflow is down. Running db clean before an upgrade is the cheapest way to shorten that window, and it is a good reason to keep the database small even if nothing else is hurting.

The 2 → 3 upgrade is larger than a normal minor bump: removed SubDAGs, removed SLAs, execution_date gone, the new task execution model, provider repackaging. Test it properly; the Airflow project ships tooling to find incompatible DAGs before you commit.

A production shape

   3 × scheduler        2 vCPU / 4 GB
   2 × triggerer        1 vCPU / 2 GB
   2 × API server       1 vCPU / 2 GB
   N × Celery worker    4 vCPU / 16 GB, concurrency 16
   1 × PgBouncer
   PostgreSQL           8 vCPU / 32 GB, fast SSD
   Redis                small; it is a broker, not a store
   DAGs                 baked into the image
   logs                 S3 with a lifecycle policy
   secrets              Vault or a cloud secret manager
   db clean             weekly, as a DAG

What to take from this chapter

Concurrency is a hierarchy and the most restrictive wins: parallelism → max_active_tasks_per_dag → pool slots → max_active_tis_per_dag, plus max_active_runs_per_dag.
parallelism above your real worker capacity just parks tasks in queued. max_active_runs=1 for any DAG whose runs must not overlap.
Pools protect external systems. Everything unassigned lands in default_pool (128 slots), which is a limit people forget exists.
Run 2–3 schedulers; more contend on the database. scheduler_loop_duration under 1 s is the health target, and high loop time with low CPU means the database.
Bake DAGs into the image. git-sync is common and has an inconsistency window; shared volumes make parsing slow and add a single point of failure.
Remote logging is not optional — and logs upload when a task finishes, so a worker that dies mid-task can lose exactly the logs you need.
Alert on import_errors > 0 (a DAG that is silently absent) and pool_starving_tasks (an invisible bottleneck).
Alert on data, not only on tasks. A green DAG can be wrong; SQLCheckOperator and friends are how you find out. Airflow 3 removed SLAs because they never worked reliably.
Back up before db migrate — migrations are irreversible and can take hours on a large task_instance. Run db clean first.

Next: failure modes — the DAG that never appears, the task stuck forever, the database that quietly stopped keeping up.

Written by Sepuri Sai Krishna