7. The executors

The executor decides where a queued task actually runs. It is one configuration line and it determines your operational model, your scaling story and your failure modes.

[core]
executor = CeleryExecutor

LocalExecutor

Tasks run as subprocesses on the scheduler machine.

executor = LocalExecutor
parallelism = 32

Good	Bad
no extra infrastructure — no broker, no cluster	bounded by one machine
lowest possible task startup (~0 s)	scheduler and tasks share CPU and memory
trivially simple to reason about	a task that OOMs can take the scheduler with it

Better than its reputation. A machine with 16 cores and 64 GB running a few hundred DAGs of tasks that submit work elsewhere is a completely legitimate production setup, and it removes Celery, a broker and a worker fleet from your architecture.

The honest limit is the second row: the scheduler competes with your tasks. A worker process consuming 30 GB starves the scheduler’s heartbeat, and the failure looks like Airflow being broken rather than a task being greedy.

Use it when: one machine is enough, tasks are light, and you value simplicity. Which is more often than people assume.

CeleryExecutor

A fixed pool of worker processes pulling from a broker.

executor = CeleryExecutor
broker_url = redis://redis:6379/0
result_backend = db+postgresql://airflow:pw@postgres/airflow
worker_concurrency = 16

   scheduler ──► Redis/RabbitMQ ──► worker 1 (16 slots)
                                ──► worker 2 (16 slots)
                                ──► worker 3 (16 slots)

Good	Bad
fast task startup (10–100 ms)	a broker to run and monitor
scales by adding workers	workers are always running — you pay for idle
workers can be specialised by queue	one image for all tasks — the dependency problem
mature, very widely deployed	a task can OOM its worker and take siblings with it

worker_concurrency is how many tasks one worker process runs at once. The rough guide: CPU cores × 2 for I/O-bound tasks, ≈ cores for CPU-bound. Too high and workers thrash; too low and you waste machines.

Queues

Celery’s genuinely useful feature. Route tasks to specific worker pools:

@task(queue="gpu")
def train():
    ...

@task(queue="default")
def report():
    ...

airflow celery worker --queues gpu          # on GPU machines
airflow celery worker --queues default      # on cheap ones

This gets you most of the benefit people expect from Kubernetes — heterogeneous hardware, isolation between workload classes — without leaving Celery. It does not solve dependency isolation; every worker still has one image.

Use it when: you need throughput, tasks start frequently, and you can live with one shared image (or use KubernetesPodOperator on top for the tasks that cannot).

KubernetesExecutor

One pod per task. The pod is created when the task starts and destroyed when it ends.

executor = KubernetesExecutor

Good	Bad
per-task resources — request exactly what it needs	1–30 s startup per task
per-task image — dependency isolation solved	pod churn is load on the Kubernetes API
no idle cost — nothing runs between tasks	needs a cluster and someone who knows it
a task cannot affect any other task	image pull time dominates short tasks

The startup cost is the whole trade and it deserves numbers:

   image cached on the node        1–3 s
   image must be pulled            10–60 s
   cluster must scale up a node    1–5 MINUTES

   a 5-second task with a 20-second startup is
   80% overhead

For long tasks this is irrelevant. For a DAG of two hundred small tasks it is the dominant cost, and people are frequently surprised that moving to Kubernetes made things slower.

Mitigations: pre-pull images onto nodes, keep images small, and use pod_template_file to standardise. Or combine executors.

Use it when: tasks are long, dependencies genuinely differ, resource needs vary widely, and you are already comfortable operating Kubernetes.

CeleryKubernetesExecutor

Both, chosen per task by queue.

executor = CeleryKubernetesExecutor
kubernetes_queue = kubernetes

@task(queue="default")     # → Celery: fast, shared image
def small_step(): ...

@task(queue="kubernetes")  # → a pod: isolated, heavy
def big_job(): ...

The pragmatic answer for a mature installation: short frequent tasks on Celery, heavy or dependency-specific tasks in pods. The cost is running both systems, which is a real operational burden and not a small one.

EdgeExecutor

New in Airflow 3, and made possible by the Task Execution API from chapter 2.

Workers run anywhere with outbound HTTPS — another datacentre, a factory floor, a laptop, a different cloud. They poll the API server; nothing needs to reach in to them, and they never touch the metadata database.

   Airflow (cloud)  ◄──── HTTPS poll ──── edge worker (on-premises)

Use it when you have work that must run somewhere Airflow is not: data that cannot leave a network, hardware in one location, a hybrid estate. It is not a general replacement for Celery.

Choosing

   One machine is enough? Tasks are light?
     → LocalExecutor. Genuinely. Stop here.

   Need throughput, tasks are short and frequent,
   one image is acceptable?
     → CeleryExecutor  (+ queues for heterogeneous hardware)

   Tasks are long, dependencies differ, you already
   run Kubernetes well?
     → KubernetesExecutor

   Both patterns, and you can afford the operations?
     → CeleryKubernetesExecutor

   Work must run outside your Airflow's network?
     → EdgeExecutor

   Airflow 3, and want dependency isolation without
   giving up fast startup?
     → CeleryExecutor + KubernetesPodOperator
       ← the underrated answer

That last option deserves emphasis. You do not need KubernetesExecutor to get per-task containers. KubernetesPodOperator on a Celery worker starts a pod for the tasks that need one, while short tasks stay fast on Celery — and you avoid the pod-per-task overhead for everything that does not benefit from it. Many large installations run exactly this.

Scaling workers

Celery scales by adding worker processes or machines. Autoscaling on queue depth is possible — KEDA is the usual tool on Kubernetes, scaling worker replicas on Redis queue length — and it works well. The floor matters: keep enough workers running to absorb your normal load without a cold start.

Kubernetes scales through the cluster autoscaler, triggered by pending pods. That means task startup includes node provisioning in the worst case, which is the 1–5 minute row in the table above. Keeping a small warm node pool for Airflow is usually worth the cost.

The queued-task problem

The most common executor failure, across all of them:

   task stuck in QUEUED

Diagnosis, in order:

Executor	Check
Celery	are workers running? airflow celery worker alive?
Celery	is the task’s queue one a worker listens to?
Celery	is the broker reachable? Redis up?
Celery	are all worker_concurrency slots taken (sensors!)
Kubernetes	kubectl get pods -n airflow — pending?
Kubernetes	pending why? Insufficient CPU/memory? No node?
Kubernetes	image pull failing? Wrong registry credentials?
any	parallelism, max_active_tasks_per_dag, pool slots

The wrong queue case is worth calling out because the symptom is silence. A task routed to queue="gpu" when no worker listens to gpu sits in queued forever with no error anywhere. Airflow does not warn you that a queue has no consumers.

There is a safety net: stalled_task_timeout (Celery) re-queues tasks that never started. It helps with lost messages; it does not help with a queue nobody listens to.

What to take from this chapter

• LocalExecutor is better than its reputation — no broker, no fleet, zero startup. The limit is that tasks compete with the scheduler on one machine.
• CeleryExecutor: fast startup (10–100 ms), scales by adding workers, needs a broker, pays for idle workers, and one image for all tasks. worker_concurrency ≈ cores × 2 for I/O-bound work.
• Celery queues give you heterogeneous hardware and workload isolation without leaving Celery — but not dependency isolation.
• KubernetesExecutor: per-task resources and images, no idle cost, but 1–30 s startup, and up to minutes if a node must be provisioned. Short tasks become mostly overhead.
• CeleryKubernetesExecutor routes per task by queue, at the cost of operating both.
• EdgeExecutor (Airflow 3) runs workers anywhere with outbound HTTPS, enabled by the Task Execution API.
• The underrated answer is Celery plus KubernetesPodOperator — dependency isolation for the tasks that need it, fast startup for everything else.
• “Stuck in queued” is an executor problem. For Celery: workers down, wrong queue, broker unreachable, or slots full of sensors. For Kubernetes: pending pods, no capacity, image pull. A task routed to a queue nobody listens to waits forever, silently.

Next: task state and dependencies — trigger rules, retries, and the idempotency requirement that everything rests on.