4. Scheduling and the data interval

This is the chapter that Airflow is famous for confusing people with. The confusion is real, it is historical, and once you see the underlying idea it stops being confusing permanently.

The idea: a run covers a period, it does not happen at a moment

Cron thinks in moments: at 02:00, run this. Airflow thinks in intervals: this run is responsible for the data between two timestamps.

   a DAILY DAG, schedule "0 2 * * *"

   data interval:  |◄────── Aug 29 00:00 → Aug 30 00:00 ──────►|
                                                                │
   the run STARTS: ────────────────────────────────────────────►│ Aug 30 02:00
                                                                  (after the
                                                                   interval ENDS)

A run for a period starts after that period is over. You cannot summarise Tuesday until Tuesday has finished.

That single sentence explains every piece of ds confusion that has ever happened.

Why ds is “yesterday”

   it is 2026-08-30 02:00. Your DAG has just started.

   {{ ds }}                  = 2026-08-29    ← "yesterday"?!
   {{ data_interval_start }} = 2026-08-29 00:00
   {{ data_interval_end }}   = 2026-08-30 00:00
   {{ logical_date }}        = 2026-08-29 00:00
   actual wall clock         = 2026-08-30 02:00

ds is not “today”. ds is the start of the interval this run covers, which for a daily DAG is the day whose data you are processing. That is correct and useful — and the name gives no hint of it, which is why everyone gets caught once.

The historical baggage makes it worse. The field used to be called execution_date, which sounded exactly like “when this executed” and meant the opposite. It was renamed logical_date in 2.2 and removed entirely in Airflow 3, replaced by explicit data_interval_start / data_interval_end plus run_after.

The practical rule:

   filtering data by the period you are processing?
     → use data_interval_start and data_interval_end

   naming an output file or partition for the period?
     → use ds  (== data_interval_start's date)

   wanting "right now"?
     → you almost certainly do not, but it is
       {{ ts }} at render time, or datetime.now()
       inside a task

-- the query nearly every Airflow task should look like
SELECT * FROM events
WHERE created_at >= '{{ data_interval_start }}'
  AND created_at <  '{{ data_interval_end }}'

Half-open interval: inclusive start, exclusive end. Every row lands in exactly one run.

Schedules

schedule="0 2 * * *"                        # cron
schedule="@daily"                           # preset
schedule=timedelta(hours=6)                 # relative to the previous run
schedule=None                               # manual / triggered only
schedule=[my_asset]                         # asset-driven (Airflow 3)
schedule=MyCustomTimetable()                # full control

Presets: @once, @hourly, @daily, @weekly, @monthly, @yearly, @continuous.

Cron and timedelta differ in a way that matters. A cron schedule is absolute — 02:00 daily, whatever happened yesterday. A timedelta is relative to the previous run’s start, so a run that starts late pushes the next one later, and the schedule drifts. Cron is usually what you want.

catchup — the setting that has bitten everyone

@dag(start_date=pendulum.datetime(2024, 1, 1, tz="UTC"),
     schedule="@daily",
     catchup=True)          # ← in Airflow 2 this was the DEFAULT

Deploy that in 2026 and Airflow creates every missed run since January 2024 — over seven hundred DAG runs, immediately. It will attempt them as fast as concurrency limits allow, and it will flatten whatever those tasks talk to.

This has happened to nearly everyone, usually on the first day.

Airflow 3 changed the default to catchup=False, which is one of the better decisions in the release. On Airflow 2, set it explicitly on every DAG.

catchup=False    # only run for intervals since deployment

When you genuinely want history, do it deliberately and with limits:

@dag(catchup=True, max_active_runs=3)      # ← never omit this

max_active_runs bounds how many runs execute concurrently. A catchup without it is a self-inflicted denial of service on your own database or API.

Backfill

Running historical intervals on purpose:

airflow backfill create --dag-id daily_sales \
  --from-date 2026-08-01 --to-date 2026-08-30

In Airflow 2 this ran in the CLI process — if your terminal died, the backfill died. In Airflow 3 backfills are managed by the scheduler, so they are durable, visible in the UI, and pausable. That is a genuine improvement and another good reason to upgrade.

Before any backfill, three questions:

Are your tasks idempotent? Running the same interval twice must produce the same result, not double it. This is the property that makes everything else in Airflow safe, and chapter 8 returns to it.

What will the concurrency do to downstream systems? Thirty days of a DAG that queries production at full parallelism is thirty simultaneous queries.

Is the DAG’s code still compatible with old data? A backfill runs today’s code against last year’s data. Schema changes make this interesting.

Timetables

When cron cannot express it — trading days, “the last business day of the month”, “every weekday but skip holidays” — write a Timetable:

from airflow.timetables.base import Timetable, DagRunInfo, DataInterval

class BusinessDays(Timetable):
    def next_dagrun_info(self, *, last_automated_data_interval,
                         restriction, **kwargs):
        # compute the next interval, skipping weekends and holidays
        start = next_business_day(last_automated_data_interval.end)
        return DagRunInfo.interval(start=start, end=start + timedelta(days=1))

    def infer_manual_data_interval(self, *, run_after):
        return DataInterval(start=run_after - timedelta(days=1), end=run_after)

Simpler built-ins cover common cases without writing one:

from airflow.timetables.trigger import CronTriggerTimetable

# run at 02:00 but the interval is the previous 6 hours, not the previous day
schedule=CronTriggerTimetable("0 2 * * *", timezone="UTC",
                              interval=timedelta(hours=6))

CronTriggerTimetable is worth knowing because it decouples when it runs from what period it covers — which is exactly the thing plain cron scheduling conflates.

Asset-driven scheduling

Instead of a clock, run when data is ready.

from airflow.sdk import Asset

orders = Asset("s3://warehouse/orders/")

@dag(schedule="@daily", start_date=...)
def produce():
    @task(outlets=[orders])
    def write_orders():
        ...

@dag(schedule=[orders], start_date=...)      # ← no clock at all
def consume():
    @task
    def read_orders():
        ...

When write_orders succeeds, Airflow marks the asset updated and triggers every DAG scheduled on it. Multiple assets means “wait for all of them”.

These were called Datasets in Airflow 2.4–2.10 and renamed to Assets in Airflow 3, with a richer model (@asset, watchers, external events). The concept is the same.

This is genuinely better than the alternatives it replaces. The traditional ways to say “run B after A” were an ExternalTaskSensor — which polls and is fragile about exact timestamps — or a TriggerDagRunOperator, which puts knowledge of the consumer inside the producer. Assets invert the dependency: the consumer declares what it needs, and the producer does not know who is listening. If you have read the Kafka guide, it is the same decoupling argument, at a much coarser grain.

Two limits worth knowing: the asset is a name, not the data — Airflow does not check that anything was written — and a failed producer task does not update the asset, which is the behaviour you want.

Timezones

import pendulum
start_date = pendulum.datetime(2026, 1, 1, tz="Europe/London")

Always use a timezone-aware start_date. A naive datetime is interpreted as UTC, which is fine until it is not.

Airflow stores everything in UTC internally and converts for display. A cron schedule in a DST-observing timezone does what you would hope — 0 2 * * * in Europe/London runs at 02:00 local both sides of the clock change — but the interval length varies, which matters if you are counting hours. Where correctness matters more than local convention, schedule in UTC and convert at the edges.

The scheduler’s decision, precisely

Putting it together — what has to be true for a run to be created:

   1. the DAG is not paused
   2. now >= the end of the next data interval
        (the interval must be OVER)
   3. the interval start is >= start_date
   4. end_date, if set, has not passed
   5. active runs < max_active_runs
   6. with catchup=False, only the MOST RECENT
      missed interval is created, not all of them

Then, for a task within that run:

   7. all upstream dependencies satisfy its trigger rule
   8. the DAG's max_active_tasks is not exhausted
   9. the task's pool has a free slot
  10. concurrency limits on the task itself allow it

“My DAG is not running” is always one of these ten. In order of likelihood: it is paused, catchup=False and the interval is not over yet, start_date is in the future, or a pool is full. Chapter 14 turns this into a diagnosis procedure.

What to take from this chapter

• A run covers an interval and starts after that interval ends. You cannot summarise Tuesday until Tuesday is over.
• ds is the start of the interval, not today. Use data_interval_start / data_interval_end for filtering data — half-open, so every row lands in exactly one run.
• execution_date → logical_date (2.2) → removed in Airflow 3 in favour of explicit interval fields.
• Cron schedules are absolute; timedelta is relative to the previous run and drifts. Prefer cron.
• catchup defaulted to True in Airflow 2 and has ambushed nearly everyone. Set catchup=False explicitly; Airflow 3 changed the default.
• Never run a catchup or backfill without max_active_runs. It is otherwise a denial of service on your own systems.
• Backfills are scheduler-managed and durable in Airflow 3; in 2 they died with your terminal. Before backfilling: is it idempotent, what will the concurrency do, and does today’s code work on old data?
• Timetables handle what cron cannot; CronTriggerTimetable decouples when it runs from what period it covers.
• Assets (Datasets in 2.x) schedule on data readiness. The consumer declares what it needs; the producer does not know who listens. The asset is a name, not the data.
• Always use a timezone-aware start_date.
• “My DAG is not running” is one of ten conditions — most often it is paused.

Next: a DAG run traced end to end, from the scheduler loop to a task’s final state.