Airflow Catchup, Backfill & Data Intervals Explained

Airflow doesn’t just schedule future runs — by default, it will also try to run every schedule interval it missed between a DAG’s start_date and the present moment. This behavior, called catchup, is powerful for legitimate historical processing and a genuine hazard when left on by accident.

What Catchup Actually Does

with DAG(
    dag_id="daily_report",
    start_date=datetime(2025, 1, 1),   # over a year in the past
    schedule="@daily",
    # catchup defaults to True if not specified
) as dag:
    ...

The moment this DAG is unpaused, with catchup left at its default of True, Airflow queues a separate DAG run for every single day between 2025-01-01 and today — potentially hundreds of runs, all competing for scheduler and worker capacity simultaneously. If each run hits an external API or database, this can also look identical to a sudden traffic spike or accidental denial-of-service against whatever the DAG talks to.

with DAG(
    dag_id="daily_report",
    start_date=datetime(2025, 1, 1),
    schedule="@daily",
    catchup=False,   # only the most recent due interval will run automatically
) as dag:
    ...

Setting catchup=False is the right default for the overwhelming majority of DAGs — dashboards, monitoring jobs, and most operational pipelines only care about “keep processing new data going forward,” not “reconstruct every historical interval automatically.”

When You Want Catchup

Catchup is genuinely useful when a DAG’s job is explicitly to backfill or reprocess historical data — e.g., a new pipeline being deployed that needs to populate a table with the last six months of history before it starts running incrementally. In that case, leaving catchup=True (or running an explicit backfill, below) is the intended mechanism, not an accident.

Manual Backfills

Even with catchup=False set for ongoing operation, you can still trigger historical runs deliberately and explicitly via the CLI:

airflow dags backfill \
  --start-date 2026-01-01 \
  --end-date 2026-01-07 \
  daily_report

This runs exactly the specified date range, on demand, independent of the DAG’s catchup setting — the preferred way to reprocess a known historical window (e.g., after fixing a bug that corrupted a week’s worth of data) without touching the DAG’s ongoing schedule behavior at all.

Data Intervals: What a Run Actually Represents

Every DAG run has a data interval — a start and end timestamp representing the slice of data it’s responsible for, distinct from the logical date discussed in Schedule Intervals, Cron Expressions & Timetables. For a @daily schedule, the run with logical date 2026-01-05 has a data interval of [2026-01-05 00:00, 2026-01-06 00:00) — a full, closed-on-one-end 24-hour window.

def my_task(**context):
    interval_start = context["data_interval_start"]
    interval_end = context["data_interval_end"]
    # Use these — not "now" — to determine what data this run should process.

Using data_interval_start/data_interval_end (rather than datetime.now()) inside task logic is what makes backfills actually correct: a backfilled run for a date six months ago should process that date’s data, not whatever “now” happens to be when the backfill executes.

Common Mistakes

Setting an old start_date on a new DAG without also setting catchup=False, triggering an unintended flood of historical runs the moment it’s unpaused — this is the single most common Airflow production incident among teams new to the tool.

Using datetime.now() inside task logic instead of the run’s data interval, which silently breaks backfills — a backfilled run six months in the past will process today’s data instead of the historical date it was supposed to reprocess.

Running ad-hoc backfills directly against production tables without first testing on a narrow date range. A backfill over months of data that turns out to have a bug in the processing logic means redoing all of it — validate against a day or two first.

Frequently Asked Questions

Does catchup=False affect manual backfills? No — catchup only controls automatic scheduler behavior; airflow dags backfill runs the specified range regardless of the DAG’s catchup setting.

Can I limit how many catchup runs execute in parallel? Yes, via max_active_runs on the DAG — this caps concurrent DAG run instances regardless of how many are queued, preventing a catchup flood from overwhelming the executor even if catchup itself is intentional.

Is there a way to skip specific dates during a backfill? Not natively within a single backfill command — you’d typically run separate backfill commands for the date ranges you actually want, explicitly excluding the ones you don’t.

Summary

Catchup and data intervals are two sides of the same design decision: Airflow models runs as processing discrete historical intervals, not as “whatever’s happening right now,” which is what makes reliable backfills possible in the first place. catchup=False should be the default for new DAGs unless historical backfill-on-deploy is genuinely the intent, and task logic should always reference the run’s actual data interval rather than the wall-clock “now.”