Data Engineering  /  Airflow

🌬️ Apache Airflow Guide 1 of 4 18 guides · updated 2026

Workflow orchestration with Airflow — DAGs, operators, scheduling, executors, and the production practices that keep pipelines reliable.

Writing Your First DAG in Apache Airflow

A DAG file is just a Python file that, when imported, defines a DAG object and a set of tasks with dependencies between them. There’s no special DSL to learn beyond the Airflow API itself — if you can write Python, you can write a DAG.


The Minimal Structure

Every DAG needs three things: an identifier, a start date, and a schedule.

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime
with DAG(
dag_id="my_first_dag",
start_date=datetime(2026, 1, 1),
schedule="@daily",
catchup=False,
tags=["tutorial"],
) as dag:
task_1 = BashOperator(task_id="print_date", bash_command="date")
task_2 = BashOperator(task_id="sleep", bash_command="sleep 5")
task_1 >> task_2

Save It to the DAGs Folder

Drop the file into your Airflow instance’s configured dags_folder (this is ./dags in the standard Docker Compose setup from Installing Airflow with Docker Compose). The scheduler will pick it up automatically within its parse interval — no restart needed.


Verify It Loaded Correctly

Before checking the UI, validate the file parses cleanly from the command line:

Terminal window
airflow dags list | grep my_first_dag
python dags/my_first_dag.py # a clean exit with no output means no syntax/import errors

If the DAG doesn’t appear in airflow dags list, the file has a parse error the scheduler is silently swallowing into its import-errors log — check airflow dags list-import-errors or the “DAG Import Errors” banner in the UI.


Trigger a Manual Run

You don’t need to wait for the schedule to test a DAG:

Terminal window
airflow dags trigger my_first_dag

Or use the ▶ “Trigger DAG” button in the UI’s DAG list view. Watch the run progress in the Grid or Graph view — each task box changes color as it moves through queued → running → success/failed.


Adding a Third Task with Branching Structure

Dependencies aren’t limited to a straight line — a task can have multiple upstream or downstream dependencies:

extract = BashOperator(task_id="extract", bash_command="echo extracting")
validate = BashOperator(task_id="validate", bash_command="echo validating")
load = BashOperator(task_id="load", bash_command="echo loading")
notify = BashOperator(task_id="notify", bash_command="echo notifying")
extract >> validate >> load >> notify

Full control over fan-out/fan-in dependency shapes (multiple parallel branches, tasks with several upstream dependencies) is covered in Task Dependencies & the Bitshift Operators.


Common Mistakes

Reusing a task_id within the same DAG. Task IDs must be unique within a DAG (though the same task_id can be reused across different DAGs) — Airflow will raise a DuplicateTaskIdFound error at parse time.

Forgetting catchup=False on a test DAG with an old start_date. Setting start_date to a date months in the past with catchup left at its default (True) will queue up potentially hundreds of backfill runs the moment the DAG is unpaused.

Editing a DAG file and expecting instant UI updates. The scheduler needs to re-parse the file first — for local testing, airflow dags reserialize (or just waiting out the parse interval) speeds this up if changes aren’t showing.

Frequently Asked Questions

Does the DAG start running the moment I save the file? No — new DAGs are paused by default in the UI; you must toggle them on (or trigger manually) before the scheduler will queue any runs for them.

Can one Python file define multiple DAGs? Yes, though it’s generally cleaner to keep one DAG per file for readability and to avoid one file’s parse error taking down multiple DAGs’ visibility at once.

What Python version does Airflow require? Check the compatibility matrix for your specific Airflow version — recent Airflow 2.x releases support Python 3.8 through 3.12 with per-minor-version support windows that change over time.

Summary

A DAG is a Python file with a DAG object, some tasks, and dependency operators between them — there’s no framework magic beyond that. Save it to the DAGs folder, verify it parses with airflow dags list, trigger a manual run to test it, and only then trust the schedule to take over. The next few articles go deeper into the operators and dependency patterns that make real pipelines more than a straight line of three tasks.