What is Apache Airflow? Workflow Orchestration Explained

Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring workflows. Instead of chaining cron jobs together and hoping nothing silently fails at 3 a.m., Airflow lets you define a pipeline as code — a set of tasks with explicit dependencies — and gives you a scheduler, a web UI, retry logic, alerting, and a full execution history for free.

Airflow was created at Airbnb in 2014 to manage increasingly complex data pipelines, became an Apache Software Foundation project in 2016, and is now the de facto standard for orchestrating batch data workflows across the industry.

The Problem Airflow Solves

Before orchestration tools, most data pipelines were a pile of cron jobs:

# crontab — no visibility into whether extract_orders.sh actually succeeded
0 2 * * * /scripts/extract_orders.sh
30 2 * * * /scripts/transform_orders.sh
0 3 * * * /scripts/load_orders.sh

This works until it doesn’t: transform_orders.sh starts at 2:30 regardless of whether extract_orders.sh finished, failures are invisible unless someone checks a log file, there’s no retry logic, and reconstructing “what ran and when” after an incident means grepping through server logs across multiple machines.

Airflow replaces this with an explicit dependency graph: transform only starts after extract succeeds, failures trigger retries and alerts automatically, and every run’s status is visible in a UI with full historical logs.

DAGs: The Core Abstraction

A workflow in Airflow is a DAG — a Directed Acyclic Graph. Each node is a task, each edge is a dependency, and “acyclic” means there are no loops: data flows in one direction from upstream tasks to downstream tasks.

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG(
    dag_id="orders_pipeline",
    start_date=datetime(2026, 1, 1),
    schedule="0 2 * * *",
    catchup=False,
) as dag:
    extract = BashOperator(task_id="extract", bash_command="python extract.py")
    transform = BashOperator(task_id="transform", bash_command="python transform.py")
    load = BashOperator(task_id="load", bash_command="python load.py")

    extract >> transform >> load

That last line — extract >> transform >> load — is the entire dependency declaration. Airflow parses this Python file, builds the graph, and the scheduler takes care of running each task only once its upstream dependencies have succeeded.

What Airflow Actually Does at Runtime

At a high level, four things happen continuously:

1. The scheduler parses your DAG files and decides which tasks are due to run based on their schedule and dependency state.
2. Tasks are handed to an executor, which runs them — locally, in Celery workers, or as Kubernetes pods, depending on configuration.
3. Task state (running, success, failed, retrying) is written to a metadata database.
4. The webserver reads that metadata database to render the UI, where you can see DAG graphs, task logs, and trigger manual runs.

The deeper mechanics of each of these pieces are covered in Airflow Architecture.

When Airflow Is the Right Tool

Airflow is a strong fit for batch, schedule-driven workflows where you need visibility and reliability: nightly ETL jobs, ML training pipelines, report generation, data warehouse loads, and any process made of discrete steps with dependencies between them.

When It Isn’t

Airflow is not a streaming engine (use Kafka Streams, Flink, or Spark Structured Streaming for that), not a low-latency event processor (a task run has real scheduling overhead — think seconds to minutes, not milliseconds), and it’s arguably heavyweight for a single standalone script that doesn’t need orchestration at all. Airflow vs Prefect vs Dagster vs Cron covers how it compares to lighter-weight alternatives.

Frequently Asked Questions

Is Airflow a data processing engine? No — Airflow orchestrates when and in what order work happens, but the actual data processing typically happens in tools it calls out to (Spark, dbt, SQL, Python scripts). Airflow is the conductor, not the orchestra.

Do I need Kubernetes to run Airflow? No. Airflow runs perfectly well with the LocalExecutor on a single machine or via Docker Compose for development and smaller production workloads — Kubernetes is one deployment option among several, covered in Executors & Deployment.

Is Airflow free? Yes, Apache Airflow itself is fully open source (Apache 2.0 license). Managed offerings like Amazon MWAA, Google Cloud Composer, and Astronomer charge for hosting and operating it, not for the software itself.

Summary

Airflow turns implicit, fragile pipeline logic (a sequence of cron jobs and hope) into an explicit, observable dependency graph defined as Python code. The DAG is the unit of work, tasks are its nodes, and the scheduler-executor-metadata-database-webserver quartet is what turns that graph definition into reliable, monitored execution. The next article covers how those four pieces fit together architecturally.