Airflow Architecture: Scheduler, Webserver, Metadata DB & Workers

Airflow is not a single process — it’s a small distributed system made of a handful of cooperating components that all read from and write to one shared metadata database. Understanding what each piece actually does is the difference between guessing and knowing where to look when a DAG doesn’t run when expected.

The Core Components

┌─────────────┐      ┌──────────────┐      ┌─────────────┐
│  DAG files    │ ──▶  │  Scheduler    │ ──▶  │  Executor    │
│ (Python)     │      │              │      │ (runs tasks) │
└─────────────┘      └──────┬───────┘      └──────┬──────┘
                              │                       │
                              ▼                       ▼
                     ┌──────────────────────────────────┐
                     │        Metadata Database          │
                     │   (task state, DAG runs, history) │
                     └──────────────┬─────────────────────┘
                                     │
                                     ▼
                              ┌─────────────┐
                              │  Webserver   │
                              │    (UI)      │
                              └─────────────┘

Scheduler — continuously parses DAG files from the configured dags/ folder, evaluates each DAG’s schedule, checks which tasks have their dependencies satisfied, and queues them for execution via the configured executor. This is the “brain” of Airflow and the single most common source of confusion when something “should have run but didn’t.”

Executor — the mechanism that actually runs a queued task. This can be as simple as running it as a local subprocess (LocalExecutor) or as involved as dispatching it to a Celery worker pool or spinning up a dedicated Kubernetes pod per task. See Airflow Executors for the full comparison.

Metadata database — a relational database (Postgres or MySQL in any real deployment; SQLite only for local single-user testing) holding every DAG run, task instance, its state, timestamps, and configuration. Every other component reads from or writes to this database — it is the single source of truth.

Webserver — a Flask application that reads the metadata database and renders the UI: DAG graph view, Gantt charts, task logs, and the ability to trigger, clear, or mark tasks manually. The webserver does not run tasks or make scheduling decisions — it’s purely a read/write interface onto the metadata database.

Triggerer — a newer component (Airflow 2.2+) that runs an asyncio event loop to efficiently manage deferrable tasks and sensors — operators that wait on an external condition without occupying a worker slot the whole time. Covered in Sensors & Deferrable Operators.

Why the Scheduler Parses DAG Files Repeatedly

A subtlety that trips people up: the scheduler doesn’t parse your DAG file once and cache it — it re-parses every file in the DAGs folder on a configurable interval (dag_dir_list_interval, default 5 minutes for directory scanning, with a much faster per-file re-parse loop). This is why DAG top-level code (anything outside a task function) should be cheap: a DAG file that makes a network call or queries a database at import time will slow down every single scheduler parse cycle, not just once at startup.

# BAD — runs on every scheduler parse, not just at DAG execution time
import requests
config = requests.get("https://internal-api/config").json()

# GOOD — defer expensive work into a task that runs only when the DAG actually executes
from airflow.operators.python import PythonOperator

def fetch_config():
    import requests
    return requests.get("https://internal-api/config").json()

task = PythonOperator(task_id="fetch_config", python_callable=fetch_config)

Single-Node vs Distributed Deployments

For local development or small workloads, all of these components can run as processes on one machine (this is exactly what docker compose up gives you with the official Airflow Docker Compose file — see Installing Airflow with Docker Compose). For production at scale, the scheduler, webserver, triggerer, and a pool of workers typically run as separate, independently-scalable processes or containers, all pointed at the same external metadata database — this is the model used by Deploying Airflow on Kubernetes and by managed services.

Common Mistakes

Assuming the webserver controls scheduling. It doesn’t — you can stop the webserver entirely and DAGs will keep running on schedule via the scheduler. The webserver is purely observability and manual control.

Not realizing the metadata database is a hard dependency. If the metadata database is down or overloaded, nothing works — no scheduling, no UI, no task state updates. Database sizing and connection pool limits become a real production concern once you have hundreds of DAGs.

Debugging “DAG doesn’t show up in the UI” by looking at the webserver logs instead of the scheduler logs. DAG parsing errors show up in the scheduler’s logs (or the “DAG Import Errors” banner in the UI, which is sourced from the scheduler’s parsing attempts), not the webserver’s.

Frequently Asked Questions

Can I run the scheduler and webserver on different machines? Yes, as long as both can reach the same metadata database — this is standard in production deployments.

What happens if the scheduler crashes mid-run? Task instances that were already queued or running are left in that state until the scheduler restarts and reconciles against the metadata database; Airflow doesn’t lose the state, but tasks won’t progress until the scheduler is back.

Is SQLite ever appropriate for production? No — SQLite doesn’t support the concurrent connections Airflow needs beyond a single-user local setup, and it’s incompatible with any executor other than SequentialExecutor.

Summary

Airflow’s architecture separates concerns cleanly: the scheduler decides what should run, the executor runs it, the metadata database records what happened, and the webserver displays it. Every debugging question ultimately reduces to “which of these four components would know the answer” — and the scheduler’s parse cycle, in particular, is worth understanding early since it directly shapes how you should (and shouldn’t) write DAG top-level code.