Installing & Running Airflow with Docker Compose

The fastest reliable way to get a full Airflow environment running locally — scheduler, webserver, metadata database, and a worker, all wired together — is the official Docker Compose file the Airflow project publishes for every release. This avoids the dependency headaches of a pip install apache-airflow (which has notoriously strict, version-pinned requirements) for local development and evaluation.

Step 1: Fetch the Compose File

curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.9.0/docker-compose.yaml'

Always pin the version in the URL to match the Airflow release you intend to run — pulling latest gives inconsistent behavior across machines and over time as new versions ship.

Step 2: Set Required Environment Variables

mkdir -p ./dags ./logs ./plugins ./config
echo -e "AIRFLOW_UID=$(id -u)" > .env

On Linux, AIRFLOW_UID matters because the containers write files (logs, in particular) as that UID on your host’s bind-mounted volumes — skip this and you’ll get permission-denied errors the first time a task tries to write a log file. macOS and Windows Docker Desktop users are less affected due to how their filesystem virtualization handles permissions, but setting it is still the documented default.

Step 3: Initialize the Database

docker compose up airflow-init

This one-time step runs database migrations against the metadata database and creates the default airflow/airflow admin user. It should complete and exit cleanly — if it doesn’t, the compose file’s depends_on health checks for Postgres haven’t been given enough time to pass, which is the most common first-run failure.

Step 4: Start Everything

docker compose up -d

This brings up the webserver, scheduler, triggerer, and a Celery worker (the default Compose file uses CeleryExecutor plus Redis as the broker), all backed by a Postgres container. Give it 30–60 seconds on first boot for every service’s health check to pass.

docker compose ps

Every service should show healthy, not just running — a container that’s running but failing its health check is still initializing (or stuck).

Step 5: Access the UI

Navigate to http://localhost:8080 and log in with airflow / airflow (the default credentials created by airflow-init — change these before anything touches real data). You should see the example DAGs Airflow ships with, which are a good way to confirm the scheduler is actually picking things up before you write your own.

Adding Your Own DAGs

Any .py file placed in the ./dags directory you created in Step 2 is automatically picked up by the scheduler within its parse interval (a few minutes by default) — no restart required.

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG("hello_world", start_date=datetime(2026, 1, 1), schedule="@daily", catchup=False) as dag:
    BashOperator(task_id="say_hello", bash_command="echo Hello from Airflow")

If it doesn’t appear in the UI within a few minutes, check docker compose logs airflow-scheduler for parse errors — a syntax error or a missing import in your DAG file will show up there, and often in the UI’s “DAG Import Errors” banner too.

Common Mistakes

Forgetting AIRFLOW_UID on Linux and hitting permission-denied errors on log writes — set it before the first docker compose up, not after.

Using the default Compose setup as a production reference. It’s explicitly documented as intended for local development/testing — production deployments need external, properly-backed-up Postgres, secrets management, and typically run on Kubernetes rather than plain Docker Compose (see Deploying Airflow on Kubernetes).

Not tearing down volumes between experiments. docker compose down alone leaves the Postgres volume intact (which is usually what you want), but if you need a truly clean slate — e.g., after intentionally corrupting local state while testing — use docker compose down --volumes.

Frequently Asked Questions

Can I run just the scheduler and skip Celery/Redis for simple local testing? Yes — override the executor to LocalExecutor in a custom compose override file and drop the worker/redis services; this is lighter weight for pure DAG-authoring iteration.

How much memory does this need? The official docs recommend at least 4GB allocated to Docker, and note that lower memory settings can cause containers to be killed mid-startup — a common cause of mysterious webserver crashes on first run.

Do I need to rebuild the image to add a Python package? Yes — the default image doesn’t include arbitrary PyPI packages; you extend it with a small Dockerfile (FROM apache/airflow:2.9.0 + RUN pip install ...) and point Compose at your built image instead of pulling the stock one.

Summary

The official Docker Compose file is the fastest path from zero to a working multi-component Airflow environment, and it mirrors the real architecture (scheduler, webserver, executor, metadata database) closely enough to be a legitimate learning environment — not just a toy. Get AIRFLOW_UID right, wait for health checks, and treat it as a development tool rather than a production deployment pattern.