Airflow Executors: Sequential, Local, Celery & Kubernetes
The executor is the component that decides how a queued task actually gets run — as a subprocess on the scheduler’s own machine, dispatched to a pool of Celery workers, or launched as its own dedicated Kubernetes pod. The executor is a single configuration setting (executor in airflow.cfg or the AIRFLOW__CORE__EXECUTOR environment variable), but the choice has major implications for scalability, isolation, and operational complexity.
SequentialExecutor
Runs exactly one task at a time, with no parallelism whatsoever. It’s the default executor when Airflow is set up with SQLite (which can’t handle concurrent connections from multiple processes), and exists purely for the most minimal local testing — it should never appear in any deployment beyond a first “does this even work” check.
LocalExecutor
Runs tasks as parallel subprocesses on the same machine as the scheduler, using a process pool bounded by parallelism/max_active_tasks_per_dag settings. This is a legitimate production choice for smaller deployments — no separate message broker or worker fleet to operate — but it’s fundamentally limited by a single machine’s CPU and memory, and if that machine goes down, everything goes down with it.
# airflow.cfg[core]executor = LocalExecutor
[database]sql_alchemy_conn = postgresql+psycopg2://airflow:airflow@postgres/airflowLocalExecutor requires Postgres or MySQL (not SQLite) since it needs genuine concurrent database access from multiple subprocesses.
CeleryExecutor
Distributes tasks across a pool of Celery worker processes, which can run on separate machines from the scheduler, coordinated through a message broker (Redis or RabbitMQ). This is the executor the official Docker Compose file from Installing Airflow with Docker Compose uses by default, and it’s a common production choice for teams that want horizontal scalability without the operational complexity of Kubernetes.
[core]executor = CeleryExecutor
[celery]broker_url = redis://redis:6379/0result_backend = db+postgresql://airflow:airflow@postgres/airflowThe tradeoff versus KubernetesExecutor: all workers in a Celery pool typically share the same environment/dependencies (unless you run multiple differently-configured worker pools with queues), so per-task dependency isolation is weaker.
KubernetesExecutor
Launches a brand-new, isolated Kubernetes pod for every single task instance, then tears it down once the task completes. This gives maximum isolation (each task can theoretically use a completely different container image with different dependencies) and elastic scaling (no idle worker capacity sitting around — pods only exist while tasks are actually running), at the cost of per-task pod startup latency and genuinely more operational complexity to run correctly. Covered in depth in Deploying Airflow on Kubernetes.
CeleryKubernetesExecutor
A hybrid: most tasks run via Celery workers as usual, but individual tasks can be routed to run as their own Kubernetes pod by assigning them to a special kubernetes queue — useful when only a handful of tasks need pod-level isolation (e.g., one task with unusual resource requirements) while the rest of the DAG runs efficiently on the shared Celery pool.
Choosing an Executor
| Executor | Scales beyond one machine | Operational complexity | Best for |
|---|---|---|---|
| Sequential | No | None | Local smoke-testing only, never production |
| Local | No | Low | Small deployments, a handful of DAGs |
| Celery | Yes | Medium | Most mid-to-large production deployments |
| Kubernetes | Yes | High | Heterogeneous workloads needing per-task isolation, teams already running Kubernetes |
| CeleryKubernetes | Yes | High | Mostly-uniform workloads with a few outlier tasks needing isolation |
Common Mistakes
Running SequentialExecutor (the SQLite-backed default) in anything resembling production because nobody explicitly changed it — this is a common “how did we not notice this” discovery in poorly-onboarded Airflow deployments.
Choosing KubernetesExecutor by default without a genuine isolation or elastic-scaling need, absorbing its operational complexity for workloads that would run just fine, more simply, on CeleryExecutor.
Undersizing the Celery worker pool or broker and then attributing the resulting task queueing delays to “Airflow being slow,” when the actual bottleneck is available worker capacity relative to task volume.
Frequently Asked Questions
Can I switch executors on an existing deployment? Yes, though it typically requires infrastructure changes (standing up Celery workers and a broker, or a Kubernetes cluster) alongside the config change — it’s not purely a one-line settings swap in practice.
Does the executor choice affect DAG-writing code? Generally no — the same DAG code runs unmodified regardless of executor; the executor is purely a runtime/infrastructure concern, which is one of Airflow’s genuinely nice separations of concerns.
Is CeleryExecutor still relevant now that KubernetesExecutor exists? Yes — plenty of production deployments deliberately choose Celery specifically to avoid Kubernetes’ operational overhead when per-task pod isolation isn’t a real requirement.
Summary
The executor determines how tasks run, entirely independent of what the DAG does — LocalExecutor for small single-machine deployments, CeleryExecutor for most mid-to-large production setups wanting horizontal scale without Kubernetes, and KubernetesExecutor when per-task isolation and elastic pod-based scaling genuinely matter. Match the choice to your actual scale and isolation needs rather than defaulting to whichever sounds most sophisticated.