Deploying Airflow on Kubernetes

Running Airflow on Kubernetes is the standard choice for teams that already operate Kubernetes infrastructure and want Airflow’s own components — scheduler, webserver, workers — to benefit from the same scheduling, self-healing, and resource management the rest of their platform gets. There are two distinct things “Airflow on Kubernetes” can mean, and conflating them is a common source of confusion.

Two Separate Concepts

Deploying Airflow’s own components on Kubernetes — the scheduler, webserver, and triggerer run as Kubernetes Deployments/pods, typically installed via the official Helm chart. This is an infrastructure/deployment decision and is largely independent of which executor you choose.

Using KubernetesExecutor — a specific executor choice (covered in Airflow Executors) where each individual task gets its own dedicated pod, launched and torn down by the scheduler. You can run Airflow’s components on Kubernetes while still using CeleryExecutor for task execution, and conversely (less commonly) run KubernetesExecutor from a non-Kubernetes-hosted scheduler pointed at an external cluster.

The Official Helm Chart

helm repo add apache-airflow https://airflow.apache.org
helm repo update

helm install airflow apache-airflow/airflow \
  --namespace airflow \
  --create-namespace \
  --values values.yaml

A minimal values.yaml selecting KubernetesExecutor and pointing at an external Postgres (production deployments should not rely on the chart’s bundled Postgres, which is intended for evaluation):

executor: KubernetesExecutor

data:
  metadataConnection:
    host: my-external-postgres.internal
    db: airflow
    user: airflow
    pass: <from-a-secret-not-here>

dags:
  gitSync:
    enabled: true
    repo: https://github.com/your-org/airflow-dags.git
    branch: main

gitSync is the standard way DAG files reach a Kubernetes-deployed Airflow instance — a sidecar container continuously pulls the latest DAG files from a git repo into a shared volume the scheduler and workers can read, avoiding the need to rebuild container images every time a DAG changes.

KubernetesPodOperator: Per-Task Container Isolation

Independent of the executor choice, KubernetesPodOperator lets an individual task run as an arbitrary container — useful for running a task that needs a completely different runtime or dependency set than the rest of your Airflow environment, without polluting the main Airflow image:

from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator

run_ml_training = KubernetesPodOperator(
    task_id="train_model",
    name="train-model-pod",
    namespace="airflow",
    image="my-registry/ml-training:v3",
    cmds=["python", "train.py"],
    get_logs=True,
)

This works with any executor — it’s the task’s own execution environment that’s containerized, distinct from whether the scheduling of that task happens via Celery or KubernetesExecutor.

Resource Requests and Limits

A genuinely important production concern on Kubernetes that doesn’t exist the same way on LocalExecutor/CeleryExecutor: every task pod should have explicit CPU/memory requests and limits, both to let the Kubernetes scheduler pack nodes efficiently and to prevent one runaway task from starving others on the same node.

from kubernetes.client import models as k8s

run_ml_training = KubernetesPodOperator(
    task_id="train_model",
    ...,
    container_resources=k8s.V1ResourceRequirements(
        requests={"cpu": "2", "memory": "4Gi"},
        limits={"cpu": "4", "memory": "8Gi"},
    ),
)

Common Mistakes

Relying on the Helm chart’s bundled Postgres/Redis in production. These are convenience defaults for trying the chart out — production deployments need externally managed, backed-up databases with their own availability guarantees.

Not setting resource requests/limits on KubernetesPodOperator tasks, leading to noisy-neighbor problems where one memory-hungry task destabilizes unrelated pods on the same node.

Underestimating pod startup latency with KubernetesExecutor for workloads with many short, frequent tasks — pulling images and scheduling a fresh pod per task adds real overhead that LocalExecutor/CeleryExecutor simply don’t have, and can make a DAG with hundreds of small tasks noticeably slower end-to-end.

Frequently Asked Questions

Do I need KubernetesExecutor if I’m already running Airflow on Kubernetes? No — plenty of Kubernetes-hosted Airflow deployments use CeleryExecutor for task execution while still benefiting from Kubernetes for the scheduler/webserver/worker deployments themselves.

How does logging work with ephemeral task pods? KubernetesExecutor and KubernetesPodOperator both need a remote logging backend (S3, GCS, Elasticsearch) configured — once a task pod terminates, its local logs are gone unless they were shipped somewhere durable first.

Is the Helm chart the only way to deploy on Kubernetes? No — it’s the officially maintained, most common path, but hand-rolled manifests or other packaging (including what managed services like Cloud Composer build internally) are viable alternatives with more control and more maintenance burden.

Summary

“Airflow on Kubernetes” bundles two separate decisions — where Airflow’s own components run, and which executor handles task execution — and the official Helm chart with KubernetesExecutor is the standard combination for teams wanting per-task isolation and elastic scaling. KubernetesPodOperator adds a third, orthogonal capability: running individual tasks in arbitrary containers regardless of executor. Get resource limits and remote logging right from the start — both are easy to skip initially and expensive to retrofit.