Airflow Logging, Monitoring & Alerting

A pipeline you can’t observe when it fails is barely better than no pipeline at all. Airflow’s logging, retry, and alerting mechanisms are what turn “a task failed at 3 a.m.” from a silent, undiscovered problem into a page someone actually receives — and remote logging in particular becomes a hard requirement, not a nice-to-have, once you move beyond LocalExecutor.

Where Task Logs Live by Default

Each task instance writes its own log file, stored locally on whatever machine actually ran it, under a structured path (dag_id/task_id/execution_date/attempt_number.log). This works fine for LocalExecutor on a single machine — the webserver can just read the file directly. It breaks down immediately for CeleryExecutor or KubernetesExecutor: the webserver has no direct filesystem access to whichever worker or ephemeral pod actually ran the task, especially once that pod has already terminated.

Remote Logging: Required for Distributed Executors

# airflow.cfg
[logging]
remote_logging = True
remote_base_log_folder = s3://my-airflow-logs/logs
remote_log_conn_id = aws_default

With remote logging configured (S3, GCS, or Elasticsearch are the common choices), every task pushes its logs to the remote store on completion, and the webserver fetches from there instead of relying on direct filesystem access — this is what makes logs actually viewable in the UI once tasks run on ephemeral Kubernetes pods or a distributed Celery worker pool. Skipping this on KubernetesExecutor specifically means losing task logs entirely the moment a pod terminates.

Retries: The First Line of Defense

from datetime import timedelta

default_args = {
    "retries": 3,
    "retry_delay": timedelta(minutes=5),
    "retry_exponential_backoff": True,
    "max_retry_delay": timedelta(minutes=30),
}

retries and retry_delay are set per-task (or via default_args applied DAG-wide) and handle the extremely common case of transient failures — a flaky network call, a momentarily unavailable API, a database connection pool briefly exhausted — without any human intervention. retry_exponential_backoff is worth enabling on anything hitting external services, so retries space themselves out rather than hammering a struggling system at a fixed interval.

SLAs: Flagging Tasks That Are Running Too Long

task = PythonOperator(
    task_id="critical_transform",
    python_callable=transform,
    sla=timedelta(hours=1),
)

An SLA doesn’t stop or fail a task — it fires an SLA-miss event (visible in the UI, and hookable into alerting) if the task hasn’t completed within the specified duration relative to the DAG’s expected schedule. This is distinct from a timeout (which actually kills the task); an SLA is purely an observability/alerting signal for “this is taking longer than it should,” useful for catching gradual performance degradation before it becomes an outright failure.

Failure Callbacks and Alerting

def notify_slack_on_failure(context):
    from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
    hook = SlackWebhookHook(slack_webhook_conn_id="slack_alerts")
    hook.send(text=f"Task {context['task_instance'].task_id} failed in {context['dag'].dag_id}")

default_args = {
    "on_failure_callback": notify_slack_on_failure,
}

on_failure_callback (and its siblings on_success_callback, on_retry_callback) fire arbitrary Python code — commonly a Slack/PagerDuty/email notification — whenever a task instance ends in that state. Setting this via default_args applies it DAG-wide without repeating the wiring on every task individually.

Metrics and External Monitoring

Beyond built-in logging and callbacks, Airflow exposes StatsD-compatible metrics (statsd_on = True in airflow.cfg) — task duration, queue depth, scheduler heartbeat health, DAG run counts — that feed into standard observability stacks (Prometheus/Grafana, Datadog) for dashboards and threshold-based alerting beyond what per-task callbacks alone provide, especially for instance-wide health signals like scheduler lag.

Common Mistakes

Not configuring remote logging before moving off LocalExecutor, then discovering task logs are simply inaccessible the first time something fails on a Celery worker or Kubernetes pod that’s already gone.

Setting retries high without retry_exponential_backoff on tasks hitting rate-limited external APIs, effectively hammering the same endpoint repeatedly at a fixed short interval and making the underlying problem worse.

Confusing SLA misses with task failures. An SLA miss means “running slower than expected,” not “failed” — treating every SLA-miss alert as a fire-drill-worthy failure leads to alert fatigue and people eventually ignoring real ones.

Frequently Asked Questions

Do retries count against SLA timing? Yes — an SLA is measured against the DAG’s expected schedule, so a task that needs several retries before succeeding is more likely to trigger an SLA miss even if each individual attempt was otherwise fast.

Can I set different alerting for different severities of failure? Yes — on_failure_callback receives the full task/DAG context, so the callback logic itself can route differently (e.g., PagerDuty for tasks tagged critical, Slack-only for everything else) based on task metadata.

Is StatsD required for basic monitoring? No — logs and failure callbacks alone cover a lot of ground; StatsD/Prometheus integration is most valuable once you need instance-wide health dashboards and trend analysis beyond individual task/DAG alerting.

Summary

Logging, retries, SLAs, and failure callbacks form Airflow’s observability stack, and remote logging specifically stops being optional the moment tasks run somewhere other than the scheduler’s own machine. Retries absorb routine transient failures automatically; SLAs and failure callbacks are what turn “silently failed at 3 a.m.” into “someone got paged” — configure both deliberately rather than discovering their absence during an actual incident.