Airflow XComs & Variables: Passing Data Between Tasks

Tasks in Airflow run as isolated units — potentially on different machines entirely, depending on executor — so they can’t just share Python variables in memory. XCom (“cross-communication”) is the mechanism for passing small pieces of data between tasks, and Variables are Airflow’s mechanism for storing configuration shared across DAGs.

XCom: Passing Data Between Tasks

Every XCom value is a key/value pair, scoped to a specific task instance, stored in the metadata database.

def extract(**context):
    context["ti"].xcom_push(key="row_count", value=1543)

def report(**context):
    count = context["ti"].xcom_pull(key="row_count", task_ids="extract")
    print(f"Extracted {count} rows")

With the TaskFlow API (see The TaskFlow API), this is handled automatically — a @task function’s return value is pushed to XCom implicitly, and passing one task’s call result as another’s argument pulls it automatically, with no explicit xcom_push/xcom_pull calls needed.

XCom Is for Data, Not Files

This is the single most important thing to internalize about XCom: it’s stored in the metadata database (by default), which means it is not designed for large payloads. Pushing a multi-megabyte DataFrame, a large JSON blob, or file contents through XCom will bloat the database, slow down the webserver rendering task details, and in the worst case cause outright failures once a database-specific size limit is hit.

# BAD — pushes the entire DataFrame through XCom
@task
def extract():
    df = pd.read_csv("orders.csv")
    return df   # multi-MB payload lands in the metadata database

# GOOD — push a reference, not the data itself
@task
def extract():
    df = pd.read_csv("orders.csv")
    path = "/tmp/orders_processed.parquet"
    df.to_parquet(path)
    return path   # a small string; downstream tasks read the file from this path

The correct pattern for genuinely large data is to write it somewhere durable (S3, a database table, a shared filesystem) and pass only the reference — a path, a table name, an object key — through XCom.

Custom XCom Backends

For teams that do need to pass larger objects routinely, Airflow supports pluggable custom XCom backends — instead of storing values in the metadata database, a custom backend can transparently store the actual payload in S3/GCS and keep only a pointer in the database, giving you XCom’s convenient task-to-task passing semantics without the database-bloat problem. This is a deliberate architecture decision for larger deployments, not a default most small teams need immediately.

Variables: Shared Configuration

Where XCom passes data between tasks within a run, Variables store configuration shared across DAGs and runs — API endpoints, feature flags, batch sizes, environment-specific settings.

from airflow.models import Variable

batch_size = Variable.get("etl_batch_size", default_var=500)
config = Variable.get("api_config", deserialize_json=True)

Managed through Admin → Variables in the UI, the CLI (airflow variables set), or bulk-imported from a JSON file — and, like Connections, can be backed by an external secrets manager for sensitive values rather than stored directly in Airflow’s own database.

A performance gotcha: calling Variable.get() at DAG top-level (outside a task) means it executes on every scheduler parse cycle, adding a database round-trip to every single parse — exactly the same top-level-code performance trap covered in Airflow Architecture. Call Variable.get() inside task callables, not at module level, unless the value is genuinely needed to shape the DAG’s structure itself (e.g., how many mapped tasks to generate).

Common Mistakes

Passing large data through XCom instead of a reference to externally-stored data — far and away the most common Airflow anti-pattern related to this topic.

Calling Variable.get() at DAG top-level, adding unnecessary database load to every scheduler parse cycle across potentially hundreds of DAG files.

Storing secrets directly as plain Variables without enabling an external secrets backend — Airflow does mask Variable values matching common secret-like key patterns in the UI, but the underlying value is still sitting in the metadata database unless a proper secrets backend is configured.

Frequently Asked Questions

Is there a hard size limit on XCom values? It depends on the backing database (e.g., MySQL’s default TEXT/BLOB column limits), but in all cases, treat “small” (kilobytes, not megabytes) as the practical ceiling for default XCom usage.

Can multiple tasks push to the same XCom key? Yes — XCom pulls default to the most recent value for a given key/task, but you can explicitly pull by task_ids (a list) to fetch values from several specific tasks distinctly.

Are Variables encrypted at rest? Values are stored using Airflow’s Fernet encryption key by default in the metadata database, but for genuinely sensitive secrets, an external secrets backend remains the recommended approach over relying on Fernet alone.

Summary

XCom and Variables solve two different problems — passing small values between tasks within a run, versus storing configuration shared across the whole Airflow instance — and both share the same core rule: keep the actual payload small, and pass references to larger data stored elsewhere. Get this right early, because retrofitting a pipeline that’s already pushing large DataFrames through XCom is a much bigger job than designing it correctly from the start.