9. XComs and passing data

XCom is short for “cross-communication”: a small value one task leaves for another. It is the mechanism people reach for when they want to pass data between tasks, and understanding its limits is what keeps an Airflow installation healthy.

What it is

A row in the metadata database.

   xcom table:
     dag_id | run_id | task_id | key      | value
     -------|--------|---------|----------|-----------------
     sales  | ...    | extract | return_value | {"rows": 4211}

In TaskFlow it is invisible — a return value becomes an XCom, and an argument reads one:

@task
def extract() -> dict:
    return {"rows": 4211, "path": "s3://bucket/2026-08-29/"}   # → XCom

@task
def transform(payload: dict):                                   # ← XCom
    print(payload["rows"])

transform(extract())

With classic operators it is explicit:

def _extract(**context):
    context["ti"].xcom_push(key="row_count", value=4211)

def _transform(**context):
    n = context["ti"].xcom_pull(task_ids="extract", key="row_count")

or in a template: {{ ti.xcom_pull(task_ids='extract') }}.

The size limit

This is the whole chapter.

XComs are stored in the metadata database. The value is serialised — JSON by default — into a column whose maximum size depends on the backend:

Backend	Practical limit
PostgreSQL	~1 GB technically; treat a few KB as the limit
MySQL	64 KB (BLOB)
SQLite	2 GB, irrelevant

The technical limit is not the real one. The real limit is what you are willing to put in the database that every scheduler query touches.

   a 5 MB DataFrame pickled into XCom
     × 200 DAG runs a day
     × 30 days
   = 30 GB in the metadata database

   → every scheduler query slower
   → the UI slower
   → backups enormous
   → and chapter 10's problem, arriving fast

The rule:

XComs are for pointers and small facts. Never for data.

# ✗ WRONG
@task
def extract() -> pd.DataFrame:
    return pd.read_sql("SELECT * FROM orders", conn)   # 2 million rows

# ✓ RIGHT
@task
def extract(**context) -> str:
    df = pd.read_sql(...)
    path = f"s3://bucket/staging/{context['ds']}/orders.parquet"
    df.to_parquet(path)
    return path                                        # ← 45 bytes

@task
def transform(path: str):
    df = pd.read_parquet(path)

The second version passes a reference. The data goes through object storage, which is built for it; Airflow carries a string.

This is chapter 1’s principle in its most concrete form. Airflow moves control, not data.

Good XCom values: a file path, a URI, a row count, a job id from an external system, a boolean, a small list of keys, a table name, a timestamp.

Bad XCom values: DataFrames, query results, file contents, images, anything you would not print in a log line.

Custom XCom backends

If you genuinely need to pass larger objects and want it to be transparent, replace the storage:

from airflow.models.xcom import BaseXCom
import uuid, json

class S3XComBackend(BaseXCom):
    PREFIX = "s3://airflow-xcom/"

    @staticmethod
    def serialize_value(value, **kwargs):
        key = f"{S3XComBackend.PREFIX}{uuid.uuid4()}.json"
        s3_write(key, json.dumps(value))
        return BaseXCom.serialize_value(key)     # only the KEY hits the DB

    @staticmethod
    def deserialize_value(result):
        key = BaseXCom.deserialize_value(result)
        return json.loads(s3_read(key))

[core]
xcom_backend = plugins.s3_xcom.S3XComBackend

Now return df writes to S3 and stores a key, and every task reads transparently. This is the right answer if your team keeps putting large values in XComs — it is easier to change the backend once than to review every DAG forever.

Two things it does not fix: you still pay serialisation and network cost on every push and pull, and you now own the lifecycle of those objects (nothing cleans them up automatically — set a bucket lifecycle rule).

Airflow 3 has object storage XCom support built in, which removes most of the need to write your own.

XComs with mapped tasks

Dynamic task mapping (chapter 3) uses XComs to distribute the list:

@task
def list_files() -> list[str]:
    return ["a.csv", "b.csv", "c.csv"]     # one XCom, a list

@task
def process(f: str):
    ...

process.expand(f=list_files())              # each mapped task gets one element

Collecting results back:

@task
def summarise(results: list[int]):
    return sum(results)                     # a list of every mapped XCom

summarise(process.expand(f=list_files()))

The trap: each mapped task instance writes its own XCom row. Five hundred mapped tasks is five hundred rows per run. Combined with the size point above, mapping over many items and returning something non-trivial from each is a fast way to fill the database.

multiple_outputs

@task(multiple_outputs=True)
def extract() -> dict:
    return {"path": "s3://...", "rows": 4211}

result = extract()
transform(result["path"])       # one XCom key per dict entry
count(result["rows"])

Each key becomes its own XCom, so downstream tasks depend only on what they use. Cleaner than passing a whole dict, and a type hint of -> dict sets it automatically in recent versions.

When not to use XComs at all

Configuration → Airflow Variables or Params, not XComs. XComs are per-run and ephemeral; configuration is not.

Large data → object storage, with a path in XCom.

State across DAG runs → a database of your own. XComs belong to a run.

Cross-DAG communication → an Asset (chapter 4). Reaching into another DAG’s XComs is possible and it couples two pipelines in a way that breaks quietly when either changes.

Secrets → never. XComs are plaintext in the database and visible in the UI to anyone who can see the DAG.

That last one is worth stating clearly: an XCom containing a token is a token in your database, in your backups, and on a web page. Chapter 12 covers where secrets belong.

Cleanup

XComs are deleted when their DAG run is deleted, and DAG runs are not deleted automatically. This is chapter 10’s problem, and XComs are frequently the largest table in an unmaintained metadata database.

airflow db clean --clean-before-timestamp 2026-01-01

What to take from this chapter

• An XCom is a row in the metadata database. TaskFlow return values are XComs.
• Never put data in XComs. The technical limit is not the point — the point is that everything you store there is in the database the scheduler queries constantly.
• Pass a pointer: an S3 path, a job id, a row count. Data goes through object storage; Airflow carries a string. This is “move control, not data” made concrete.
• A custom XCom backend (or Airflow 3’s built-in object storage support) makes large values transparent — the right fix if your team keeps doing it anyway. You then own the cleanup.
• Mapped tasks write one XCom row each; mapping over many items multiplies database rows.
• multiple_outputs=True splits a dict into separate XComs so downstream tasks depend only on what they use.
• Do not use XComs for configuration (Variables), cross-run state (your own database), cross-DAG signalling (Assets), or secrets — an XCom is plaintext in the database and visible in the UI.
• XComs are not cleaned up automatically. airflow db clean is chapter 10.

Next: the metadata database — the bottleneck in almost every large Airflow installation, and how to keep it from becoming one.