Data Lineage: Tracing Where Data Comes From and What Breaks If You Change It

Two questions haunt every data platform. The first arrives with an angry Slack message: “the numbers on this dashboard look wrong — where does this data actually come from?” The second arrives before a deploy: “if I change this column, what breaks?” Both are the same question asked in opposite directions, and answering them from memory stops working at roughly the twentieth table. Data lineage — the recorded graph of how data flows and transforms from source to consumption — is the infrastructure that answers them at any scale.

This post covers what lineage is (and its levels of resolution), how it’s actually captured, the four use cases that justify the investment, and why lineage efforts fail when they’re attempted as documentation instead of automation.

What Lineage Is: The Dependency Graph of Your Data

Lineage records, for every data asset, its upstream inputs and downstream consumers, forming a directed graph across the whole platform:

Resolution matters enormously, and it’s the main axis along which lineage systems differ:

Table-level lineage — fct_revenue reads from stg_orders and stg_payments. Useful, and easy to capture.
Column-level lineage — fct_revenue.net_amount is computed from stg_orders.amount minus stg_payments.refund_amount. This is where the real value lives: most changes and most bugs are column-scoped, and table-level lineage over-alerts so badly (every downstream table “depends on” your change, even if it touches none of the same columns) that people learn to ignore it.
Beyond the warehouse — lineage that continues into BI (which dashboards use this column?) and out to sources (which service owns the origin table?). The dashboard hop is the one that turns lineage from an engineering tool into an organizational one, because it maps data to people affected.

A dimension often forgotten: lineage has a time axis. The graph as it exists today doesn’t explain last quarter’s numbers if the pipeline changed in between. Mature systems version their lineage; at minimum, know that “current lineage” is what you have.

How Lineage Is Captured (Ranked by Trustworthiness)

1. Parsed from transformation code — the gold standard. If your transformations are SQL (dbt, warehouse pipelines), lineage is derivable: parse every SELECT, and the reads/writes fall out — at column level, with effort. dbt’s ref() convention made table-level lineage a free, always-correct byproduct of just building models, which is quietly one of the biggest reasons for its adoption. Warehouse-native metadata (access history in Snowflake, audit logs in BigQuery) extends this to observed lineage — what queries actually read and wrote, whether or not they came from your nice tooling.

2. Emitted by pipelines. Where logic isn’t SQL (Spark jobs, Python), the job itself reports “I read X, wrote Y” via instrumentation. OpenLineage is the open standard here — emitters exist for Spark, Airflow, Flink and friends, so heterogeneous stacks can feed one graph rather than one graph per vendor.

3. Declared by humans — the fallback. Manually drawn lineage for the pieces automation can’t see (that cron job from 2019, the vendor black box). Necessary in corners; fatal as a strategy. Hand-maintained lineage is wrong within a quarter, and — worse than missing — quietly wrong. The operating rule: automate coverage of the 95%, manually annotate the stubborn 5%, and mark the manual parts as such.

The Four Use Cases That Pay for It

Impact analysis (looking downstream). Before changing stg_orders.amount, one query returns every downstream model, dashboard, and ML feature that touches it — and, joined with ownership metadata, who to warn. This is the difference between a schema change shipped with a checklist and one shipped with hope. Wired into CI, it becomes preventive: a PR that alters a column gets an automated comment listing blast radius, before merge, every time. (This is “active metadata” from the data-fabric post, in its most concretely useful form.)

Root-cause analysis (looking upstream). The dashboard is wrong; walk upstream through four hops; find the source whose freshness check failed at 3 a.m. What was an afternoon of ad-hoc archaeology becomes a two-minute traversal. Data-observability tools lean on lineage for exactly this: an anomaly in one table plus lineage equals a correlated incident instead of forty independent alerts.

Compliance and audit. “Prove where this regulated figure comes from” (BCBS 239 in banking demands precisely this) and “find every asset derived from personal data” (GDPR erasure and processing records) are lineage queries. Classification tags (see the governance post) propagate along lineage edges — tag PII at the source and the graph tells you everywhere it flowed, including that forgotten export.

Trust and debugging avoidance. The humblest and maybe highest-frequency use: an analyst deciding between fct_revenue and revenue_final_v3 opens each one’s lineage; one traces cleanly to sources and refreshed this morning, the other dead-ends in a dropped table. No meeting required.

Why Lineage Projects Fail

The failure mode is consistent enough to name: lineage attempted as a documentation project. A team is assigned to “map the data flows,” produces a beautiful diagram over a quarter, and the diagram is a lie by the next quarter because nothing updates it. The fixes are the same lessons as the metadata post in this series, applied here:

Lineage must be a byproduct of running, not an act of recording. If it isn’t extracted from code and logs automatically, it will not stay true.
Partial-but-correct beats complete-but-stale. Full automated coverage of the dbt project is worth more than a hand-drawn map of everything, because people can trust it — and trust is the entire product.
Resolution follows use case. Don’t pay the column-level complexity tax everywhere on day one; get table-level everywhere, column-level where changes hurt most (the core marts), BI integration next.
Lineage without ownership is trivia. “Twelve downstream assets” is a number; “twelve assets owned by these three teams, notified automatically” is a capability. Join the graph to owner metadata or the impact analysis stops at knowing, not acting.

Getting Started, Concretely

If you’re on dbt: you already have table-level lineage — expose it beyond the docs site by loading it into your catalog. Cost: near zero.
Turn on warehouse access-history/audit-log parsing (native features or a catalog’s integration) to catch lineage from tools that bypass dbt.
Instrument the non-SQL pipelines with OpenLineage emitters.
Integrate BI: dashboards-to-tables is the hop that makes impact analysis meaningful to the business.
Wire the first automation: schema-change PRs get an auto-generated downstream-impact comment. This single workflow converts lineage from a diagram people admire into a tool people rely on.

Lineage is ultimately the platform’s memory of its own construction. Every pipeline you ship encodes decisions — this joins that, this filters those — and without lineage, that knowledge lives only in the heads of whoever shipped it, depreciating with every departure. Captured automatically and joined to ownership, it becomes the thing every mature data organization eventually realizes it can’t operate without: the map that’s always as current as the territory.