Metadata Management: Making Data About Data Actually Useful

Every data team has lived this scene: an analyst finds a table called cust_f_v2, containing a column called status with values 1 through 5, last touched by an employee who left in 2023. The data is right there — and completely unusable, because everything needed to trust it is missing. What does status 3 mean? Is v2 the current version? Is this refreshed daily or abandoned?

That missing layer is metadata, and managing it deliberately is what separates data platforms people trust from data swamps people route around. This post breaks down the three kinds of metadata that matter, why manual documentation always fails, what “active metadata” changes, and a realistic path to getting your estate described.

The Three Kinds of Metadata (and Who Needs Each)

“Metadata” gets used loosely; splitting it into three layers makes the management problem tractable, because each layer has a different source and a different audience.

Technical metadata — what the data is. Schemas, types, constraints, partitioning, file formats, sizes. Audience: engineers. Source: the systems themselves — this layer can and must be harvested automatically, because it changes with every deploy. Any technical metadata maintained by hand is fiction within a month.

Business metadata — what the data means. Definitions (“an active customer has transacted in the trailing 90 days”), the mapping from status = 3 to “suspended,” ownership, sensitivity classification, quality expectations. Audience: everyone, especially analysts and business users. Source: humans — this is the layer that cannot be fully automated, because meaning lives in heads until someone writes it down. It’s also, not coincidentally, the highest-value layer.

Operational metadata — what is happening to the data. Pipeline run times, freshness, row counts, quality-check results, lineage, who queries what and how often. Audience: engineers for reliability, governance for compliance, and — underused — analysts, for whom “this table is queried 500 times a day and was refreshed 2 hours ago” is the strongest trust signal that exists. Source: automated, from orchestration and query logs.

A concrete way to see the layers working together — everything in this snippet beyond the column name and type is metadata management in action:

# dbt schema.yml — metadata living next to the model that it describes
models:
  - name: fct_orders
    description: "One row per order line. Grain: order_id + line_number."
    meta:
      owner: commerce-analytics@company.com
      sla_freshness: "06:00 UTC daily"
    columns:
      - name: status
        description: "1=placed 2=paid 3=suspended 4=shipped 5=returned"
        meta: { sensitivity: internal }
        tests: [not_null, accepted_values: {values: [1,2,3,4,5]}]

Why Documentation Projects Always Fail (and What Works Instead)

Every organization has attempted the wiki: a quarter of enthusiastic documentation, followed by permanent decay. The failure is structural, not moral — documentation stored away from the thing it describes rots, because nothing forces it to update when the thing changes. Three principles fix this:

1. Harvest everything harvestable. Schemas, lineage, freshness, usage — machines should collect all of it continuously. Reserve scarce human effort for the business layer that machines can’t produce.

2. Put human-authored metadata in the development workflow, not beside it. The dbt example above is the pattern: descriptions and ownership live in the same repo as the model, reviewed in the same pull request, versioned in the same git history. A schema change that doesn’t update its description gets caught in review — because they’re in the same diff. Contrast this with a wiki page nobody’s PR touches. The general rule: metadata that isn’t in the path of change will drift from the truth.

3. Let usage prioritize effort. You cannot document 40,000 tables, and you don’t need to. Query logs tell you which 200 assets serve 90% of consumption; document those to a high standard, and mark the abandoned long tail for archival instead of description. (Deleting an unused table is the most efficient metadata management there is.)

From Passive to Active Metadata

Traditional metadata sits in a repository waiting to be read. The significant shift of the last few years — discussed in the data fabric post in this series — is metadata that acts:

• Lineage + a detected schema change → automatic impact analysis and downstream owner alerts before the breakage.
• Column profiling → auto-classification of PII → masking policy attached without a human remembering to.
• Usage stats → “this dashboard’s source hasn’t refreshed in 3 days” warnings shown in the BI tool, where the consumer actually is.
• Cost + usage → automated candidate lists for archival.

Active metadata is also the honest answer to “why bother collecting all this?” — because every automation above is impossible without the underlying layers being populated and current.

The Semantic Layer: Metadata’s Most Valuable Special Case

One category of business metadata deserves its own callout: metric definitions. When “monthly revenue” is computed independently in six dashboards, the definitions drift — currency handling here, refund treatment there — and leadership meetings devolve into reconciling numbers. A semantic layer (dbt’s semantic layer, LookML, Cube, and warehouse-native equivalents) moves the definition into governed code:

metrics:
  - name: net_revenue
    model: fct_orders
    calculation: sum(amount) - sum(refund_amount)
    dimensions: [order_date, region, product_category]

Every tool that asks for net_revenue now gets the same formula. This is metadata management at its most concrete: a definition, written once, enforced everywhere. If your organization does nothing else from this post, centralizing the top twenty metric definitions delivers the fastest visible payoff.

Governance Metadata: The Compliance Angle

Regulators have quietly turned metadata into a legal requirement. GDPR’s “records of processing,” right-to-erasure requests, and breach-notification windows all presuppose you can answer: where does personal data live, where did it come from, who accesses it? Those are metadata queries — classification, lineage, and access logs respectively. Organizations discover this either at design time (cheap: tag sensitivity in your schemas and contracts as you model, and let harvesters propagate it) or during an audit (expensive: consultants grepping column names for ssn). The modeling takeaway from this series applies again: classification is a schema-time concern.

A Realistic Roadmap

1. Week one: turn on automated harvesting — schema crawling and query-log collection into whatever catalog or store you have. Zero human effort, immediate inventory.
2. First month: identify the top ~50 assets by usage; assign each a named owner and require grain + column descriptions, enforced by a CI check (dbt makes “description required” a one-line rule).
3. First quarter: centralize the top metric definitions in a semantic layer; add sensitivity tags to the modeled core; wire freshness and lineage into alerts (first active-metadata win).
4. Ongoing: metadata coverage as a visible health metric per team; archival campaigns for unqueried assets; definitions reviewed when — and only when — the underlying business rule changes.

The mental shift that makes all of this stick: stop treating metadata as documentation about the platform and start treating it as part of the platform — versioned like code, collected like telemetry, and consumed by machines as much as by people. Data without metadata isn’t an asset yet; it’s inventory nobody can find the label for.