10. Documentation and lineage

Most data documentation is a wiki page that was accurate once. dbt’s bet is that documentation written next to the code, reviewed with the code, and rendered from the same graph the code builds has a chance of staying true.

Descriptions

Anywhere a resource is defined in YAML, it can carry a description.

version: 2

models:
  - name: customers
    description: >
      One row per customer, with lifetime order metrics.
      The grain is customer_id. Excludes internal test accounts
      (see the filter in stg_customers).

    columns:
      - name: customer_id
        description: Primary key. Sourced from the app database.
        tests: [unique, not_null]

      - name: lifetime_value
        description: >
          Sum of amount_usd across all non-cancelled orders,
          in USD at time of order. Not currency-adjusted.

      - name: first_order_date
        description: NULL for customers who have never ordered.

Descriptions work on models, columns, sources, seeds, snapshots, macros, tests and exposures.

Descriptions are Markdown, so lists, links and emphasis all render.

What is worth writing

The thing that separates useful docs from ceremony:

   ✗ USELESS
     customer_id: The customer ID.
     order_date:  The date of the order.
     amount:      The amount.

   ✓ USEFUL
     customer_id: Primary key. From the app DB.
                  Note: differs from Stripe's
                  customer id, see stg_payments.
     order_date:  Date the order was PLACED, not
                  shipped. UTC. Orders before
                  2021-03 have this as NULL.
     amount:      Gross, in USD cents, before
                  discounts and tax. Use
                  amount_net for revenue reporting.

Document the surprises, not the obvious. Nobody needs to be told customer_id is a customer id. They need to know the timezone, the units, where the nulls are, which of two similar columns to use, and what was true before a migration. If a column has ever caused someone to compute the wrong number, that story belongs in its description.

Doc blocks

For descriptions used in several places, or long enough to clutter the YAML:

{# models/docs.md #}

{% docs customer_status %}

The lifecycle state of a customer account.

| Value    | Meaning                                    |
|----------|--------------------------------------------|
| `trial`  | Signed up, no payment method               |
| `active` | Paying, last invoice settled               |
| `churned`| Cancelled or 90+ days past due             |

Set by the billing service nightly. A customer can move
back from `churned` to `active` if they resubscribe.

{% enddocs %}

      - name: status
        description: '{{ doc("customer_status") }}'

One definition, referenced from every model with that column. When the meaning of a status changes, there is one place to edit — which is the whole reason to bother.

Doc blocks live in .md files anywhere under your model paths.

Generating the site

dbt docs generate
dbt docs serve

docs generate runs a dbt compile and then queries the warehouse’s information schema for every relation in the project, producing target/catalog.json: actual columns, data types, row counts and sizes.

   manifest.json   what you WROTE
                   models, refs, tests, descriptions

   catalog.json    what EXISTS
                   real columns, types, row counts,
                   bytes — read from the warehouse

   the docs site   both, rendered together

That pairing is what makes the docs honest. A column you documented but never built shows up as missing. A column in the warehouse that nobody documented shows up as undocumented. The site is not a description of your intentions; it is a diff against reality.

docs serve starts a local web server. The site itself is static — index.html plus the two JSON files — so it can be hosted anywhere: S3, GitHub Pages, an internal bucket. dbt Cloud hosts it for you.

Note that docs generate needs the models to actually exist. Run it after a build, against a target where the relations are present.

The lineage graph

The feature people actually use the docs for.

   raw.jaffle_shop.raw_customers
        └─▶ stg_customers ─┐
                           ├─▶ customers ─┬─▶ customer_ltv
   raw.jaffle_shop.raw_orders             │
        └─▶ stg_orders   ─┘               └─▶ [exposure]
                                              Exec Dashboard

Clicking any node shows its ancestors and descendants. The questions it answers, quickly:

• “If I change this, what breaks?” — look downstream.
• “Where does this number come from?” — look upstream, to the raw table.
• “Is anything actually using this model?” — no descendants and no exposure means probably not.

This graph is free and correct, because it is derived from ref() rather than maintained by hand. That is the payoff for chapter 3’s rule about never hard-coding a model name — every hard-coded reference is a missing edge in this picture, and a missing edge is worse than no picture, because people trust it.

The docs site’s graph explorer supports the same selection syntax as the CLI, so +customers+ works there too.

Exposures

The graph stops at the edge of dbt. Exposures extend it into the things that consume your models.

version: 2

exposures:
  - name: executive_dashboard
    label: Executive Dashboard
    type: dashboard
    maturity: high
    url: https://looker.company.com/dashboards/42
    description: >
      Board-level revenue and retention. Reviewed weekly.
      Breaking this is a Monday-morning problem.

    depends_on:
      - ref('customers')
      - ref('customer_ltv')
      - ref('monthly_revenue')

    owner:
      name: Data Team
      email: data@company.com

type is one of dashboard, notebook, analysis, ml, application.

Two concrete payoffs:

They appear in the lineage graph, so “what breaks if I change this?” includes the dashboard, and you know who to warn.

They are selectable:

dbt build --select +exposure:executive_dashboard

Build exactly what the board dashboard needs, and nothing else. Useful for a targeted refresh before a meeting.

Exposures are maintained by hand, which means they drift. The realistic advice is to declare them for the handful of consumers that genuinely matter — the dashboards someone would ring you about — rather than attempting a complete inventory that will be wrong within a month.

persist_docs

Descriptions can be pushed into the warehouse itself, as native comments:

models:
  jaffle_shop:
    +persist_docs:
      relation: true
      columns: true

Now the description appears in Snowflake’s UI, in information_schema, in whatever BI tool reads warehouse metadata, and to anyone who never opens the dbt docs.

This is high-value and underused. Most people querying your tables will never see the dbt docs site. They will see the column comment in their SQL editor’s autocomplete. Adapter support varies — check yours.

meta

Arbitrary key-value metadata on any resource:

models:
  - name: customers
    config:
      meta:
        owner: analytics-team
        contains_pii: true
        sla_hours: 6
        cost_centre: marketing

dbt does nothing with meta beyond storing it in the manifest and displaying it. Its value is that something else reads it — a script that finds every PII model, a governance tool, a cost report, a CI check that every mart has an owner.

dbt ls --select "config.meta.contains_pii:true"

Selection on meta makes it operational rather than decorative.

Groups and access

For larger projects, models can be grouped and given visibility:

groups:
  - name: finance
    owner:
      name: Finance Analytics
      email: finance-data@company.com

models:
  - name: revenue_internal
    config:
      group: finance
      access: private

private means only models in the same group may ref() it; protected allows any model in the project; public allows other projects (the dbt Mesh pattern, chapter 13).

This turns a naming convention into something dbt enforces. A ref() that violates access fails at parse time rather than being caught in review.

Keeping docs honest

Documentation rots. The mechanisms that slow it down:

Require descriptions in CI. dbt-project-evaluator flags undocumented models; a CI step that fails on a mart without a description makes it non-negotiable rather than aspirational.

Document at the right layer. Every column of every staging model does not need prose — those are renames of documented sources. Marts are where documentation pays, because marts are what people query.

Review descriptions in pull requests. A change to a column’s meaning without a change to its description should get the same comment as an untested model.

Prefer doc blocks for shared meanings, so there is one place to update.

Let persist_docs do the distribution. Documentation people encounter without seeking it out is documentation that gets read.

What to take from this chapter

• Descriptions live in YAML next to the model and are reviewed with the code, which is the only reason they stay true.
• Document the surprises — units, timezones, nulls, which of two similar columns to use, what changed in a migration. Not “the customer id is the customer id”.
• Doc blocks give one definition many uses, so a changed meaning is a one-line edit.
• manifest.json is what you wrote; catalog.json is what exists. The docs site renders both, which is what makes it a diff against reality rather than a statement of intent.
• The lineage graph is free and correct because it comes from ref(). Every hard-coded name is a missing edge, and a missing edge is worse than no graph.
• Exposures extend lineage to dashboards and models, make “what breaks?” complete, and are selectable with +exposure:name. Declare the ones that matter, not all of them.
• persist_docs pushes descriptions into the warehouse, where people who never open the docs site will actually see them.
• meta is arbitrary metadata that becomes useful when something reads it — and it is selectable.
• Groups and access turn conventions into rules dbt enforces at parse time.

Next: configuration and precedence — where config can live, the order it resolves in, and the two places it genuinely surprises people.