12. State, artifacts and slim CI

A project with two thousand models cannot rebuild everything for every pull request. This chapter is about the mechanism that makes it unnecessary: compare the current project against a saved manifest, and act only on the difference.

State is a manifest

Chapter 7 introduced manifest.json — the whole project, resolved: every node, its config, its dependencies, and a checksum of its SQL.

Point dbt at a previous one and it can compare:

dbt build --select "state:modified+" --state ./prod-artifacts

   ./prod-artifacts/manifest.json     what production has
   ./target/manifest.json             what this branch has

   dbt diffs them node by node

--state is a directory containing artifacts, not a file. DBT_STATE sets it as an environment variable if you prefer.

The state selectors

Selector	Matches
`state:new`	nodes not in the previous manifest
`state:modified`	nodes that changed in any way
`state:modified.body`	SQL changed
`state:modified.configs`	config changed
`state:modified.contract`	contract changed
`state:modified.relation`	database/schema/alias changed
`state:modified.macros`	a macro it uses changed
`result:error`	failed in the previous run
`result:fail`	tests that failed
`source_status:fresher`	sources with new data

Composable with graph operators, as always:

state:modified+      # changed nodes and everything downstream
+state:modified      # changed nodes and their ancestors
state:modified+1     # changed nodes and one level down

state:modified+ is the one you want in CI. A changed model plus everything that depends on it — because a change to a staging model can break a mart three layers away.

state:modified is deliberately broad: a description edit counts, because it changes the manifest. The .body and .configs variants narrow it when that matters.

Defer

State selection halves the problem. Defer solves the other half.

Build only customers in CI, and its parents do not exist in the CI schema:

   CI schema  dbt_ci_pr_412
   ├── customers        ← built
   └── (nothing else)

   customers refs stg_customers
   → analytics.dbt_ci_pr_412.stg_customers
   → does not exist. run fails.

--defer fixes it:

dbt build --select "state:modified+" \
          --state ./prod-artifacts \
          --defer --favor-state

Unbuilt refs resolve to production instead.

   customers        → dbt_ci_pr_412.customers  (built here)
   stg_customers    → analytics.stg_customers  (production)

The rule: if a node is in the current selection, use the CI version; otherwise use the one from the state manifest. --favor-state strengthens it — always prefer the state version for anything not selected, even if a stale relation happens to exist in the CI schema. Use --favor-state, or a leftover table from a previous CI run will quietly be used instead of production.

Defer is what makes slim CI cheap. You build three models and read the other 1,997 from production.

The slim CI pipeline

The whole shape:

   ┌──────────────────────────────────────────┐
   │ PRODUCTION JOB (nightly / hourly)        │
   │   dbt build --target prod                │
   │   upload target/manifest.json  ──────┐   │
   └──────────────────────────────────────│───┘
                                          ▼
                              ┌────────────────────┐
                              │ artifact storage   │
                              │ S3 / GCS / CI cache│
                              └─────────┬──────────┘
   ┌──────────────────────────────────────│───────┐
   │ PULL REQUEST JOB                     ▼       │
   │   download manifest → ./prod-artifacts       │
   │   dbt deps                                   │
   │   dbt build --select state:modified+         │
   │             --state ./prod-artifacts         │
   │             --defer --favor-state            │
   │             --target ci                      │
   └──────────────────────────────────────────────┘

   WITHOUT SLIM CI     2000 models   45 min   £££
   WITH SLIM CI        3 models      2 min    £

The manifest upload is the piece people forget. Without a current production manifest there is nothing to diff against, and CI either builds everything or nothing. Publish it from the production job every time it succeeds.

A GitHub Actions sketch

- name: Fetch production manifest
  run: |
    mkdir -p prod-artifacts
    aws s3 cp s3://our-dbt-artifacts/manifest.json prod-artifacts/ \
      || echo "no manifest yet — will build everything"

- name: Install packages
  run: dbt deps

- name: Build changed models
  run: |
    dbt build --select "state:modified+" \
              --state ./prod-artifacts \
              --defer --favor-state \
              --target ci
  env:
    DBT_CI_PASSWORD: ${{ secrets.DBT_CI_PASSWORD }}

Handle the missing-manifest case explicitly. The first ever run has no state, and --select state:modified+ with no state file is an error. Falling back to a full build — or to nothing — should be a decision, not a crash.

CI schemas

Give each pull request its own schema, and drop it afterwards:

{% macro generate_schema_name(custom_schema_name, node) -%}
    {%- if target.name == 'ci' -%}
        dbt_ci_pr_{{ env_var('PR_NUMBER') }}
    {%- elif target.name == 'prod' and custom_schema_name is not none -%}
        {{ custom_schema_name | trim }}
    {%- else -%}
        ...
    {%- endif -%}
{%- endmacro %}

dbt run-operation drop_ci_schema --target ci

Schedule the cleanup regardless of PR outcome. Abandoned pull requests leave schemas behind, and a year of them is real money.

Retry

dbt build          # fails at model 340 of 500
dbt retry

dbt retry reads run_results.json and re-runs only the failed nodes and everything skipped because of them. The 339 successful models are not rebuilt.

This is the right response to a transient failure — a warehouse timeout, a lost connection — and it turns a forty-minute rebuild into a two-minute recovery. Put it in your orchestration’s retry policy rather than re-running the whole job.

The older explicit form still works and is useful when you want to be selective:

dbt build --select "result:error+" --state ./target

Source-driven runs

dbt source freshness
dbt build --select "source_status:fresher+" --state ./previous

Build only what sits downstream of sources that actually have new data. Useful when sources update on wildly different schedules — a table refreshed monthly should not trigger its dependents hourly.

It needs a previous sources.json to compare against, so the same artifact-publishing discipline applies.

Where to keep artifacts

   ✓ object storage (S3, GCS)
     versioned, durable, easy to fetch
     the standard answer

   ✓ CI cache
     free, fast, but eviction is not
     under your control

   ✓ dbt Cloud
     handles it for you

   ✗ git
     manifest.json is large and changes
     completely every run. do not.

Never commit manifest.json. It is generated, it is big, and every run rewrites it entirely — the diffs are useless and the repository grows without bound.

The sharp edges

Manifest version mismatch. Artifacts carry a schema version. A manifest from a different dbt minor version may fail to parse. Pin the dbt version in CI to whatever production runs, and re-publish the manifest after upgrading.

Everything looks modified. Usually one of: a package upgrade changing macro checksums, a dbt_project.yml edit, a version change, or an environment variable read at parse time differing between the two runs. That last one is the subtle case — env_var() in a config makes the manifest environment-dependent.

Nothing looks modified. Almost always a stale state manifest. Confirm the production job is actually publishing it.

Defer picks up something wrong. Add --favor-state.

state:modified misses a hard-coded reference. Chapter 3’s rule, arriving with consequences: a model that reads analytics.analytics.stg_orders directly is not a descendant of stg_orders in the graph, so state:modified+ will not select it when stg_orders changes. CI passes and production breaks. This is the most expensive version of that bug.

What else the artifacts are for

State selection is the headline use, but the artifacts have a wider job — they are the reason dbt has an ecosystem.

   manifest.json     lineage tools, catalogues,
                     column-level lineage,
                     Cosmos rendering dbt as
                     Airflow tasks

   run_results.json  observability, cost
                     attribution, alerting,
                     "which models got slower?"

   catalog.json      data catalogues, table
                     sizes, column inventories

   sources.json      freshness dashboards

dbt has no metadata API to speak of; it has files. Elementary, Cosmos, Montecarlo, DataHub, OpenLineage and a long tail of internal scripts all work by reading these. A simple job that loads run_results.json into a table after every run gives you model-level timing history — which is where chapter 13’s tuning work starts.

What to take from this chapter

State is a saved manifest.json. --state points at a directory containing it.
state:modified+ is the CI selector — changed nodes and everything downstream. Narrow with .body or .configs when description edits are noise.
--defer --favor-state resolves unbuilt refs to production, so CI builds three models instead of two thousand. Always include --favor-state.
Publishing the production manifest is the piece people forget, and without it CI has nothing to diff.
Handle the missing-manifest case explicitly — the first run has no state.
Give each PR its own schema and schedule the cleanup, regardless of outcome.
dbt retry re-runs only failures and their skipped descendants. Make it the retry policy.
source_status:fresher+ builds only what has new data — good for mixed-cadence sources.
Never commit artifacts to git. Object storage or the CI cache.
Pin dbt versions between CI and production, or the manifest may not parse.
A hard-coded relation name defeats state selection entirely, so CI passes and production breaks. This is why chapter 3’s rule matters.
The artifacts are dbt’s real metadata interface and the whole ecosystem reads them.

Next: tuning and operations — warehouse cost, project structure at scale, orchestration and the practices that keep a large project healthy.

Written by Sepuri Sai Krishna