7. A dbt run traced

Everything so far has been pieces. This chapter follows one command from the shell to the warehouse and back.

dbt build --select marts.customers+ --target prod

The project

   sources:  raw.jaffle_shop.raw_customers
             raw.jaffle_shop.raw_orders

   models:   stg_customers   (view)
             stg_orders      (view)
             customers       (table)   ← selected
             customer_ltv    (table)   ← downstream of it

   tests:    unique(customers.customer_id)
             not_null(customers.customer_id)

Phase 1 — configuration, before anything else

dbt walks up from the working directory looking for dbt_project.yml. Finding it fixes the project root; everything else is relative to that.

Then it resolves configuration from four layers:

   1. dbt_project.yml       the project, its paths, defaults
   2. profiles.yml          which target, credentials
   3. environment variables env_var() calls, resolved now
   4. CLI flags             --target prod, --select ...
                            (highest priority)

--target prod selects the prod output. Credentials come out of the environment. If DBT_PROD_PASSWORD is unset, dbt fails here — before parsing, before connecting, with a clear message.

Then one connection is opened and a trivial query issued to confirm the warehouse answers. This is the same check dbt debug performs on its own.

Phase 2 — parse, and the manifest

The expensive phase, and the one people do not know is happening.

dbt reads every file in the project: every .sql under models/, every .yml, every macro, every seed header, every snapshot, plus everything in dbt_packages/. For each it builds a node.

   models/staging/stg_customers.sql
      → node  model.jaffle_shop.stg_customers
        path, raw SQL, config, refs, sources,
        tags, description, columns, tests

Jinja’s first pass

This is the subtle bit. dbt renders the Jinja twice, for different reasons.

At parse time it renders each model with a set of capture functions installed. ref() does not return a table name; it records “this model depends on that model” and returns a placeholder. Same for source() and config().

   PARSE-TIME ref('stg_orders')
     → records an edge
     → returns a dummy relation

   RUN-TIME ref('stg_orders')
     → returns analytics.analytics.stg_orders

This is why ref() cannot be inside a conditional that depends on runtime data. dbt needs to know the graph before it runs anything, so every ref() in the file is captured regardless of whether the branch would execute:

{% if some_condition %}
  select * from {{ ref('model_a') }}
{% else %}
  select * from {{ ref('model_b') }}
{% endif %}

Both edges are recorded. The graph is the union of every ref() in the text — a deliberately conservative choice, since building something unnecessarily is safer than building in the wrong order.

The manifest

All nodes and edges are written to target/manifest.json — the complete, resolved description of the project: every model, its compiled dependencies, its config, its tests, its documentation, its checksum.

The manifest is the single most important artifact dbt produces. The docs site is a rendering of it. state:modified selection compares against it. Every dbt metadata tool reads it. Chapter 12 is largely about using it.

Partial parsing

Parsing a large project takes real time — tens of seconds at a few thousand models. So dbt caches:

   target/partial_parse.msgpack

On the next invocation dbt compares file checksums and re-parses only files that changed. Parse time drops from 30 seconds to under 2.

It is normally reliable. When it is not — a macro change that does not seem to take effect, a stale-looking graph — the fix is:

dbt parse --no-partial-parse
# or simply
rm target/partial_parse.msgpack

Certain changes invalidate the cache wholesale: editing dbt_project.yml, changing environment variables used at parse time, installing packages, or changing dbt version. That is why the first run after a config change is slow.

Phase 3 — the graph, and selection

The manifest’s edges become a directed acyclic graph. dbt checks for cycles here; a circular ref() fails now, with the cycle printed.

Then selection is applied. --select marts.customers+ means customers and everything downstream of it:

   stg_customers ─┐
                  ├─▶ customers ─▶ customer_ltv
   stg_orders   ──┘
                     └───────┬────────┘
                          SELECTED

   stg_customers, stg_orders: NOT selected.
   They exist and are assumed already built.

Selection syntax, all of it composable:

   customers        just that model
   +customers       customers and all ancestors
   customers+       customers and all descendants
   +customers+      the whole lineage through it
   2+customers      two levels of ancestors only
   tag:finance      everything tagged finance
   path:models/marts  everything in a directory
   source:jaffle_shop+  everything from a source
   state:modified+  changed nodes + descendants
   config.materialized:incremental

Because dbt build was the command, tests attached to selected models are pulled in too, and so are the seeds and snapshots the selection covers.

The final selection here:

   model.customers
   test.unique_customers_customer_id
   test.not_null_customers_customer_id
   model.customer_ltv

Phase 4 — scheduling

dbt walks the graph and hands ready nodes to a thread pool sized by threads in the profile — 16 here.

A node is ready when all its parents have succeeded. This is the whole scheduler; there is no priority, no weighting, no cost model.

   t=0    customers                    (running)
          nothing else ready

   t=48s  customers ✓
          → unique test        (running)
          → not_null test      (running)
          → customer_ltv       (running)
            all three, in parallel

   t=53s  tests ✓
   t=71s  customer_ltv ✓

Note that customer_ltv starts before the tests on customers finish. dbt build’s guarantee is that a node’s own tests gate its children — and with the default severity: error, a failing test marks the child as skipped. What it cannot do is un-run a child that started in parallel. In practice the parallelism is worth it; if you need a hard barrier, split the run.

threads is the main throughput dial, and it is a warehouse setting in disguise. Sixteen threads means sixteen concurrent queries. Chapter 13 covers where that stops helping.

Phase 5 — one node, in detail

Take customers.

Compile

The model’s raw SQL is rendered with the full runtime context — ref() now returns real names, var() returns values, macros execute, this resolves.

-- target/compiled/jaffle_shop/models/marts/customers.sql

select
    c.customer_id,
    c.customer_name,
    count(o.order_id) as order_count
from analytics.analytics.stg_customers as c
left join analytics.analytics.stg_orders as o
    on c.customer_id = o.customer_id
group by 1, 2

Pure SQL. No Jinja left. This file is the ground truth for debugging — paste it into a warehouse console and you are running exactly what dbt ran.

Materialize

The compiled SQL is passed to the materialization macro named by the config — here materialization_table_default, or the Snowflake override if the adapter ships one. The macro is Jinja that emits statements:

   1. does analytics.analytics.customers exist?
      → catalogue lookup
   2. create or replace table
        analytics.analytics.customers__dbt_tmp
      as ( <compiled SQL> )
   3. alter table ... swap / rename
   4. drop the old relation
   5. apply grants
   6. run post-hooks

Written to target/run/.../customers.sql — the compiled SQL plus its DDL wrapper. The distinction is worth remembering: target/compiled/ is your query; target/run/ is what was actually executed.

Execute

Statements go over the adapter’s connection. dbt waits. It is not doing anything else for this node — no data passes through dbt, only SQL out and status back.

The adapter returns rows affected, timing, and warehouse-specific metadata (a Snowflake query ID, BigQuery bytes billed). dbt records it and marks the node success.

The tests

A generic test is a model too — it compiles to a SELECT that returns failing rows.

-- the compiled unique test
select customer_id as unique_field, count(*) as n_records
from analytics.analytics.customers
where customer_id is not null
group by customer_id
having count(*) > 1

dbt wraps it in select count(*) from ( ... ). Zero rows means pass. Any rows means fail, and the count is what gets reported.

That is the entire test mechanism. There is no assertion framework — a test is a query that should return nothing.

Phase 6 — results

Every node’s outcome is collected into target/run_results.json: status, execution time, rows affected, adapter response, failure messages.

   Finished running 1 view model, 2 tests,
   1 table model in 0 hours 2 minutes 4 seconds.

   Completed successfully

   Done. PASS=4 WARN=0 ERROR=0 SKIP=0 TOTAL=4

Exit code 0 on full success, 1 on any error, 2 on an internal failure.

run_results.json is what CI and observability tools read. Chapter 12 uses it for result:error reruns; chapter 13 uses its timings to find slow models.

The artifacts, together

   target/
   ├── manifest.json          the whole project, resolved
   ├── run_results.json       what happened this run
   ├── catalog.json           warehouse columns+stats
   │                          (only from `dbt docs generate`)
   ├── sources.json           freshness results
   ├── partial_parse.msgpack  the parse cache
   ├── compiled/              your SQL, Jinja resolved
   └── run/                   compiled SQL + DDL wrapper

These are a public interface. They are versioned and documented, and the ecosystem around dbt — Elementary, Cosmos, lineage tools, cost tools — is built on reading them rather than on any API.

The shape of the whole thing

   dbt build --select marts.customers+ --target prod
        │
        ▼
   ┌────────────────────────────────────────┐
   │ 1  RESOLVE CONFIG                      │
   │    project → profile → env → flags     │
   ├────────────────────────────────────────┤
   │ 2  PARSE          (partial cache)      │
   │    every file → nodes                  │
   │    Jinja pass 1: capture refs          │
   │    → manifest.json                     │
   ├────────────────────────────────────────┤
   │ 3  GRAPH + SELECT                      │
   │    edges → DAG, cycle check            │
   │    apply --select                      │
   ├────────────────────────────────────────┤
   │ 4  SCHEDULE                            │
   │    ready nodes → thread pool           │
   ├────────────────────────────────────────┤
   │ 5  PER NODE                            │
   │    Jinja pass 2: compile               │
   │    → target/compiled/                  │
   │    materialization macro → DDL         │
   │    → target/run/                       │
   │    adapter → WAREHOUSE  ← all the work │
   ├────────────────────────────────────────┤
   │ 6  COLLECT → run_results.json          │
   └────────────────────────────────────────┘

Nothing in steps 1 to 4 touches data. dbt spends its time reading files and building a graph. Step 5 is where the warehouse does everything, and it is where all the money goes.

What to take from this chapter

Config resolves before parsing: project, then profile, then environment, then CLI flags. A missing env_var fails immediately.
dbt renders Jinja twice — once at parse time with capture functions to build the graph, once at run time to produce real SQL.
Because of that, every ref() in the file becomes an edge, including ones inside branches that will not execute.
manifest.json is the project, fully resolved. Docs, state selection and the whole metadata ecosystem read it.
Partial parsing caches the parse in partial_parse.msgpack. Delete it when a change mysteriously fails to take effect; config and version changes invalidate it wholesale.
Selection operates on the graph — + for ancestors and descendants, plus tag:, path:, source:, state: and config. methods, all composable.
The scheduler is simple: a node runs when its parents succeed, on a pool sized by threads. dbt build gates children on a node’s own tests, but siblings started in parallel cannot be recalled.
target/compiled/ is your SQL; target/run/ is your SQL plus the DDL. Compiled SQL is the ground truth for debugging.
A test is a query that should return zero rows. There is no assertion framework.
run_results.json records the run and is what CI and observability tools consume.
Everything expensive happens in the warehouse. dbt is reading files and waiting on a connection.

Next: Jinja and macros — the templating layer, the context you get, writing macros that work across adapters, and how not to overuse it.

Written by Sepuri Sai Krishna