15. When to use dbt, and when not

dbt won its category so completely that “should we use dbt?” is often answered before it is asked. That is usually the right answer and sometimes not, so here is the argument properly.

The one question

Is most of your transformation logic expressible as SQL over tables that already live in one warehouse?

Yes → use dbt. Nothing else in this chapter is close.

No → read on, because the alternatives differ sharply depending on why the answer is no.

Three ways to answer no, each pointing somewhere different:

   "the logic isn't SQL"
     → Spark, or Python where it belongs

   "the data isn't in one warehouse"
     → fix that first (ingestion), or Trino

   "it can't wait for a batch"
     → Flink, or streaming tables

What dbt is genuinely good at

Worth stating plainly, because the comparisons only make sense against it:

• Dependency management from ref(). The graph is derived, not maintained.
• Environments for free. One flag, no code changes.
• Testing that is trivial to write, so it actually gets written.
• Documentation and lineage generated from the graph, so they stay true.
• An enormous ecosystem — packages, adapters, metadata tools, and a hiring pool who already know it.

That last point is a real engineering argument, not a soft one. An analytics engineer joining your team already knows dbt. Any bespoke framework, however elegant, costs every new joiner weeks.

dbt vs SQLMesh

The serious open-source competitor, and the comparison that deserves the most care.

   dbt                        SQLMesh
   ─────────────────────      ─────────────────────
   treats SQL as text         PARSES SQL (sqlglot)
   Jinja templating           understands columns
   full refresh to change     virtual data
     logic                      environments
   state = manifest diff      state = column-level
                                change detection
   huge ecosystem             small but growing
   the hiring default         a learning curve

SQLMesh’s core idea is that it understands your SQL. Because it parses into an AST, it can tell a breaking change from a non-breaking one, work out which columns a change actually affects, and rebuild only what genuinely needs it.

The practical consequences:

• Column-level lineage without a separate tool.
• Virtual environments — a dev environment is a set of views over production data, so creating one is nearly free rather than a full rebuild.
• Change categorisation — adding a column does not force a full refresh of a billion-row table the way a state:modified rebuild might.
• Less Jinja, because the engine can do statically what dbt needs templating for.

Where dbt still wins: ecosystem, adapters, packages, documentation, tooling, and people who know it. That is not a small list, and for most teams it decides it.

Take SQLMesh seriously if full refreshes of very large tables are genuinely painful, or you want column-level lineage without buying it. Note also that dbt’s Fusion engine is aimed squarely at this gap — a SQL-comprehending engine — which may narrow the difference. Check the current state of both before choosing.

dbt vs Spark

Covered from the other side in Spark’s chapter 15, which is blunt: “if your ETL is a chain of SQL transformations, dbt over a warehouse is a better tool than Spark.”

   USE dbt                    USE SPARK
   ─────────────────────      ─────────────────────
   SQL over warehouse tables  logic that is not SQL
   data already loaded        data in files, many
                                formats, many places
   the warehouse can do it    genuinely huge data
                                with non-SQL logic
   analysts should own it     ML feature pipelines,
                                complex Python

The overlap has shrunk from dbt’s side. Work that used to need Spark — large joins, big aggregations — is now comfortably within a warehouse’s reach, and dbt over that warehouse is simpler, cheaper to operate and easier to hire for.

Spark’s remaining territory is very large data with logic that is not SQL. Real, substantial, and much narrower than it once was.

dbt vs Airflow, Dagster, Prefect

Not competitors. dbt has no scheduler; something must run it. Airflow’s chapter 15 puts it the same way.

   Airflow:  extract → load → [ dbt build ] → notify
   dbt:                   the SQL DAG inside
                          the warehouse

The only real question is whether you need an orchestrator at all. If dbt is the only thing you run on a schedule, dbt Cloud jobs or cron are enough. Bring in Airflow when there are other things to sequence — ingestion, ML training, reverse ETL, downstream notifications — and dbt is one step among many.

Chapter 13 covers the one-task-versus-many decision.

dbt vs warehouse-native tooling

Every warehouse now ships something in this space.

   Snowflake   Dynamic Tables, Streams + Tasks
   BigQuery    Dataform (Google-owned, dbt-like),
               scheduled queries, materialized views
   Databricks  DLT / declarative pipelines
   Redshift    materialized views, scheduled queries

These are real and improving. Dynamic Tables in particular handle incremental refresh declaratively, which is genuinely nicer than hand-written incremental logic.

Where dbt still wins:

• Portability. dbt structure survives a warehouse migration; Dynamic Tables do not.
• Testing and documentation, which native tooling mostly lacks.
• One tool across several warehouses, if you have them.
• The ecosystem, again.

Where native tooling wins: deep integration, no extra tool to run, and incremental semantics the warehouse maintains rather than you.

They also compose. A dbt model materialized as a Dynamic Table gets both. That is often the right answer rather than choosing.

dbt vs stored procedures

The thing dbt replaced. The comparison is one-sided but worth spelling out, because plenty of teams still have them.

   STORED PROCEDURES          dbt
   ────────────────────       ────────────────────
   live in the warehouse      live in git
   no version control         reviewed in PRs
   no tests                   tests are one line
   order maintained by hand   graph derived
   no environments            --target
   no lineage                 generated
   imperative                 declarative

Migrate. The only argument for keeping them is procedural logic dbt genuinely cannot express — cursors, loops, transactional control. That is rare in analytics, and where it exists it can stay as a small procedure called from a hook.

dbt vs GUI ETL tools

Matillion, Informatica, Talend, and their kin.

The argument for them is that non-engineers can build pipelines by dragging boxes, and that they bundle ingestion with transformation.

The argument against is everything dbt is for. Version control is bolted on or absent; review is impossible; testing is limited; the logic is locked in a proprietary format; and diffing two versions of a pipeline is genuinely hard.

If you are choosing today, choose dbt — plus a dedicated ingestion tool. The bundling is the only real advantage, and it costs the ability to treat pipelines as software.

dbt vs streaming

   dbt          batch. every run re-evaluates
                a graph. minutes to hours.

   Flink        continuous. per-record. state
                in the operator. sub-second.

   Materialize  streaming SQL, incrementally
   RisingWave   maintained views over a stream

dbt has no streaming story and is not trying to have one. microbatch (chapter 5) reduces latency; it does not change the model.

If freshness requirements are in seconds, dbt is the wrong tool for that path. Flink’s guide covers what it takes.

The common architecture uses both: streaming for the small number of things that genuinely need it, dbt for everything else. Very few things need it. “Real-time” in a requirements document usually means “not eight hours old”, and a fifteen-minute dbt run satisfies it.

dbt vs building it yourself

Every data team has considered a small script that runs SQL files in order.

Version one takes a week and works. Then you need environments, a dependency graph, tests, incremental handling, documentation, a way to run a subset, a way to see what broke. You are rebuilding dbt, worse, with one maintainer.

The only defensible case is a genuinely tiny project — a handful of models, one schedule — where dbt’s structure is overhead. That case exists. It is smaller than people think, and it grows.

When dbt is the wrong choice

   ✗ THE DATA IS NOT IN A WAREHOUSE
     dbt has no ingestion. fix that first
     — Fivetran, Airbyte, Debezium.

   ✗ THE LOGIC IS NOT SQL
     ML training, heavy text processing,
     arbitrary API calls. Python models
     exist and are not the answer at scale.

   ✗ SUB-MINUTE FRESHNESS IS REQUIRED
     dbt is batch. → Flink, Materialize,
     streaming tables.

   ✗ THE PROJECT IS TINY AND WILL STAY TINY
     three models on one schedule. a cron
     job and a .sql file is fine.

   ✗ HEAVY PROCEDURAL LOGIC
     cursors, loops, transactional control.
     dbt is declarative by design.

   ✗ YOUR WAREHOUSE HAS NO ADAPTER
     check first. community adapters vary
     a great deal in quality.

When dbt is obviously right

   ✓ data is in one warehouse
   ✓ transformations are SQL
   ✓ more than one person touches them
   ✓ anyone has ever asked "where does
     this number come from?"
   ✓ anything has ever silently broken
   ✓ you need dev and prod to be the same code
   ✓ you want tests that people will write

Every one of those is common. Most teams tick all seven.

The honest summary

dbt is not technically remarkable. It generates SQL and sends it over a connection; the warehouse does everything hard. There is no distributed system, no clever algorithm, no novel storage format.

What it did was notice that the people writing SQL transformations had none of the tooling software engineering had taken for granted for thirty years, and give it to them in a form they would actually adopt. The barrier to entry is a SELECT statement. That is why it won.

The competition in 2026 is real — SQLMesh from below, warehouse-native tooling from the side, and dbt’s own Fusion rewrite from within. But the default answer for “we have SQL transformations in a warehouse and more than one person cares about them” is still dbt, and it is not close.

What to take from this chapter

• The deciding question: is most of your logic SQL over tables already in one warehouse? Yes means dbt.
• SQLMesh is the serious competitor — it parses SQL rather than templating it, giving column-level lineage, virtual environments and smarter change detection. dbt wins on ecosystem and hiring. Fusion is aimed at the same gap; check both.
• Spark’s remaining territory is very large data with non-SQL logic, and it is narrower than it was.
• Orchestrators are complements, not competitors. Add one when dbt is one step among many.
• Warehouse-native tooling is real and composes with dbt rather than replacing it; dbt keeps portability, testing and documentation.
• Stored procedures and GUI ETL both lose on the same axis: you cannot treat them as software.
• dbt has no streaming story and is not trying to. Very few requirements genuinely need one.
• Building it yourself means rebuilding dbt, worse.
• dbt is not technically remarkable. It won by giving SQL authors the tooling software engineering already had, at a barrier to entry of one SELECT.

Next: the cheat sheet — commands, selectors, configs and Jinja, on one page.