1. What Debezium is, and why it exists

Ten guides on this site mention Debezium. Kafka’s chapter 12 calls it “the most valuable Connect use case” and immediately warns that it can fill your database’s disk. Flink, Iceberg, ClickHouse and Elasticsearch all describe it as the thing feeding them.

This is the guide about the thing they are all fed by.

The problem

Your application writes an order to PostgreSQL. Four other systems need to know.

   the search index needs the new order
   the analytics warehouse needs it
   the fraud service needs it
   the customer's email needs sending

The obvious solution is to write to all of them:

def create_order(order):
    db.insert(order)              # 1
    search.index(order)           # 2
    kafka.publish(order)          # 3
    email.send(order)             # 4

This is the dual-write problem, and it is broken.

   step 1 succeeds, step 2 fails
     → the order exists, search does not know

   step 1 succeeds, the process dies
     → nothing downstream ever hears

   step 2 succeeds, step 1 is ROLLED BACK
     → search has an order that does not exist

   steps 1 and 3 succeed, in that order,
   but arrive at Kafka out of order
     → downstream sees the wrong sequence

There is no ordering of those four lines that is correct, because there is no transaction spanning a database and a message broker. Every arrangement leaves a window where one has committed and the other has not.

Distributed transactions across both — two-phase commit — exist and are avoided for good reasons: they are slow, they hold locks across a network, and a coordinator failure blocks everybody.

The idea

The database already writes down every change it makes. Read that.

   PostgreSQL   the write-ahead log
   MySQL        the binary log
   MongoDB      the oplog
   SQL Server   the transaction log
   Oracle       the redo log

Every one of those exists so the database can recover from a crash and feed its own replicas. They are a complete, ordered, durable record of every committed change — and they were there all along.

   ┌──────────────┐
   │ APPLICATION  │  ONE write. one transaction.
   └──────┬───────┘
          ▼
   ┌──────────────┐
   │ POSTGRESQL   │
   │              │
   │   WAL ───────┼──▶ DEBEZIUM ──▶ Kafka
   │              │                   ├─▶ search
   │              │                   ├─▶ warehouse
   └──────────────┘                   ├─▶ fraud
                                      └─▶ email

The application does one thing: commit. If the commit succeeds, the change is in the log, and therefore it will reach every consumer. If it fails, nothing is in the log and nothing propagates. There is no window.

This is the whole argument, and it is a good one.

Why not just poll?

The alternative most teams try first.

SELECT * FROM orders WHERE updated_at > :last_seen;

   ✗ MISSES DELETES
     a deleted row does not appear in any
     query. it just stops being there.
     ← the big one

   ✗ MISSES INTERMEDIATE STATES
     a row updated three times between polls
     is seen ONCE, in its final state. the
     two intermediate versions never existed
     as far as you know.

   ✗ NEEDS A RELIABLE updated_at
     every writer must maintain it. one that
     forgets is invisible.

   ✗ LOAD ON THE SOURCE
     a query every few seconds, against
     production, forever. and the index on
     updated_at is one more index to maintain.

   ✗ CLOCK AND BOUNDARY PROBLEMS
     a transaction that commits with an
     earlier updated_at than one already
     read is skipped permanently.
     ← this one is subtle and it happens

   ✗ LATENCY IS THE POLL INTERVAL

That last-mentioned boundary problem is worth understanding, because it is the one that silently loses data. Two transactions start at 10:00:00. One commits at 10:00:01, the other at 10:00:03. A poll at 10:00:02 with updated_at > 10:00:00 reads the first. The next poll uses updated_at > 10:00:01 — and the second transaction, whose updated_at was set at 10:00:00 when it started, is never seen.

Reading the log has none of these problems, because the log records what actually happened, in commit order, including deletes.

Why not triggers?

CREATE TRIGGER orders_audit AFTER INSERT OR UPDATE OR DELETE ON orders
  FOR EACH ROW EXECUTE FUNCTION write_to_audit_table();

Better than polling — it catches deletes and every intermediate state.

   ✗ IT RUNS INSIDE YOUR TRANSACTION
     every write now costs more. the audit
     insert is part of the commit.

   ✗ IT DOUBLES THE WRITE VOLUME
     and the WAL volume, and the disk

   ✗ TRIGGERS ARE HARD TO MAINTAIN
     one per table, invisible in the
     application code, and a source of
     surprising behaviour for years

   ✗ IT DOES NOT SOLVE DELIVERY
     something still has to get the audit
     table out to Kafka. you have moved the
     problem, not removed it.

Log-based CDC is outside the transaction entirely. The application’s write path is unchanged. That is the property triggers cannot offer.

What Debezium actually is

   A SET OF CONNECTORS that read database
   logs and emit a structured change event
   per row change.

   ✓ PostgreSQL, MySQL, MongoDB, SQL Server,
     Oracle, Db2, and more
   ✓ one event per row change: before, after,
     source metadata, operation
   ✓ ordered, per table and per key
   ✓ resumable from where it left off
   ✓ an initial snapshot, then live streaming
   ✓ Apache 2.0

It is not a broker, a database, or a transformation engine. It reads a log and produces events; something else carries and consumes them.

Most commonly that something is Kafka, via Kafka Connect. Chapter 3 covers the three other ways to run it, which matter more than people realise.

Where it came from

2016 — started at Red Hat by Randall Hauch. Built for Kafka Connect from the beginning.
2017–2021 — connectors broaden well beyond MySQL and PostgreSQL; Debezium Server arrives so Kafka stops being mandatory.
2022 — Debezium 2.0. Incremental snapshots (chapter 5) land, which is the single largest usability improvement in the project’s history.
2024 — Debezium 3.0, requiring a newer Java and Kafka. The project also moved to the Commonhaus Foundation, giving it neutral governance rather than a single vendor’s.

Debezium has become the de facto standard for open-source CDC, to the extent that several commercial products and cloud services embed it rather than compete with it. Flink CDC uses it underneath; so do a number of managed pipelines.

What it is not

   ✗ NOT A BROKER
     it produces events. Kafka, Pulsar,
     Kinesis or a file carries them.

   ✗ NOT AN ETL TOOL
     no transformation beyond simple
     per-record SMTs (chapter 9). complex
     work belongs in Flink or a warehouse.

   ✗ NOT A REPLICATION TOOL
     it will not keep a second Postgres in
     sync for you. use the database's own
     replication for that. CDC is for
     getting changes to DIFFERENT systems.

   ✗ NOT ZERO-IMPACT ON THE SOURCE
     the most important caveat in this
     guide. chapters 2 and 13.

   ✗ NOT A WAY TO AVOID SCHEMA DESIGN
     your table schema becomes your event
     schema, and every consumer now depends
     on it. chapter 11.

That last one is more consequential than it looks. CDC publishes your internal database schema as a public interface. A column rename that was a private refactoring is now a breaking change for four downstream teams. Chapters 10 and 11 are largely about living with that, and the outbox pattern is the escape from it.

The cost, stated up front

Three things, and the rest of the guide keeps returning to them.

1. It couples you to the source database’s operations. A PostgreSQL replication slot that stops being consumed prevents the database from deleting WAL segments, and the disk fills. This takes down the source database, not merely the pipeline. Kafka’s guide flags it; chapter 13 covers it properly. This is the single most dangerous thing about running Debezium.

2. It needs privileged database access. Replication privileges, a replication slot, binlog_format=ROW, or CDC enabled per table. Your DBA is now part of this project.

3. It publishes your schema. See above.

Against those: exactly one thing that no alternative offers. A guaranteed, ordered, complete, replayable stream of every change, with no window in which the database and the downstream disagree.

For a large class of systems that is worth all three costs. For a small one it is not, and chapter 15 is honest about which is which.

What to take from this chapter

The dual-write problem has no correct ordering. There is no transaction spanning a database and a broker, so every arrangement leaves a window.
Debezium’s idea: the database already writes down every change, for its own recovery and replication. Read that log.
The application does one thing — commit. If it succeeds the change is in the log and will reach every consumer; if it fails nothing propagates.
Polling misses deletes and intermediate states, loads the source, and has a subtle commit-ordering boundary that silently loses rows.
Triggers catch everything but run inside your transaction, double the write volume, and still do not solve delivery.
Log-based CDC is outside the transaction entirely — the application write path is unchanged.
Debezium is connectors, not a broker. Kafka is the usual carrier, and chapter 3 covers the alternatives.
It came from Red Hat in 2016, gained incremental snapshots in 2.0, and now sits under the Commonhaus Foundation. Several commercial products embed it.
Three costs: operational coupling to the source database, privileged access, and publishing your internal schema as a public interface.
The replication slot filling the source database’s disk is the most dangerous thing here, and it is covered in chapters 2 and 13.

Next: the log, per database — WAL, binlog, oplog and the rest, and what each one requires of you.

Written by Sepuri Sai Krishna