6. A change traced

Everything so far has been structure. This chapter follows one row change through all of it.

BEGIN;
UPDATE orders SET status = 'shipped', updated_at = now()
 WHERE id = 4471;
COMMIT;

   PostgreSQL 16, table public.orders
   REPLICA IDENTITY FULL
   Debezium 3.x on Kafka Connect
   Avro + Schema Registry
   topic cdc.public.orders, 12 partitions

t=0 — the transaction

The application’s UPDATE runs. PostgreSQL’s write path does its usual work:

   1  find the heap tuple for id=4471
   2  write a NEW tuple version with the
      new values (MVCC — nothing is
      overwritten in place)
   3  mark the old version dead as of this
      transaction
   4  APPEND WAL RECORDS describing both

   WAL, at LSN 0/1A2B3C4D

   ┌─────────────────────────────────────┐
   │ XLOG_HEAP_UPDATE                    │
   │   relation 16384 (public.orders)     │
   │   old tuple: page 4291, offset 12   │
   │   new tuple: page 4291, offset 88   │
   │   old tuple IMAGE (because REPLICA  │
   │     IDENTITY FULL)                  │
   │   new tuple image                   │
   └─────────────────────────────────────┘

REPLICA IDENTITY FULL is why the old row’s contents are in the WAL at all. With the default, only id=4471 would be recorded, and chapter 4’s before field would be nearly empty. This is decided here, at write time, not by Debezium.

t=0.4 ms — COMMIT

   ┌─────────────────────────────────────┐
   │ XLOG_XACT_COMMIT   txid 8841207     │
   │   LSN 0/1A2B3C9F                    │
   └─────────────────────────────────────┘

   WAL flushed to disk (fsync)
   → the transaction is DURABLE
   → COMMIT returns to the application

The application is now done. It made one call, to one system, in one transaction. Whatever happens next in this chapter, the change is committed and will eventually reach every consumer. That is chapter 1’s entire argument, and this is the moment it is satisfied.

t=0.4 ms — logical decoding wakes up

A walsender process is attached to the replication slot debezium_orders. It reads WAL as it is written.

But it does not emit anything yet.

   THE REORDER BUFFER

   WAL is written in the order operations
   HAPPEN, interleaved across concurrent
   transactions:

     txid 8841207  UPDATE orders
     txid 8841208  INSERT audit
     txid 8841207  UPDATE orders
     txid 8841209  UPDATE users
     txid 8841208  COMMIT     ← 8841208 emits
     txid 8841207  COMMIT     ← 8841207 emits
     txid 8841209  ROLLBACK   ← discarded

   Logical decoding BUFFERS each transaction
   and emits it ONLY AT COMMIT, in COMMIT
   ORDER.

Two consequences that matter operationally.

A transaction produces nothing until it commits. An hour-long transaction is invisible for an hour, then arrives all at once. Pipeline lag graphs show a flat line and then a spike, and nothing is wrong.

A large transaction is buffered on the source. PostgreSQL holds it in memory up to logical_decoding_work_mem and then spills to disk on the database server. A single transaction updating ten million rows is a large amount of temporary disk on your production database, at decode time. Chapter 14 has this as a failure mode.

And rolled-back transactions never appear at all — which is correct and worth knowing, because it means CDC shows you committed reality only.

t=0.5 ms — pgoutput

The decoding plugin serialises the buffered transaction into PostgreSQL’s logical replication protocol.

   BEGIN   (txid 8841207, LSN 0/1A2B3C9F)

   RELATION  id 16384, public.orders
             columns: id, customer_id,
                      status, amount,
                      updated_at
             ← sent once, when the schema is
               first seen or changes

   UPDATE  relation 16384
           OLD TUPLE: (4471, 812, 'pending',
                       149.99, '...')
           NEW TUPLE: (4471, 812, 'shipped',
                       149.99, '...')

   COMMIT  (LSN 0/1A2B3C9F)

Values are text at this layer. Type interpretation happens in Debezium, using the relation metadata.

Note that only tables in the publication are decoded and sent. Filtering happens on the database side, which is why publication.autocreate.mode=filtered (chapter 2) matters — it keeps the volume down before it reaches the network.

t=0.6 ms — Debezium receives it

The connector holds a streaming replication connection. Its loop receives the protocol messages.

   1  BEGIN     → start a transaction context
   2  UPDATE    → look up the schema for
                  relation 16384 in the
                  in-memory schema cache
               → convert text values to typed
                 values
   3  build the CHANGE EVENT (chapter 4)
   4  COMMIT    → the transaction's events
                  are complete

{
  "before": {"id":4471,"customer_id":812,"status":"pending",
             "amount":"149.99","updated_at":1757400000000000},
  "after":  {"id":4471,"customer_id":812,"status":"shipped",
             "amount":"149.99","updated_at":1757413200000000},
  "source": {"connector":"postgresql","name":"cdc","db":"app",
             "schema":"public","table":"orders",
             "ts_ms":1757413200123,"txId":8841207,
             "lsn":439108255,"snapshot":"false"},
  "op": "u",
  "ts_ms": 1757413200456
}

source.ts_ms is 1757413200123 — the database commit time. Top-level ts_ms is 1757413200456 — when Debezium built the event. 333 ms apart, and that gap is the pipeline lag chapter 13 tells you to graph.

The key:

{ "id": 4471 }

t=0.7 ms — transforms

transforms=unwrap,route
transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
transforms.unwrap.add.fields=op,source.ts_ms
transforms.route.type=org.apache.kafka.connect.transforms.RegexRouter
transforms.route.regex=cdc\\.public\\.(.*)
transforms.route.replacement=$1

   unwrap  envelope → the `after` row, flat,
           plus __op and __source_ts_ms
   route   topic cdc.public.orders → orders

Chapter 9 covers these properly. They run in order, in the connector’s own thread, per record.

t=0.8 ms — the converter

Avro, with a schema registry.

   1  compute the schema of this record
   2  is it in the local schema cache?
        yes → use the cached ID
        no  → POST to the registry
            → registry checks COMPATIBILITY
              against the subject's previous
              versions
            → returns a schema ID
   3  serialise:
        magic byte 0x00
        schema ID  (4 bytes)
        Avro binary payload

   JSON with embedded schema   2,847 bytes
   JSON, schemas disabled        312 bytes
   AVRO + registry                94 bytes

Thirty times smaller than the default. Chapter 4 said the embedded-schema default is a trap; this is the number.

The registry call happens once per schema, not per record. But a registry that is down blocks the connector on the first new schema it sees — a real coupling worth knowing about.

t=1.0 ms — the producer

   partition = hash({id: 4471}) % 12  =  7

   → EVERY change to order 4471 goes to
     partition 7, forever
   → ordering per row is guaranteed
   → out-of-order across DIFFERENT rows is
     possible and expected

This is why the key matters (chapter 4). Per-row ordering is a consequence of partitioning by primary key, and nothing else provides it.

producer.override.acks=all
producer.override.enable.idempotence=true
producer.override.compression.type=zstd
producer.override.linger.ms=20

acks=all is not optional here. Debezium is about to tell PostgreSQL that this change is safely handled; if Kafka has not durably stored it, that statement is a lie and the change is lost when a broker fails.

linger.ms batches records. A little latency for a lot of throughput.

t=4 ms — Kafka acknowledges

   record written to partition 7,
   offset 1,204,338
   replicated to 3 in-sync replicas
   → the producer callback fires

t=4 ms — Connect records its offset

   → connect-offsets topic

   key:   ["orders-cdc", {"server": "cdc"}]
   value: {"lsn": 439108255,
           "txId": 8841207,
           "ts_usec": 1757413200123456,
           "lsn_commit": 439108255}

Written periodically, not per record — offset.flush.interval.ms, commonly 60 seconds. This is the source of at-least-once delivery, and chapter 7 is about exactly this gap.

t=4 ms — and back to PostgreSQL

The step that closes the loop, and the one that matters most.

   Debezium → PostgreSQL, on the replication
   connection:

     STANDBY STATUS UPDATE
       "confirmed_flush_lsn = 0/1A2B3C9F"

   PostgreSQL:
     → advances the slot
     → WAL SEGMENTS BEFORE THAT LSN MAY NOW
       BE RECYCLED

   THE FULL LOOP

   Postgres ──WAL──▶ Debezium ──▶ Kafka
       ▲                              │
       │                              │ ack
       │      confirm LSN             ▼
       └──────────────────────── Debezium

The slot advances only after Kafka has acknowledged. That is the correctness guarantee — nothing is forgotten by PostgreSQL until it is durable somewhere else.

And it is exactly why chapter 2’s danger exists. Break any link in that loop — Debezium stops, Kafka is unavailable, the connector’s task fails — and the confirmation never arrives, the slot never advances, and PostgreSQL retains WAL forever.

   Kafka down for 6 hours
     → no acks
     → no LSN confirmations
     → slot frozen at 0/1A2B3C9F
     → pg_wal/ grows for 6 hours
     → the source database's disk fills

A Kafka outage becomes a PostgreSQL outage. This is the single most important sentence in the guide, and it follows directly from the loop above.

t=5 ms — a consumer

   consumer group `search-indexer`
   partition 7, offset 1,204,338

   → deserialise via the registry
   → { id: 4471, status: "shipped", ...,
       __op: "u" }
   → PUT /orders/_doc/4471
   → commit the Kafka offset

End to end: about five milliseconds, of which the database’s own commit was 0.4.

The whole path

   t=0.0ms  UPDATE writes WAL records
            ← REPLICA IDENTITY decides what
              the `before` image contains
   t=0.4ms  COMMIT. WAL fsynced.
            THE APPLICATION IS DONE.
   t=0.4ms  logical decoding: BUFFERED until
            COMMIT, then emitted in commit
            order
            ← long transactions emit nothing
              until they end; large ones
              SPILL TO DISK on the source
   t=0.5ms  pgoutput → logical replication
            protocol, filtered by publication
   t=0.6ms  Debezium builds the change event
            ← source.ts_ms vs ts_ms = lag
   t=0.7ms  SMTs: unwrap, route
   t=0.8ms  Avro + registry → 94 bytes
            (vs 2,847 for the JSON default)
   t=1.0ms  producer → partition by KEY
            → per-row ordering
   t=4.0ms  Kafka acks (acks=all)
   t=4.0ms  offset flushed to connect-offsets
            ← periodic. this is why delivery
              is AT-LEAST-ONCE.
   t=4.0ms  LSN CONFIRMED TO POSTGRES
            → the slot advances
            → WAL may be recycled
            ← BREAK THIS AND THE DATABASE
              FILLS UP
   t=5.0ms  the consumer indexes it

What to take from this chapter

The application makes one call, to one system. Once COMMIT returns, the change is in the log and will reach every consumer — with no window where the two disagree.
REPLICA IDENTITY is decided at write time, in the WAL. Debezium cannot recover a before image the database never recorded.
Logical decoding buffers each transaction and emits at commit, in commit order. Long transactions produce nothing until they end; large ones spill to disk on the database server; rolled-back ones never appear.
The publication filters on the database side, before the network.
source.ts_ms versus top-level ts_ms is your pipeline lag, measurable per event.
Avro with a registry was 94 bytes where the JSON default was 2,847.
Partitioning by the primary key is what gives per-row ordering. Nothing else does.
acks=all is mandatory, because Debezium is about to tell PostgreSQL the change is safe.
Offsets are flushed periodically, not per record — the source of at-least-once delivery.
The loop closes when Debezium confirms the LSN and the slot advances. Break any link — Debezium, Kafka, the task — and WAL accumulates until a Kafka outage becomes a PostgreSQL outage.

Next: offsets and delivery — what a restart actually replays, and how close to exactly-once you can get.

Written by Sepuri Sai Krishna