4. The change event

Everything Debezium produces is one shape. Learn it once and every connector’s output is readable.

The envelope

{
  "before": {
    "id": 4471,
    "customer_id": 812,
    "status": "pending",
    "amount": 149.99,
    "updated_at": 1757400000000000
  },
  "after": {
    "id": 4471,
    "customer_id": 812,
    "status": "shipped",
    "amount": 149.99,
    "updated_at": 1757413200000000
  },
  "source": {
    "version": "3.0.0.Final",
    "connector": "postgresql",
    "name": "cdc",
    "ts_ms": 1757413200123,
    "snapshot": "false",
    "db": "app",
    "schema": "public",
    "table": "orders",
    "txId": 8841207,
    "lsn": 439108236,
    "sequence": "[\"439108100\",\"439108236\"]"
  },
  "op": "u",
  "ts_ms": 1757413200456,
  "transaction": null
}

   before   the row BEFORE. null for inserts.
   after    the row AFTER. null for deletes.
   source   where and when it happened,
            in the DATABASE
   op       what happened
   ts_ms    when DEBEZIUM processed it
   transaction  optional grouping metadata

The operation codes

   c   CREATE     an insert
       before: null      after: the new row

   u   UPDATE
       before: the old row (see below)
       after:  the new row

   d   DELETE
       before: the deleted row
       after:  null

   r   READ       from the initial SNAPSHOT
       before: null      after: the row
       ← NOT an insert. the row already
         existed. chapter 5.

   t   TRUNCATE   the table was truncated
       both null

   m   MESSAGE    a logical message, not a
                  row change

r versus c is the distinction consumers get wrong. During a snapshot, every existing row arrives as r — “this row exists”, not “this row was just created”. A consumer that sends a welcome email on c is fine; one that sends it on any event with an after will email your entire customer base the first time the connector snapshots.

The before field is conditional

Chapter 2’s point, arriving where you see it.

   POSTGRESQL, REPLICA IDENTITY DEFAULT
     "before": { "id": 4471 }
                ▲ the primary key. that is all.

   POSTGRESQL, REPLICA IDENTITY FULL
     "before": { the entire old row }

   MySQL, binlog_row_image = MINIMAL
     only the changed columns, plus the key

   MySQL, binlog_row_image = FULL
     the entire old row

If before is nearly empty and you expected a full row, the problem is in the database configuration, not in Debezium. This is the single most common “Debezium is broken” report that turns out not to be.

The key

Every event has a key as well as a value.

{ "id": 4471 }

The key is the table’s primary key, and it is what makes everything downstream work:

   ✓ all changes to one row go to ONE Kafka
     partition → ORDERED per row
   ✓ log compaction keeps the latest state
     per row (Kafka chapter 10)
   ✓ upsert sinks know what to upsert on

A table with no primary key produces events with a null key. They round-robin across partitions, so ordering per row is lost and compaction is meaningless.

message.key.columns=public.orders:order_uuid

Set this for tables without a primary key, or accept that downstream cannot rely on ordering.

Tombstones

After a delete, Debezium emits a second record: the same key, a null value.

   1.  key={id:4471}  value={op:"d", before:{...}, after:null}
   2.  key={id:4471}  value=null      ← THE TOMBSTONE

This is entirely for Kafka log compaction. A null value is compaction’s delete marker; without it, a compacted topic keeps the deleted row’s last state forever. Kafka’s chapter 10 explains the mechanism.

tombstones.on.delete=true    # default

The trap: consumers that do not expect a null value. A deserialiser that assumes every record has a body throws on the tombstone, and the error appears one record after the delete rather than on it.

Turn tombstones off only if nothing consumes the topic compacted, and know that you are giving up compaction’s delete semantics.

The source block

More useful than it looks.

   ts_ms      when the DATABASE committed it
              ← the real event time
   snapshot   "true" | "last" | "false"
   txId/lsn   the database position
   sequence   ordering within the log
   db/schema/table   where it came from

Two timestamps, and they mean different things. The top-level ts_ms is when Debezium processed the event; source.ts_ms is when the database committed it. The gap between them is your pipeline lag, and it is the number to graph (chapter 13).

For event-time processing — Flink watermarks, for instance — source.ts_ms is the one you want.

Serialisation

   JSON, schemas EMBEDDED     the default
     every record carries its full schema
     ✗ enormous — often 10× the payload
     ✓ self-describing, no registry
     → fine for a demo, wrong for production

   JSON, schemas DISABLED
     value.converter.schemas.enable=false
     ✓ small, readable
     ✗ no type information at all

   AVRO + SCHEMA REGISTRY
     ✓ compact binary
     ✓ schemas versioned and enforced
     ✓ compatibility checking
     ← the production answer

   PROTOBUF / JSON SCHEMA + REGISTRY
     same benefits, different format

key.converter=io.confluent.connect.avro.AvroConverter
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://registry:8081

The default embedded-schema JSON is a trap for new deployments. It works, so nobody changes it, and then the topics are five times larger than they should be and every consumer parses a schema it already knows.

Use Avro with a registry unless you have a specific reason not to. Chapter 11 covers what the registry does for schema changes, which is the larger benefit.

Type mapping surprises

Where most of the real confusion lives.

Decimals

decimal.handling.mode=precise      # DEFAULT

"amount": "AJiWgA=="

A base64 string. precise encodes decimals as Kafka Connect’s Decimal type — bytes plus a scale — which JSON has no way to represent, so it base64-encodes them.

   precise   exact. base64 bytes in JSON.
             correct, and unreadable.
   double    a JSON number. LOSES PRECISION.
             wrong for money.
   string    "149.99". exact AND readable.
             ← usually the right answer

string is the pragmatic choice for most pipelines. Exact, human-readable, and every consumer can parse it. precise is correct if your consumers handle Connect’s Decimal type; double should never be used for money.

Temporal types

time.precision.mode=adaptive_time_microseconds   # a common default

"updated_at": 1757413200000000

Microseconds since epoch, as an integer. Not milliseconds — a consumer that assumes milliseconds gets a date fifty thousand years in the future, and it does not error.

   adaptive*   matches the column's precision.
               days / micros / nanos depending
               on the type. FLEXIBLE, and you
               must know which you got.
   connect     always Connect's own types —
               millisecond precision.
               ← predictable; loses sub-ms

connect is easier to consume; adaptive is more faithful. Whichever you pick, write it down, because “is this milliseconds or microseconds?” is a question your consumers will ask repeatedly.

Others worth knowing

   BOOLEAN in MySQL      tinyint(1) → integer,
                         not boolean
   ENUM                  a string
   JSON / JSONB          a string containing
                         JSON. parse it again.
   BYTEA / BLOB          base64 in JSON;
                         bytes in Avro
   UNSIGNED BIGINT       may not fit a signed
                         64-bit integer
   ARRAY (Postgres)      an array, if the
                         converter supports it
   custom types          often a string —
                         verify

Test the actual output for every column type before consumers are written. Twenty minutes with a console consumer saves a week of downstream surprises.

Transaction metadata

Off by default, and occasionally exactly what you need.

provide.transaction.metadata=true

   → a separate topic with BEGIN and END
     events per transaction, including the
     event count per table
   → every change event gains:

   "transaction": {
     "id": "8841207",
     "total_order": 3,
     "data_collection_order": 1
   }

This is how a consumer reconstructs transaction boundaries. An order and its line items written in one transaction arrive as separate events on separate topics; without this metadata there is no way to know they belonged together, and a consumer can observe the order before its items.

Enable it when downstream needs transactional consistency across tables. It costs an extra topic and some complexity, and most pipelines do not need it — but when you need it, nothing else provides it.

Unwrapping

Most consumers do not want the envelope. Chapter 9 covers the transform properly; the shape is:

transforms=unwrap
transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
transforms.unwrap.delete.handling.mode=rewrite
transforms.unwrap.add.fields=op,source.ts_ms

{
  "id": 4471,
  "status": "shipped",
  "amount": "149.99",
  "__op": "u",
  "__source_ts_ms": 1757413200123,
  "__deleted": "false"
}

The flat row, plus the metadata you asked for. This is what most sink connectors expect.

What to take from this chapter

• One envelope: before, after, source, op, ts_ms, transaction.
• op is c/u/d/r/t/m, and r means “read from snapshot”, not “created”. Confusing them is how you email your entire customer base.
• before is only complete if the database was configured for it — REPLICA IDENTITY FULL or binlog_row_image=FULL. An empty before is a database setting, not a bug.
• The key is the primary key, and it is what gives per-row ordering, compaction and upserts. Set message.key.columns for tables without one.
• Tombstones are null-valued records for Kafka compaction, and consumers that assume every record has a body will break on them.
• source.ts_ms is when the database committed; top-level ts_ms is when Debezium saw it. The gap is your lag.
• Default JSON with embedded schemas is a trap — use Avro with a registry.
• decimal.handling.mode=precise gives base64 in JSON. string is usually right; double is never right for money.
• Temporal values are often microseconds, not milliseconds, and getting it wrong produces plausible-looking wrong dates rather than errors.
• Test the real output for every column type before writing consumers.
• provide.transaction.metadata is the only way to reconstruct transaction boundaries across tables.

Next: snapshots — how the existing data gets out, and the incremental snapshot that changed how CDC is operated.