14. Failure modes

CDC failures split cleanly. A few take down the source database. Most are silent — the pipeline is green and the sink has been wrong since March.

1. The slot that filled the database

What it looks like. The application is down. PostgreSQL is refusing writes. pg_wal/ is 900 GB on a 1 TB volume. Nothing was deployed.

Why. Chapters 2, 6 and 13. Debezium stopped — a failed task, a Kafka outage, a bad config push — the LSN was never confirmed, and PostgreSQL kept every WAL segment since.

   THE ORDER OF EVENTS

   Kafka has a rough afternoon
     → no acks → no LSN confirmations
     → slot frozen
     → WAL accumulates for six hours
     → THE DATABASE STOPS

A Kafka outage became a PostgreSQL outage.

The fix. max_slot_wal_keep_size, alerts on pg_replication_slots from the database side, and a runbook that authorises someone to drop the slot. Chapter 13 has all three.

This is the worst thing in this guide and it is entirely preventable with one setting.

2. The green dashboard

What it looks like. A sink has been stale for two days. The Connect dashboard says RUNNING. Nobody was paged.

Why. Kafka’s chapter 12 flags it and it bears repeating: a connector reports RUNNING while its only task is FAILED.

# what you must alert on
curl -s http://connect:8083/connectors/orders-cdc/status | jq -r '.tasks[].state'

The fix. Alert on task state. And on PostgreSQL, this failure is not merely stale data — it is failure mode 1 in slow motion.

3. The heartbeat nobody set

What it looks like. WAL grows steadily on a busy database. The connector is healthy, lag is zero, and the captured table gets one write a day.

Why. Chapter 2. Debezium only confirms its LSN when it processes something. A quiet captured table on a busy database means nothing to process — while every other table’s activity fills the WAL the slot is holding.

The fix.

heartbeat.interval.ms=10000
heartbeat.action.query=INSERT INTO debezium_heartbeat (ts) VALUES (now())

Set both on every connector with a filtered table list. This is confusing, common, and one configuration line.

4. The binlog that expired

What it looks like. After a long weekend outage, the MySQL connector will not start: “The connector is trying to read binlog starting at … but this is no longer available.”

Why. MySQL retains the binlog by time, regardless of who has read it. The recorded position no longer exists.

The fix, which is prevention. binlog_expire_logs_seconds longer than your worst realistic outage plus detection time. Seven days is a common, sane floor.

Recovery is a full re-snapshot, which is failure mode 6 waiting to happen.

5. The schema history topic somebody cleaned up

What it looks like. A MySQL connector will not start after a restart: “Could not find existing database schema history.” It ran fine yesterday.

Why. Chapter 11. A cluster-wide retention or compaction policy was applied, and the schema history topic was included.

The fix, which is prevention. Single partition, replication factor 3, retention.ms=-1, never compacted, and documented as untouchable wherever your topic policies live.

This topic is the most commonly damaged piece of Debezium state, and it is always damaged by someone who did not know it existed.

6. The snapshot that took down everything downstream

What it looks like. A snapshot is started. Within minutes the Elasticsearch cluster is at 100% CPU, the warehouse is queuing, and consumer lag is measured in days.

Why. Chapter 5. Steady state is 200 events/second. The snapshot produces 40,000.

The fix. Test the snapshot’s downstream impact before running it in production. Use additional-conditions to snapshot in slices, pause when consumers lag, or snapshot into a topic nothing consumes yet.

Steady-state throughput tells you nothing about snapshot throughput.

7. The snapshot that never finished

What it looks like. An initial snapshot of a 400-million-row table has been running for eleven hours. The connector dies at 380 million. It starts again from zero.

Why. Chapter 5. Initial snapshots do not resume.

The fix. Incremental snapshots — resumable, pausable, per table. If you are still doing initial snapshots on very large tables, that is the thing to change.

8. The empty before

What it looks like. A consumer that needs to know what a row looked like before an update gets {"id": 4471} and nothing else.

Why. Chapter 2. REPLICA IDENTITY DEFAULT on PostgreSQL, or binlog_row_image=MINIMAL on MySQL. The database never wrote the old values to the log.

The fix. ALTER TABLE orders REPLICA IDENTITY FULL, at the cost of more WAL.

Debezium cannot recover information the database did not record, and this is the most common “Debezium is broken” report that is not.

9. The sink full of deleted rows

What it looks like. A search index returns orders that were cancelled months ago. Counts do not match the source. No errors anywhere.

Why. One of three, all silent:

   ✗ transforms.unwrap delete handling =
     drop  (chapter 9)
   ✗ tombstones.on.delete=false with a
     compacted topic  (chapter 4)
   ✗ the sink connector's delete.enabled
     is false  (chapter 12)

The fix. Choose delete semantics explicitly, and reconcile counts between source and sink on a schedule (chapter 12). That reconciliation is the only thing that catches this class of failure.

10. The customers who got the email twice

What it looks like. A deploy restarts the connector. Four thousand customers receive a duplicate shipping notification.

Why. Chapter 7. Offsets are flushed periodically; a restart replays everything since the last flush. This is by design and it is not going to stop.

The fix. Deduplicate on source.lsn, use an idempotency key on the email API, or — best — drive the effect from state rather than from events: “email everyone whose status is shipped and email_sent is false”.

11. The welcome email to every customer

What it looks like. A snapshot runs. Every customer who has ever existed receives a welcome email.

Why. Chapter 4. The consumer treated op values with an after field as creations. Snapshot rows arrive as op = "r", not "c".

The fix. Handle r explicitly, everywhere. Any consumer with side effects must distinguish r from c, and it should be in the code review checklist for CDC consumers.

12. The migration that emitted 400 million events

What it looks like. A routine ALTER TABLE at 2 a.m. By 2:15 the pipeline is emitting at forty times normal rate and every consumer is drowning.

Why. Chapter 11. A rewriting ALTER — adding a NOT NULL column with a default, changing a type — rewrites every row, and every rewritten row is a change in the log.

The fix. Know which ALTER forms rewrite on your database version. Schedule rewriting migrations like you would a snapshot: with notice, with consumers warned, and ideally at a quiet time.

The related version: a DELETE FROM ... WHERE that removes ten million rows emits ten million delete events. Batch large deletes, or expect the flood.

13. The transaction that spilled to disk

What it looks like. The database’s disk usage spikes during a nightly batch job. It is not WAL. Logical decoding temporary files.

Why. Chapter 6. Logical decoding buffers a whole transaction until commit, in memory up to logical_decoding_work_mem, then spills to disk on the database server. A single transaction updating ten million rows is a lot of temporary disk.

The fix. Batch large writes into smaller transactions. This is good practice regardless; CDC makes it load-bearing.

The related symptom is a lag graph that is flat and then spikes — a long transaction produces nothing until it commits, and nothing is wrong.

14. The dates fifty thousand years in the future

What it looks like. A dashboard shows records dated year 57,000. No errors.

Why. Chapter 4. Temporal values arrive as microseconds since epoch under the adaptive precision modes; the consumer assumed milliseconds. The multiplication by a thousand produces a valid timestamp, so nothing fails.

The fix. Know your time.precision.mode, write it down, and test the actual output for every column type before writing consumers. The same applies to decimal.handling.mode=precise producing base64 strings.

15. The primary key that changed

What it looks like. After a migration, a compacted topic has two entries for every row, the sink has duplicates, and ordering guarantees are gone.

Why. Chapter 8. The key determines the partition. Change the primary key and every row hashes somewhere new, while its history sits in the old partition under the old key.

The fix. Treat a primary key change as a new table: a new topic, a fresh snapshot, and a migration for consumers. There is no in-place fix.

16. Two connectors, one slot

What it looks like. Both connectors flap — connecting, disconnecting, erroring. Neither makes progress.

Why. Two connectors configured with the same slot.name, or the same MySQL database.server.id. PostgreSQL allows one consumer per slot; MySQL replicas with identical server ids disconnect each other.

The fix. Unique slot names and unique server ids, enforced by naming convention. The classic cause is copying a connector configuration to create a second one.

17. The DLQ nobody read

What it looks like. errors.tolerance=all was set to stop a bad record breaking the pipeline. Six months later the DLQ has 1.4 million records and the sink is missing them all.

Why. Chapter 9. all converts a loud failure into a silent one.

The fix. errors.tolerance=all plus an alert on DLQ depth plus a runbook. Set errors.deadletterqueue.context.headers.enable=true, or DLQ records have no explanation attached.

The pattern

   TAKES DOWN THE DATABASE
     the slot (1, 2, 3)
     decoding spills (13)
     → PostgreSQL only, and preventable
       with configuration

   BREAKS THE PIPELINE, LOUDLY
     expired binlog (4)
     lost schema history (5)
     duplicate slot/server id (16)
     → recoverable, usually with a
       re-snapshot

   SILENTLY WRONG
     dropped deletes (9)
     duplicate side effects (10, 11)
     empty before (8)
     wrong timestamps (14)
     the unread DLQ (17)
     → months. and only reconciliation
       finds them.

The defences are three. Alert on the slot from the database side. Make every consumer idempotent and explicit about r and d. And reconcile the sink against the source on a schedule — because for the entire third category, that is the only thing that will ever tell you.

What to take from this chapter

• An unadvanced replication slot takes down PostgreSQL. max_slot_wal_keep_size, database-side alerts, and a runbook that authorises dropping the slot.
• A RUNNING connector with a FAILED task is a dead pipeline with a green dashboard — and on PostgreSQL, a lit fuse.
• Set heartbeats on filtered connectors, or a quiet table on a busy database fills the WAL invisibly.
• MySQL binlog retention bounds how long you may be down; the schema history topic must never be compacted or expired.
• Snapshot throughput is nothing like steady-state throughput, and initial snapshots do not resume.
• An empty before is a database setting, not a Debezium bug.
• Dropped deletes leave sinks diverging silently for years. Only reconciliation finds it.
• Duplicates are guaranteed. r is not c. Both cause user-visible incidents in consumers with side effects.
• A rewriting ALTER TABLE emits an event per row, and a large DELETE emits one per deleted row.
• Large transactions spill logical decoding to disk on the database server.
• Microseconds versus milliseconds produces valid-looking wrong dates, not errors.
• Changing a primary key is a new table. Duplicate slot names and server ids come from copied configurations.
• errors.tolerance=all without a DLQ alert is silent data loss.

Next: when to use CDC and when not — against polling, triggers, application events and the managed alternatives.