13. Tuning and operations

Running Debezium is mostly about one thing: the pipeline’s health is now the source database’s health. Everything else in this chapter is secondary to that.

The slot runbook

Chapters 2 and 6 explained the mechanism. This is what to do about it.

   THE FAILURE

   Debezium stops (crash, deploy, bad config,
   Kafka outage, a failed task nobody noticed)
     → LSN never confirmed
     → the slot never advances
     → PostgreSQL retains WAL
     → pg_wal/ grows
     → THE SOURCE DATABASE'S DISK FILLS
     → it stops accepting writes
     → your application is down

Prevention, in order

1. max_slot_wal_keep_size. The most important single setting.

max_slot_wal_keep_size = 64GB

Past that, PostgreSQL invalidates the slot rather than filling its disk.

   WITHOUT IT   pipeline outage → DATABASE
                OUTAGE
   WITH IT      pipeline outage → an
                invalidated slot and a
                re-snapshot

   → a bad day instead of a catastrophe.
     SET IT.

2. Alert on the slot, not on the connector.

SELECT slot_name, active, active_pid,
       pg_size_pretty(
         pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)
       ) AS retained_wal,
       pg_current_wal_lsn() - confirmed_flush_lsn AS lag_bytes
FROM pg_replication_slots;

   PAGE (wake someone up)
     active = false for > 5 minutes
     retained_wal > 25% of the WAL volume
   WARN
     retained_wal growing steadily for
     30 minutes

3. Heartbeats on every filtered connector.

heartbeat.interval.ms=10000
heartbeat.action.query=INSERT INTO debezium_heartbeat (ts) VALUES (now())

Chapter 2’s quiet-table problem. Without these, a low-traffic captured table on a busy database accumulates WAL invisibly.

4. Alert on task state, not connector state.

curl -s http://connect:8083/connectors/orders-cdc/status | jq -r '.tasks[].state'

A RUNNING connector with a FAILED task is a dead pipeline with a green dashboard, and on PostgreSQL it is a lit fuse.

The runbook

   WHEN THE SLOT IS NOT ADVANCING

   1  Is the TASK running?
        no → restart it, find out why
   2  Is Kafka accepting writes?
        no → fix Kafka. the slot cannot
             advance until it does.
   3  Is disk the immediate risk?
        → how long until full, at the
          current rate?
   4  If minutes:
        DROP THE SLOT.
        SELECT pg_drop_replication_slot('debezium_orders');
        → the database is saved
        → the connector cannot resume
        → a full re-snapshot is required
   5  Afterwards: why did nobody notice
      in step 1?

Step 4 is a real decision someone must be authorised to make at 3 a.m. Dropping the slot costs a re-snapshot; not dropping it costs the database. Write it down, decide in advance who decides, and make sure the DBA on call knows the slot exists.

The other databases have gentler versions. MySQL’s binlog expires regardless — the connector breaks, the database survives. MongoDB’s oplog window is the same shape. SQL Server’s capture tables grow but do not usually threaten the instance. PostgreSQL is the one that needs the runbook.

Read replicas

The question worth settling early.

   CAPTURE FROM A REPLICA?

   POSTGRESQL   yes, from a physical
                standby, with caveats —
                logical decoding on a standby
                requires recent versions and
                careful configuration, and a
                failover invalidates the slot.
                Many teams capture from the
                primary for this reason.

   MySQL        YES, and it is standard.
                a replica's binlog is
                equivalent. use GTIDs so a
                failover is survivable.

   MONGODB      yes — change streams work
                against secondaries.

For MySQL, capture from a replica. For PostgreSQL, verify what your version supports and understand the failover behaviour before committing to it.

Either way, the load Debezium adds is real but modest — it is reading a log, not scanning tables — except during a snapshot, which is a full table scan and should be pointed at a replica if at all possible.

Throughput

Chapter 3’s constraint: one task, which you cannot scale out. The dials that remain:

max.batch.size=8192
max.queue.size=32768
poll.interval.ms=500

producer.override.compression.type=zstd
producer.override.linger.ms=20
producer.override.batch.size=131072
producer.override.acks=all
producer.override.enable.idempotence=true

   max.batch.size     records per poll.
                      larger = better
                      throughput, more memory,
                      coarser offset flushes
   max.queue.size     the buffer between the
                      log reader and the
                      producer. must exceed
                      max.batch.size.
   linger.ms          batching at the
                      producer. 20ms buys a
                      lot of throughput for
                      little latency.

Serialisation format matters more than any of these. Chapter 6’s measurement: 2,847 bytes with the JSON default, 94 with Avro. Switching to Avro is usually the single largest throughput improvement available, and it is a configuration change.

After that, in order of value:

   1  AVRO + REGISTRY               ~10–30×
      less data
   2  CAPTURE FEWER TABLES/COLUMNS
      table.include.list,
      column.exclude.list
      → the data is never read at all
   3  compression.type=zstd
   4  larger batches and linger.ms
   5  KEEP SMTs THIN (chapter 9)
      they run in the one task
   6  a faster machine for the connector

There is no seventh option involving more tasks.

Monitoring

Debezium exposes JMX metrics under debezium.<connector>:type=connector-metrics,....

   SNAPSHOT
     SnapshotRunning / SnapshotCompleted
     RowsScanned                 progress
     SnapshotDurationInSeconds

   STREAMING
     MilliSecondsBehindSource    ← THE
                                   HEADLINE
                                   NUMBER
     TotalNumberOfEventsSeen
     NumberOfCommittedTransactions
     QueueRemainingCapacity      ← backpressure
     Connected                   ← is it
                                   attached?

   SCHEMA
     NumberOfChangedTables

   THE FIVE TO ALERT ON

   1  slot retained_wal            (from the
                                    DATABASE,
                                    not JMX)
   2  task state != RUNNING
   3  MilliSecondsBehindSource rising
   4  QueueRemainingCapacity near zero
   5  DLQ depth (chapter 9)

Number 1 is from the database and matters most. The others tell you the pipeline is unhealthy; that one tells you the database is at risk.

End-to-end latency is the number users care about, and it is now() − source.ts_ms measured at the consumer (chapter 4). Graph it; the difference between it and MilliSecondsBehindSource tells you whether lag is in Debezium or downstream.

Sizing

   CONNECT WORKER (or Debezium Server)
     CPU     moderate. serialisation and
             SMTs dominate.
     MEMORY  2–8 GB heap typical. the queue
             plus schema caches.
     DISK    almost none, unless Debezium
             Server stores offsets locally
     NETWORK the whole event volume, twice
             (in from the DB, out to Kafka)

   THROUGHPUT, ROUGHLY
     10k–50k+ events/sec per connector is
     routine with Avro
     JSON with embedded schemas can be an
     order of magnitude worse

Run each high-volume connector on a dedicated Connect worker if you can. A noisy sink connector on the same worker competes for threads and heap with the one thing that must not fall behind.

Failover and HA

   CONNECT     a worker dies → the connector
               is reassigned → resumes from
               connect-offsets
               → seconds of gap. fine.

   THE DATABASE FAILS OVER
     POSTGRES  ⚠ the slot does NOT follow.
               a promoted standby has no
               slot. → recreate it, and
               likely re-snapshot.
               (Newer PostgreSQL versions
               can synchronise slots to
               standbys — check yours; it
               changes this materially.)
     MySQL     ✓ with GTIDs, the connector
               resumes against the new
               primary.
               ✗ without them, file+position
               is meaningless. → re-snapshot.
     MONGO     ✓ resume tokens survive
               elections. this is what
               change streams are for.

“Use GTIDs” and “check whether your PostgreSQL can sync slots” are both database-failover decisions, not Debezium ones, and they need to be made before the failover rather than during it.

Upgrades

   DEBEZIUM
     property names change between major
     versions (database.server.name →
     topic.prefix; schema_only → no_data).
     read the migration notes.
     a renamed property is often SILENTLY
     IGNORED, which is worse than an error.

   KAFKA CONNECT
     rolling. connectors are reassigned.

   THE DATABASE
     a major-version upgrade may invalidate
     slots. plan for a re-snapshot, or a
     careful slot recreation.
     ← the one people forget

Test upgrades against a copy with a real snapshot, not an empty table. Most upgrade problems appear during snapshotting or schema handling, neither of which a smoke test exercises.

Testing

   ✓ TEST THE SNAPSHOT AT PRODUCTION SCALE
     the flood (chapter 5) is invisible in a
     small environment, and it is what takes
     downstream systems down.

   ✓ TEST A RESTART MID-SNAPSHOT
     initial snapshots do not resume.
     find out now.

   ✓ TEST DUPLICATE HANDLING
     kill the connector, let it replay,
     verify the sink converges.

   ✓ TEST A SCHEMA CHANGE
     add and drop a column with the
     pipeline running.

   ✓ TEST A KAFKA OUTAGE
     watch the slot grow. confirm the alert
     fires. run the runbook.
     ← the most valuable test here, and
       almost nobody does it

The organisational part

The advice that matters most and is least technical.

   □ THE DBA TEAM KNOWS THE PIPELINE EXISTS
   □ slot alerts go to the DATABASE on-call,
     not only to the data team
   □ the runbook is written and someone is
     authorised to drop the slot
   □ migrations on captured tables require
     sign-off (chapter 11)
   □ there is a list of who consumes which
     topic
   □ someone owns end-to-end latency

Debezium creates a dependency between two teams that usually do not share an on-call rotation. The data team runs the pipeline; the database team suffers when it stops. Every serious CDC incident in practice has that gap somewhere in its causes.

What to take from this chapter

• The pipeline’s health is the source database’s health. Everything else is secondary.
• Set max_slot_wal_keep_size. It converts a database outage into a re-snapshot.
• Alert on the slot from the database side, on task state (not connector state), on MilliSecondsBehindSource, on queue capacity and on DLQ depth.
• Set heartbeats on every filtered connector, or a quiet captured table accumulates WAL invisibly.
• Write the runbook, including who may drop the slot at 3 a.m.
• Capture from a replica where you can — standard for MySQL, version-dependent for PostgreSQL.
• Throughput cannot be scaled by adding tasks. Avro instead of JSON is usually the largest single win; capturing fewer tables and columns is next.
• MilliSecondsBehindSource versus end-to-end latency tells you whether lag is Debezium’s or downstream’s.
• Database failover is the dangerous one: PostgreSQL slots historically do not follow a promotion, MySQL needs GTIDs, MongoDB resume tokens survive.
• Renamed properties across Debezium versions are often silently ignored.
• Test the snapshot at production scale, and test a Kafka outage. Almost nobody does the second, and it is the most valuable.
• The DBA team must be part of this. Every serious CDC incident has that gap in its causes.

Next: failure modes — the specific ways CDC pipelines go wrong, including the two that take down the database.