12. Consumers and sinks

Debezium produces a stream. This chapter is what reads it, and the one pattern they all share.

The pattern everything converges on

   UPSERT BY PRIMARY KEY, WITH A VERSION.

   for each event:
     op = c or u or r  →  upsert `after`,
                          if its version is
                          newer than what
                          we hold
     op = d            →  delete by key

Every sink below is a variation on those four lines, and it works because of three properties Debezium gives you: the event carries the full row, it carries the primary key, and it carries a monotonic position (source.lsn or equivalent).

The version check is what makes it robust:

   WITHOUT A VERSION
     upsert is idempotent — the same event
     twice is fine
     ✗ but a REPLAYED OLD event overwrites
       a newer value

   WITH A VERSION
     ignore any event whose source.lsn is
     not greater than the stored one
     ✓ safe under duplicates AND reordering

Chapter 7 made the case; this is where it is spent.

Sink connectors

The Kafka Connect answer: no code.

{
  "connector.class": "io.debezium.connector.jdbc.JdbcSinkConnector",
  "tasks.max": "4",
  "topics": "cdc.public.orders",
  "connection.url": "jdbc:postgresql://analytics:5432/warehouse",
  "insert.mode": "upsert",
  "primary.key.mode": "record_key",
  "primary.key.fields": "id",
  "delete.enabled": "true",
  "schema.evolution": "basic",
  "auto.create": "true"
}

   ⚠ delete.enabled=true REQUIRES tombstones
     (chapter 4). without them the sink
     never learns a row was deleted.

   ✓ tasks.max > 1 IS fine here — the sink
     is not the single-task constraint of
     chapter 3. sinks scale with partitions.

Sink connectors are the fastest way to get somewhere, and they are limited to what their configuration expresses. The moment you need a join, a lookup or conditional logic, you want a stream processor.

Common sinks: JDBC, Elasticsearch, Iceberg, S3, ClickHouse, MongoDB, Redis, Snowflake, BigQuery, HTTP.

Stream processors

When the sink connector is not enough.

   FLINK        the most capable. changelog
                streams are native, temporal
                joins against a CDC stream
                are a first-class feature.
                → the Flink guide

   KAFKA STREAMS  a library in your JVM
                service. KTable from a
                compacted CDC topic is
                exactly a materialised table.

   SPARK STRUCTURED STREAMING
                micro-batch. fine for
                minute-scale sinks into a
                lake.

Flink’s chapter 11 calls the temporal join “the join that makes CDC worth it”:

SELECT o.order_id, o.amount, u.tier
FROM  orders o
JOIN  users FOR SYSTEM_TIME AS OF o.proc_time AS u
  ON  o.user_id = u.id;

Each order is joined against the user row as it was at that moment, not as it is now. A CDC changelog of users is a versioned table, and that is what makes this expressible at all.

A CDC stream plus Flink is the standard shape for enrichment, and it is where the interesting work lives — deduplication, joining several tables back into a document, windowed aggregation.

Per-sink specifics

ClickHouse

CREATE TABLE orders (
    id          Int64,
    customer_id Int64,
    status      String,
    amount      Decimal(10,2),
    _version    UInt64,          -- source.lsn
    _deleted    UInt8            -- __deleted
) ENGINE = ReplacingMergeTree(_version, _deleted)
ORDER BY id;

ClickHouse’s guide names this as the standard CDC sink pattern. ReplacingMergeTree keeps the highest _version per sort key and drops rows flagged deleted — which is the upsert-with-version pattern implemented by the storage engine.

   ⚠ CONVERGENCE IS EVENTUAL.
     duplicates exist until a merge runs.
     use FINAL, or an argMax aggregation,
     when reading — or accept approximate
     answers.

That caveat catches people: the table is correct eventually, and a query run seconds after an update may see both versions.

Iceberg

MERGE INTO warehouse.orders t
USING cdc_batch s ON t.id = s.id
WHEN MATCHED AND s.__op = 'd' THEN DELETE
WHEN MATCHED                  THEN UPDATE SET *
WHEN NOT MATCHED              THEN INSERT *;

Iceberg’s chapter 12 is blunt about the cost: CDC into Iceberg is the highest-maintenance pattern in that guide. Equality deletes plus small files, continuously, both of which degrade reads.

   ✓ open format, any engine reads it
   ✓ schema evolution is genuinely safe
     (field ids — Iceberg chapter 6)
   ✗ YOU MUST RUN COMPACTION.
     not optional. a schedule, not a risk.

Iceberg is the best-behaved sink for schema changes and the worst for maintenance. Both are worth knowing before choosing it.

Elasticsearch

   ✓ upsert by document id = the primary key
   ✓ deletes work naturally
   ✗ TYPE CHANGES REQUIRE A REINDEX
     (chapter 11)
   ⚠ a nested document usually needs
     several tables joined first
     → Flink, not a sink connector

Elasticsearch’s chapter 15 covers the pairing. The recurring shape is that a search document is an aggregate — an order with its items and its customer — and assembling it needs a join that a sink connector cannot do.

A cache

   Debezium → Kafka → a consumer that
   invalidates or updates Redis

The most underrated CDC use case. Cache invalidation is famously hard because the application must remember to invalidate everywhere it writes. CDC makes it automatic: the row changed, therefore the cache entry is stale, regardless of which code path changed it.

Chapter 3’s embedded engine is a good fit here — one consumer, your own service, no broker needed.

A warehouse

   raw CDC → landing tables → dbt → marts

dbt is the natural next step: land the change stream, then model it. dbt snapshots and CDC solve overlapping problems — dbt’s chapter 6 captures history the source overwrites, and CDC captures it at the source. CDC is strictly better where you can have it, because it sees every intermediate state rather than daily samples.

Bootstrapping a new consumer

The question that comes up constantly.

   OPTION 1  READ THE COMPACTED TOPIC FROM
             THE BEGINNING
     the topic IS the table (chapter 8)
     ✓ no load on the source database
     ✓ no coordination with anyone
     ✓ works whenever you like
     ← the best property of this whole
       architecture

   OPTION 2  SIGNAL AN INCREMENTAL SNAPSHOT
     re-emit the table (chapter 5)
     ✓ works with non-compacted topics
     ✗ load on the source, and a flood for
       every OTHER consumer too

   OPTION 3  COPY THE TABLE DIRECTLY, THEN
             START CONSUMING FROM A KNOWN
             OFFSET
     ✓ fast
     ✗ fiddly to get the boundary right

Option 1 is why compacted topics are worth the storage. A new team can build a consumer, read from the beginning, and have full current state without asking anyone for anything or touching production.

Handling deletes

The thing consumers most often get wrong.

   SOFT DELETE     keep the row, set a flag
                   ✓ history preserved
                   ✓ safe against a bad
                     delete event
                   ✗ the sink grows forever

   HARD DELETE     remove it
                   ✓ matches the source
                   ✗ history gone
                   ✗ a replayed delete after
                     a re-insert is destructive

   IGNORE DELETES  ✗ the sink diverges
                     silently.
                   ← and this is the DEFAULT
                     if you use unwrap with
                     delete handling set to
                     `drop` (chapter 9)

Soft delete is the safer default for analytical sinks, and hard delete is right when the sink must mirror the source — a search index, a cache.

Whichever you choose, choose it explicitly. The failure mode is a sink that has been accumulating deleted rows for a year and nobody noticed.

Monitoring the consumer side

   CONSUMER LAG           per group, per
                          partition. the
                          headline number.
   PROCESSING ERROR RATE
   DLQ DEPTH              (chapter 9)
   END-TO-END LATENCY     now() − source.ts_ms
                          ← the number that
                            matters to users
   SINK DIVERGENCE        periodic row counts
                          and checksums,
                          source vs sink

That last one deserves more attention than it gets. Everything can look healthy — no lag, no errors, no DLQ — while the sink is quietly wrong because deletes were dropped or a schema change was mishandled.

-- the reconciliation that finds it
SELECT count(*), sum(amount) FROM source.orders;
SELECT count(*), sum(amount) FROM sink.orders WHERE NOT deleted;

Run it on a schedule and alert on divergence. It is the only check that catches the failures chapter 14 describes as silent.

What to take from this chapter

• Every sink converges on upsert-by-primary-key with a version check. The full row, the key and source.lsn are what make it possible.
• The version check protects against reordering, which idempotent upsert alone does not.
• Sink connectors need no code and scale with partitions — the single-task constraint applies to the source, not the sink. delete.enabled requires tombstones.
• Reach for Flink when you need joins, deduplication or enrichment — its temporal join against a CDC changelog is the feature that makes CDC genuinely powerful.
• ClickHouse’s ReplacingMergeTree implements the pattern in the storage engine, and converges only eventually.
• Iceberg is the best-behaved sink for schema changes and the worst for maintenance — compaction is a schedule, not a risk.
• Elasticsearch needs a reindex for type changes, and its documents are usually aggregates that require a join first.
• CDC-driven cache invalidation is underrated — it works regardless of which code path wrote.
• A new consumer bootstraps by reading the compacted topic from the beginning, with no load on the source and no coordination. That is the best property of the architecture.
• Choose your delete semantics explicitly. Ignoring deletes is the silent default and the sink diverges for years.
• Reconcile counts and sums between source and sink on a schedule. It is the only check that catches silent divergence.

Next: tuning and operations — the replication slot, the monitoring that matters, and keeping the source database safe.