8. Topics and routing

Debezium’s default topic layout is one topic per table, and it is usually right. The decisions worth making deliberately are partitioning, retention and the cases where one topic per table is wrong.

The naming convention

   <topic.prefix>.<schema>.<table>

   topic.prefix = cdc
   → cdc.public.orders
     cdc.public.order_items
     cdc.public.customers

   MySQL has no schema layer, so:
   → cdc.app.orders
     (prefix, database, table)

topic.prefix was called database.server.name before Debezium 2.0, and it appears in old documentation and blog posts under that name. It also becomes the logical server name inside source.name (chapter 4), so changing it invalidates offsets — pick it once.

   ONE TOPIC PER TABLE

   ✓ independent retention, partition count
     and compaction per table
   ✓ consumers subscribe to what they need
   ✓ one table's volume does not affect
     another's
   ✓ schema per topic is stable
   → the right default

Plus, for MySQL/SQL Server/Oracle, a schema change topic named after the prefix alone (cdc), carrying DDL events — separate from the internal schema history topic of chapter 7.

Topic creation

topic.creation.enable=true
topic.creation.default.replication.factor=3
topic.creation.default.partitions=12
topic.creation.default.cleanup.policy=compact
topic.creation.default.compression.type=zstd

# a group with different settings
topic.creation.groups=highvolume
topic.creation.highvolume.include=cdc\\.public\\.events.*
topic.creation.highvolume.partitions=48
topic.creation.highvolume.cleanup.policy=delete
topic.creation.highvolume.retention.ms=604800000

Configure topic creation explicitly rather than relying on broker auto-creation. Broker defaults are usually one partition and a replication factor of one — fine for a laptop, and a single point of data loss in production.

The groups mechanism is how you give a high-volume table different treatment without creating every topic by hand.

Partitioning

   partition = hash(key) % partition_count
   key = the primary key

This is the mechanism behind every ordering guarantee you have (chapters 4, 6 and 7). Everything else about partitioning follows from not breaking it.

   PARTITION COUNT

   too few   → limits consumer parallelism.
              one partition can be read by
              at most one consumer per group.
   too many  → more metadata, more files,
              slower rebalances

   START AT 12–24 for a busy table, 3–6 for
   a quiet one.

You can add partitions later, and it breaks ordering when you do. Adding partitions changes hash(key) % n for most keys, so a row’s changes move to a different partition — and its old changes are in the old one. In-flight ordering is broken across the change, permanently for anything still unconsumed.

   → PLAN THE PARTITION COUNT UP FRONT.
     over-provisioning is much cheaper than
     re-partitioning.

Tables without a primary key

Chapter 4’s point, with the consequence:

message.key.columns=public.events:event_uuid,public.logs:id

Without a key, events round-robin. No per-row ordering, no compaction, no upsert semantics. Set the key explicitly, or accept that the topic is an unordered stream of facts.

You can also key by something other than the primary key when it suits the consumer:

message.key.columns=public.order_items:order_id

Now every line item of an order lands in the same partition as every other, which gives ordering across the parent-child relationship — at the cost of compaction no longer working per item.

Retention and compaction

The choice that determines what a new consumer can do.

   cleanup.policy = compact
     keeps the LATEST value per key, forever
     → the topic IS the table
     → a new consumer bootstraps the full
       current state by reading from the
       beginning
     → tombstones (chapter 4) remove deleted
       keys
     ← the classic CDC choice

   cleanup.policy = delete, retention.ms = 7d
     a rolling window of changes
     → smaller
     → a new consumer sees only the last week
     → needs a snapshot to bootstrap

   cleanup.policy = compact,delete
     compact, AND drop anything older than
     retention
     → bounded size, latest state for active
       keys
     → a key untouched for longer than the
       retention DISAPPEARS
     ⚠ subtle. usually not what you want.

Kafka’s chapter 10 is the reference for the mechanism. The CDC-specific point is that a compacted topic keyed by primary key is a live replica of the table — which is what makes “a new consumer can bootstrap without touching the source database” true, and it is one of the best properties of this whole architecture.

For compaction to work you need tombstones on (tombstones.on.delete=true, the default) — otherwise deleted rows persist in the compacted topic forever.

   ⚠ COMPACTION IS NOT IMMEDIATE
     the active segment is never compacted,
     and cleaning is periodic. a compacted
     topic still contains duplicate keys.
     consumers must handle that — which,
     per chapter 7, they already must.

Routing

Renaming topics

transforms=route
transforms.route.type=org.apache.kafka.connect.transforms.RegexRouter
transforms.route.regex=cdc\\.public\\.(.*)
transforms.route.replacement=$1

cdc.public.orders → orders. Cosmetic, and worth doing if consumers find the prefix noisy.

Merging tables into one topic

The genuinely useful case: sharded tables.

   orders_2024, orders_2025, orders_2026
   or
   orders_shard_00 … orders_shard_15

   → all one logical table
   → consumers should not know about shards

transforms=byLogicalTable
transforms.byLogicalTable.type=io.debezium.transforms.ByLogicalTableRouter
transforms.byLogicalTable.topic.regex=cdc\\.public\\.orders_(shard_\\d+)
transforms.byLogicalTable.topic.replacement=cdc.public.orders
transforms.byLogicalTable.key.field.name=__shard
transforms.byLogicalTable.key.field.regex=cdc\\.public\\.orders_(shard_\\d+)
transforms.byLogicalTable.key.field.replacement=$1

The key gets the shard appended, which matters: two shards may both contain id=4471, and without the shard in the key they would collide in a compacted topic and overwrite each other.

That key modification is the part people miss, and the failure is silent data corruption in the sink.

Content-based routing

transforms=route
transforms.route.type=io.debezium.transforms.ContentBasedRouter
transforms.route.language=jsr223.groovy
transforms.route.topic.expression=\
  value.after != null && value.after.region == 'EMEA' \
    ? 'orders-emea' : 'orders-global'

Powerful and worth being suspicious of. A scripting engine evaluating an expression per record is a performance cost, a dependency, and logic that lives in connector configuration rather than in code anyone reviews.

Prefer routing downstream — in Flink, or in the consumer — unless the split genuinely needs to happen before the topic.

One topic per table, or not

   TOPIC PER TABLE            ← the default
     ✓ independent settings
     ✓ selective subscription
     ✓ stable schema per topic
     ✗ no cross-table ordering
     ✗ many topics on a wide schema

   TOPIC PER AGGREGATE
     orders + order_items + shipments → one
     topic, keyed by order_id
     ✓ ORDERING ACROSS THE AGGREGATE
     ✓ a consumer sees a coherent unit
     ✗ mixed schemas in one topic
     ✗ needs routing and careful keying
     → this is the OUTBOX PATTERN's shape,
       and chapter 10 argues you should do
       it deliberately at the source rather
       than by routing CDC

   ONE TOPIC FOR EVERYTHING
     ✗ ordering across all tables (a real
       benefit, occasionally)
     ✗ one schema soup, one retention,
       one bottleneck
     → almost never right

If you want per-aggregate topics, chapter 10’s outbox is the better route there. Merging CDC topics by routing gives you the topic shape without the semantic benefit — you still have raw table rows, just mixed together.

Ordering, summarised

   ONE ROW           ✓ guaranteed
   ONE TABLE, per key ✓ guaranteed
   ACROSS ROWS       ✗
   ACROSS TABLES     ✗
   ACROSS TOPICS     ✗

   → a child row can arrive before its
     parent, routinely

The options when you need more, in order of preference: key both tables by the parent id into one topic; use provide.transaction.metadata (chapter 4) and buffer; join downstream in Flink; or make the consumer retry until the parent exists.

That last one is unglamorous and works well. A sink that fails to insert a child, retries in thirty seconds, and succeeds because the parent has arrived is simple and robust.

Sizing

   PER TABLE, ESTIMATE

     changes/sec × event size
       = throughput

     × retention (or, for compacted topics,
       row count × row size × ~1.5)
       = storage

   A CDC EVENT IS BIGGER THAN THE ROW
     before + after + source metadata
     ≈ 2–3× the row, with Avro
     ≈ 10× or worse with JSON+schemas
       (chapter 4)

A compacted topic holds roughly your table, plus overhead and un-compacted segments. A 400-million-row table is not a small topic, and it is worth computing before someone is surprised.

What to take from this chapter

One topic per table is the right default, and topic.prefix should be chosen once — changing it invalidates offsets.
Configure topic creation explicitly. Broker defaults are one partition and no replication.
Partition by key, which is the primary key — that is where every ordering guarantee comes from.
Plan the partition count up front. Adding partitions later permanently breaks ordering for in-flight data.
Set message.key.columns for tables with no primary key, or lose ordering, compaction and upserts.
A compacted topic keyed by primary key is a live replica of the table, which lets new consumers bootstrap without touching the source. It needs tombstones, and it is never instantaneous.
ByLogicalTableRouter merges sharded tables — and you must add the shard to the key, or rows collide silently.
Content-based routing works and belongs downstream wherever possible.
Ordering is per row and nothing more. Key by the parent, use transaction metadata, join downstream, or retry until the parent exists — which is simple and effective.
A CDC event is two to three times the row in Avro, and far more in JSON. Size topics accordingly.

Next: transformations — what SMTs are good for, the unwrap everybody uses, and where to stop.

Written by Sepuri Sai Krishna