Skip to content
Toolforge

3. Deployment shapes

Debezium is usually described as “a Kafka Connect connector”, and for most deployments that is right. It is not the only option, and the alternatives matter more than their share of the documentation suggests.

The constraint they all share

Section titled “The constraint they all share”

Before the options, the thing that shapes all of them:

   A DEBEZIUM CONNECTOR IS ALWAYS ONE TASK.


   `tasks.max` is ignored (or capped at 1)
   for the log-reading connectors.

You cannot parallelise reading one database’s log. The log is a single ordered sequence; two readers would either duplicate work or lose ordering, and ordering is the entire value proposition.

   ✗ scale out ONE connector       impossible
   ✓ scale out MANY connectors     yes — one
                                   per database
   ✓ scale the CONSUMERS           yes, freely
     (Kafka partitions, chapter 8)

This means the connector is a throughput ceiling you cannot raise by adding machines. In practice a single Debezium task handles a great deal — tens of thousands of changes per second is routine — but when it is the bottleneck, the answer is a faster machine, a lighter serialisation format, or fewer captured tables. Never more tasks.

It also means HA is failover, not load balancing. One task runs; if it dies, another starts where it left off.

1. Kafka Connect

Section titled “1. Kafka Connect”

The default, and what most of the ecosystem assumes.

   ┌──────────────────────────────────────┐
   │  KAFKA CONNECT CLUSTER               │
   │  (distributed mode, N workers)       │
   │                                      │
   │   ┌──────────────────────────────┐   │
   │   │ debezium-postgres  (1 task)  │   │
   │   ├──────────────────────────────┤   │
   │   │ debezium-mysql     (1 task)  │   │
   │   ├──────────────────────────────┤   │
   │   │ elasticsearch-sink (8 tasks) │   │
   │   └──────────────────────────────┘   │
   └───────────────────┬──────────────────┘
                       ▼
                     KAFKA
     ├─ connect-offsets   ← where each
     ├─ connect-configs     connector is
     ├─ connect-status
     └─ cdc.public.orders   ← the data

Connect gives you the operational machinery for free: distributed workers, automatic rebalancing when one dies, offsets stored durably in Kafka, a REST API for management, and a large ecosystem of sink connectors on the other side.

Terminal window
curl -X POST http://connect:8083/connectors -H 'Content-Type: application/json' -d '{
  "name": "orders-cdc",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres-replica",
    "database.dbname": "app",
    "topic.prefix": "cdc",
    "plugin.name": "pgoutput",
    "slot.name": "debezium_orders",
    "table.include.list": "public.orders,public.order_items",
    "publication.autocreate.mode": "filtered",
    "heartbeat.interval.ms": "10000"
  }}'
Terminal window
# the two calls you will make constantly
curl http://connect:8083/connectors/orders-cdc/status
curl -X POST http://connect:8083/connectors/orders-cdc/restart?includeTasks=true

Kafka’s chapter 12 has the warning worth repeating: check the task state, not the connector state. A connector reports RUNNING while its only task is FAILED. The pipeline is dead and the dashboard is green.

Terminal window
# what you actually want to alert on
curl -s http://connect:8083/connectors/orders-cdc/status \
  | jq -r '.tasks[].state'

Use Connect when you are sending to Kafka, want sink connectors on the other side, or already run Connect for something else.

2. Debezium Server

Section titled “2. Debezium Server”

A standalone application. No Kafka, no Connect.

   ┌────────────────────────────┐
   │  DEBEZIUM SERVER           │
   │  one process               │
   │  the same connectors       │
   │  a pluggable SINK          │
   └─────────────┬──────────────┘
                 ▼
     Kinesis / Pub-Sub / Pulsar / Redis
     Streams / NATS / RabbitMQ / Event Hubs
     / HTTP / a file / Kafka too
debezium.source.connector.class=io.debezium.connector.postgresql.PostgresConnector
debezium.source.database.hostname=postgres-replica
debezium.source.topic.prefix=cdc
debezium.source.offset.storage.file.filename=/data/offsets.dat


debezium.sink.type=pubsub
debezium.sink.pubsub.project.id=my-project
   ✓ NO KAFKA CLUSTER
     the single biggest reason people choose
     it. running Kafka to move a few hundred
     changes a second is a lot of machinery.
   ✓ one process, simple to reason about
   ✓ works with cloud-native messaging you
     already pay for


   ✗ OFFSETS ARE A LOCAL FILE by default
     lose it and you re-snapshot. put it on
     durable storage, or use a Redis/
     database offset store.
   ✗ no rebalancing — restarting it IS the
     HA story
   ✗ no sink-connector ecosystem
   ✗ fewer people have run it; less written
     about its edges

Use Debezium Server when your destination is not Kafka, or when Kafka would exist solely for this pipeline. NATS or Redis Streams plus Debezium Server is a genuinely much smaller system than Kafka plus Connect, and for moderate volumes it is the honest answer.

The offset file is the thing to get right. It is small, it is critical, and losing it means re-snapshotting the source. Persistent volume, or an offset backing store.

3. The embedded engine

Section titled “3. The embedded engine”

Debezium as a library inside your own application.

DebeziumEngine<ChangeEvent<String, String>> engine =
    DebeziumEngine.create(Json.class)
        .using(props)
        .notifying(record -> {
            // your code, per change event
            handle(record.value());
        })
        .build();


executor.execute(engine);
   ✓ no infrastructure at all
   ✓ full control of what happens per event
   ✓ good for a single-purpose service —
     a cache invalidator, an audit writer,
     a search indexer


   ✗ YOU own offsets, retries, restarts,
     backpressure, error handling and
     poison messages
   ✗ your application is now a stateful
     singleton — it cannot simply run
     three replicas
   ✗ upgrades of your app are upgrades of
     the pipeline

That singleton point is the one that bites. Two instances of your service means two readers of the same log, two slots, duplicate events and confusion. Leader election, or one replica.

Use the engine when you have exactly one consumer, that consumer is your own service, and adding a broker would be more infrastructure than the problem deserves. It is a bad choice for a shared pipeline — the moment a second consumer appears you want a broker.

4. The Kubernetes operator

Section titled “4. The Kubernetes operator”

An operator that runs Debezium Server instances as custom resources.

apiVersion: debezium.io/v1alpha1
kind: DebeziumServer
metadata:
  name: orders-cdc
spec:
  sink:
    type: kafka
    config:
      producer.bootstrap.servers: kafka:9092
  source:
    class: io.debezium.connector.postgresql.PostgresConnector
    config:
      database.hostname: postgres-replica
      topic.prefix: cdc
  storage:
    type: persistent
    claimName: debezium-offsets

Declarative, GitOps-friendly, and it handles the persistent volume for offsets — which is the main thing Debezium Server gets wrong when hand-rolled.

Worth evaluating if you are on Kubernetes and not on Kafka. Check its maturity for your version; it is newer than the other three options.

Choosing

Section titled “Choosing”
   Destination is Kafka, and you want sink
   connectors too
      └─▶ KAFKA CONNECT


   Destination is Kinesis/Pub-Sub/Pulsar/
   NATS/Redis, or Kafka would exist only
   for this
      └─▶ DEBEZIUM SERVER
          (+ the operator, on Kubernetes)


   Exactly one consumer, and it is your own
   service
      └─▶ THE EMBEDDED ENGINE


   Not sure, and Kafka is already there
      └─▶ KAFKA CONNECT. it is the
          well-trodden path and most of the
          writing assumes it.

Where offsets live

Section titled “Where offsets live”

Different per shape, and it is the piece that determines what a disaster costs you.

   CONNECT    the connect-offsets Kafka topic
              ✓ replicated, durable, survives
                worker loss
              ← the best of the four


   SERVER     a local file, by default
              ⚠ persistent volume, or an
                alternative store


   ENGINE     whatever you configure —
              a file, or your own code
              ⚠ entirely your problem


   OPERATOR   a PersistentVolumeClaim
              ✓ handled for you

Losing offsets means re-snapshotting, which on a large table means hours of load on the source and a flood of r events downstream (chapter 5). Treat the offset store as production data.

The same applies to the schema history topic for MySQL, SQL Server and Oracle — chapter 11 covers it, and losing it is worse than losing offsets.

High availability

Section titled “High availability”
   ONE TASK RUNS AT A TIME. ALWAYS.


   CONNECT    a worker dies → Connect
              reassigns the connector to
              another worker → it resumes
              from the offset topic
              → seconds to a minute of gap


   SERVER     the process dies → your
              supervisor restarts it → it
              resumes from its offset store
              → Kubernetes handles this fine


   ENGINE     your problem. leader election,
              or accept single-instance.

“Highly available” here means fast failover, not redundancy. There is always a gap while the new instance picks up — and that gap is fine, because the log is durable and nothing is lost. This is the property that makes CDC restartable at all, and it comes from chapter 2’s logs being persistent.

What to take from this chapter

Section titled “What to take from this chapter”
  • A Debezium connector is always one task. You cannot parallelise reading a log, so the connector is a throughput ceiling that more machines will not raise.
  • HA is failover, not load balancing. One task runs; another takes over from the stored offset.
  • Kafka Connect is the default: durable offsets in Kafka, rebalancing, a REST API, and sink connectors on the other side.
  • Check task state, not connector state. A RUNNING connector with a FAILED task is a dead pipeline with a green dashboard.
  • Debezium Server removes Kafka entirely and sinks to Kinesis, Pub/Sub, Pulsar, NATS, Redis Streams and others. Its offset file is the thing to get right — persistent storage, or a backing store.
  • The embedded engine is a library for a single consumer that is your own service. It makes your application a stateful singleton, which is the constraint that bites.
  • The Kubernetes operator runs Debezium Server declaratively and handles offset persistence, which is what hand-rolled deployments get wrong.
  • Offsets and the schema history topic are production data. Losing them means re-snapshotting.

Next: the change event — the envelope, the operation codes, tombstones, and the schema that every consumer will depend on.

Written by Sepuri Sai Krishna

© 2026CC BY 4.0Source