10. Anti-entropy — hints, read repair, and repair

“Eventually consistent” is not a shrug. It is a claim that three specific mechanisms drive replicas back into agreement, and each has costs, limits, and failure modes you need to know.

   ① HINTED HANDOFF   fixes writes missed by a briefly-down node
                      automatic, best-effort, 3-hour window

   ② READ REPAIR      fixes disagreement on data that gets read
                      automatic, free-ish, only touches read data

   ③ REPAIR           fixes everything, including data nobody reads
                      NOT automatic — you must schedule it

The first two are opportunistic. Only the third is a guarantee, and it is the one that is your responsibility.

① Hinted handoff

A replica is down when a write arrives. The coordinator cannot deliver it, so it stores a hint locally: a note saying “node C missed this mutation; deliver it when C comes back.”

   write ──▶ coordinator ──┬──▶ node A  ✓
                           ├──▶ node B  ✓
                           └──▶ node C  ✗ down
                                  │
                           store a hint locally
                                  │
                        ... node C returns ...
                                  │
                           replay the hint to C
                                  │
                              delete the hint

hinted_handoff_enabled: true
max_hint_window: 3h              # stop collecting hints after this
hinted_handoff_throttle: 1024KiB # per second, per delivery thread
max_hints_delivery_threads: 2

Hints have a hard limit, and it is the point most people miss. After max_hint_window (3 hours by default), the coordinator stops storing hints for that node entirely. Writes during the outage are simply not recorded anywhere for the missing replica.

So a node down for four hours comes back missing an hour’s worth of writes, and hints will not fix it. Only repair will.

The rule: a node down longer than max_hint_window must be repaired when it returns. A node down longer than gc_grace_seconds (chapter 7) should be wiped and rebuilt, not restarted, because it may resurrect deleted data.

nodetool statushandoff       # is it enabled?
nodetool listendpointspendinghints
nodetool truncatehints       # drop them all — occasionally needed if
                             # hint accumulation is itself the problem

Hints do not count toward consistency level

With CL=QUORUM and RF=3, two replicas must acknowledge. Storing a hint for a third does not count. But CL=ANY is satisfied by a hint alone — which is why ANY is dangerous: the write “succeeds” with zero replicas holding the data, and if the coordinator dies before delivering the hint, the write is gone.

② Read repair

During a read at any level above ONE, the coordinator gets a full response from one replica and digests from the others (chapter 6). If a digest disagrees, the replicas hold different data — and the coordinator fixes it.

   read at QUORUM
        │
        ├──▶ node A: full data,  digest X
        └──▶ node B:             digest Y     ← mismatch!
                    │
        coordinator requests full data from both,
        merges by timestamp, and writes the merged
        result back to whichever replicas were stale
                    │
        THEN returns the answer to the client

The write-back happens before the client gets its answer, which is why this is called blocking read repair. It costs a little latency on the reads that hit a mismatch, and it guarantees the consistency-level promise actually holds.

Older versions also had read_repair_chance and dclocal_read_repair_chance, which repaired a random fraction of reads in the background. These were removed in Cassandra 4.0 — they were unreliable and their probabilistic nature meant they gave no real guarantee. Modern read repair is driven purely by digest mismatches at the consistency level you asked for.

ALTER TABLE t WITH read_repair = 'BLOCKING';   -- default
ALTER TABLE t WITH read_repair = 'NONE';       -- disable entirely

The limit of read repair is the important part: it only fixes data that someone reads. A partition nobody has queried in six months can be inconsistent across replicas indefinitely. Read repair is a useful supplement and never a substitute for repair.

③ Repair

The real mechanism. nodetool repair compares data between replicas and streams whatever is missing, whether or not anyone has read it.

How it works: Merkle trees

Comparing terabytes of data between nodes byte by byte is impossible. So Cassandra builds a Merkle tree: a hash tree over token ranges.

                    ┌──────────────┐
                    │  hash(all)   │        compare the roots.
                    └──────┬───────┘        equal → done, nothing to do.
              ┌────────────┴────────────┐
        ┌─────┴─────┐             ┌─────┴─────┐
        │ hash(0-4) │             │ hash(5-9) │   differ → descend
        └─────┬─────┘             └─────┬─────┘
        ┌─────┴─────┐             ┌─────┴─────┐
     ┌──┴──┐     ┌──┴──┐       ┌──┴──┐     ┌──┴──┐
     │ 0-2 │     │ 3-4 │       │ 5-7 │     │ 8-9 │  ← leaf ranges
     └─────┘     └─────┘       └─────┘     └─────┘

Each replica builds the tree over the same token ranges. The trees are exchanged and compared top-down. Only the ranges whose hashes differ are streamed.

If two replicas are identical, one hash comparison ends the whole thing. If they differ in one small range, only that range moves.

The phases

   1. VALIDATION   each replica reads its data and builds a Merkle tree
                   → CPU and disk heavy. This is the expensive part.

   2. COMPARISON   trees exchanged, differing ranges identified

   3. STREAMING    differing ranges sent between replicas
                   → network heavy

   4. (the streamed data lands as new SSTables, which then compact)

Repair is expensive on all three of CPU, disk, and network. It is also essential, and skipping it is the root cause of a large fraction of Cassandra data problems.

Why it is mandatory

Three reasons, in order of severity:

Zombie data. Chapter 7’s mechanism. If a replica misses a delete and the tombstone is dropped elsewhere after gc_grace_seconds, the deleted data comes back and stays back. This is why repair must complete within gc_grace_seconds.

Missed writes. Hints expire after 3 hours; dropped mutations under load are never hinted at all. nodetool tpstats showing dropped MUTATION messages means data is missing somewhere, right now.

Bit rot and divergence. Disk errors, bugs, partial failures. Repair is the mechanism that heals them.

The rule

Run repair on every table at least once every gc_grace_seconds — with the default of 10 days, that means weekly.

Types of repair

# Full repair of one node's primary ranges
nodetool repair -pr my_keyspace

# Full repair, all ranges this node replicates (redundant across a cluster)
nodetool repair my_keyspace

# Confine to the local data centre (avoids WAN traffic)
nodetool repair -pr -local my_keyspace

# Incremental (the default since 2.2 — see the warning)
nodetool repair my_keyspace

# Explicitly full, not incremental
nodetool repair -full my_keyspace

# One token range at a time — the most controllable form
nodetool repair -st <start_token> -et <end_token> my_keyspace

-pr (primary range) means each node repairs only the ranges it is the first replica for. Run it on every node and every range is covered exactly once, instead of RF times. Use it for scheduled cluster-wide repair. Do not use -pr when repairing a single node that has been down — you need its full replica set covered.

Incremental repair marks repaired data so it is never re-examined, which should make subsequent repairs much cheaper. In practice it has a history of subtle problems, and it forces repaired and unrepaired data into separate compaction pools — raising SSTable counts and complicating compaction.

Cassandra 4.0 improved it substantially. Many experienced operators still prefer full subrange repairs for predictability. If you use incremental, monitor for repair failures and be prepared to run a full repair to reset state.

Use Reaper

Scheduling repair correctly across a cluster is genuinely fiddly: you want subrange repairs, run one at a time, with backoff, retries, and visibility into what succeeded.

Cassandra Reaper does exactly this. It is the standard tool, it is free, and hand-rolled cron scripts running nodetool repair are a well-known source of 3 a.m. incidents.

   Reaper gives you:
     - repairs split into small subranges
     - controlled parallelism so the cluster is not overwhelmed
     - automatic retries and resume after failures
     - scheduling per keyspace and per table
     - a UI showing what has and has not been repaired

If you run Cassandra in production and you are not running Reaper or an equivalent, repair is probably not actually happening. That is worth checking today.

Watching it

nodetool netstats              # streaming progress
nodetool compactionstats       # validation shows up here
nodetool tpstats | grep -i -E "validation|repair|antientropy"

-- 4.0+: repair history
SELECT keyspace_name, columnfamily_name, status, started_at, finished_at
FROM system_distributed.repair_history
ORDER BY id DESC LIMIT 20;

When repair is not the answer

Two cases where you should do something else.

A node was down longer than gc_grace_seconds. Do not repair it. It may hold data whose tombstones have already been dropped elsewhere, and repair will propagate that resurrected data outward. Wipe its data directories and rebuild it with nodetool rebuild (from another DC) or by removing and re-adding the node.

A node’s disk is corrupt or it lost data. Same answer — replace it rather than repair it:

# Start the replacement node with:
-Dcassandra.replace_address_first_boot=<old_ip>

The new node claims the dead node’s tokens and streams everything from the other replicas.

Adding and removing nodes

Related operations that use the same streaming machinery.

# Adding: just start a new node with correct seeds and cluster_name.
# It auto-bootstraps: claims tokens, streams its data, then joins.
nodetool status         # shows UJ (Up, Joining) during bootstrap

# AFTER adding, on every pre-existing node, one at a time:
nodetool cleanup        # removes data this node no longer owns

# Removing a live node gracefully — streams its data away first
nodetool decommission

# Removing a node that is already dead
nodetool removenode <host_id>

# Rebuilding a node's data from another data centre
nodetool rebuild -- <source_datacenter>

Three rules that prevent most topology incidents:

• Add or remove one node at a time, waiting for each to finish. Concurrent topology changes produce inconsistent token ownership.
• Always run cleanup after adding nodes. Otherwise old nodes keep serving disk to data they no longer own, and you get no space back from the expansion.
• Never put a joining node in its own seed list. It will skip bootstrap entirely, join the ring, and serve empty results for its ranges (chapter 2).

The full picture

   node down < 3 hours       → hints replay automatically.  ✓
   node down 3h – 10 days    → hints expired.
                               MUST run repair.
   node down > gc_grace      → tombstones may have been dropped.
                               DO NOT repair. Wipe and rebuild.

   data that is read often   → read repair keeps it converged
   data that is never read   → only repair will ever fix it

   no repair for > gc_grace  → deleted data comes back. Permanently.

What to take from this chapter

• Hinted handoff covers short outages only — 3 hours by default. Hints do not count toward consistency level.
• Read repair is automatic and blocking on digest mismatch, but only touches data someone actually reads. read_repair_chance was removed in 4.0.
• Repair is the only guarantee, and it is not automatic. Run it on every table within gc_grace_seconds — weekly, at the default.
• Repair uses Merkle trees: validation (expensive), comparison, then streaming only the differing ranges.
• Use -pr for scheduled cluster-wide repair. Use Reaper rather than hand-rolled cron.
• A node down longer than gc_grace_seconds must be wiped and rebuilt, never repaired.
• Add and remove nodes one at a time, and always nodetool cleanup afterwards.

Next: the closest thing Cassandra has to transactions.