4. Replication and Raft

You have met Raft already. The etcd guide works through it properly: a leader, an append-only log, entries committed once a majority has them on disk, elections when heartbeats stop. CockroachDB uses etcd’s Raft library, so none of that changes.

What changes is the number.

   etcd                        CockroachDB
   ────                        ───────────
   ONE Raft group              ONE Raft group PER RANGE
   one leader                  thousands of leaders,
   one log                     spread over every node
   one machine's write         write capacity grows as
   throughput, forever         ranges multiply

A cluster holding a terabyte at 512 MiB per range runs roughly two thousand ranges, each replicated three times: six thousand replicas, two thousand independent consensus groups, all on the same handful of machines, each electing leaders and counting majorities on its own.

That is how CockroachDB scales writes where etcd structurally cannot. It also creates two problems that shape this chapter: the heartbeat traffic of thousands of groups, and the fact that leading consensus is not the same job as serving reads.

Multiraft: not paying per group

Raft leaders heartbeat followers constantly to keep their term alive. With one group that is trivial. With two thousand groups between the same three machines it is a storm of tiny packets that say only “still here”.

The answer, usually called multiraft, is to heartbeat per node instead of per range.

   NAIVE                        MULTIRAFT

   n1 ─2000 heartbeats─► n2     n1 ──one message──► n2
   n1 ─2000 heartbeats─► n3     n1 ──one message──► n3

   plus COALESCED Raft traffic: many ranges' messages
   batched into a single RPC between the same two nodes

Range liveness is then derived from node liveness — one special cluster-wide record each node updates every few seconds. If n8 is live, every group it leads is live, and no per-range heartbeat is needed to prove it.

Danger

This makes node liveness the most safety-critical range in the cluster. If nodes cannot write their liveness records — a stalled disk, a saturated network, a clock problem — leadership is in doubt everywhere at once and the whole cluster stops accepting writes. Almost every “the entire cluster froze” story ends here. Chapter 14 covers the symptoms.

The leaseholder, and why it is not the Raft leader

Raft makes a committed write durable on a majority. It says nothing useful about reads: a follower’s log may lag, so reading from any replica risks reading the past — and CockroachDB has promised serializability.

The obvious fix is to route reads through consensus too. Correct, and far too slow: every read becomes a quorum round trip.

So CockroachDB adds a concept above Raft. One replica holds the range lease: a time-bounded, cluster-recognised, exclusive right to serve reads and sequence writes for that range. While it holds the lease, no other replica may answer a read, so its local copy is authoritative and a read costs one local lookup.

   RANGE 137

   n2  replica ─ RAFT LEADER ─ LEASEHOLDER   reads served here,
   n5  replica ─ follower                    writes proposed here
   n8  replica ─ follower

In the steady state the lease sits on the Raft leader, and CockroachDB actively works to keep them together — proposals from the leader skip a hop. When they drift apart, the lease moves to the leader rather than the reverse.

They remain separable, and the separation earns its keep:

   leases move FOR LOAD        even out traffic without
                               disturbing consensus
   leases move CHEAPLY         a small operation; moving
                               Raft leadership is an election
   leases move GEOGRAPHICALLY  put the lease in the region
                               doing the reading      → ch 11

Two kinds of lease

Expiration-based leases carry a wall-clock expiry and must be actively renewed. Used for the system ranges that everything else depends on, because they cannot rely on the liveness machinery they implement.

Epoch-based leases are tied to the holder’s node-liveness epoch and need no per-range renewal — which is what makes thousands of leases affordable at all.

Recent releases have been unifying leases with Raft leadership to remove the remaining cases where the two disagree. The operational picture in this chapter holds either way.

What a write costs

One write to range 137, replicas on n2, n5, n8, lease on n2:

   1. gateway sends the write to the LEASEHOLDER (n2)

   2. n2 sequences it against other transactions,
      evaluates it, proposes a Raft command

   3. n2  append to log ──► fsync
      n5  append to log ──► fsync ──► ack
      n8  append to log ──► fsync ──► ack

   4. MAJORITY has it (n2 + whoever answers first)
      ───────► COMMITTED

   5. each replica APPLIES it to local Pebble

Three things follow.

Commit needs two of three, not three of three. The slowest replica never holds you up; the second-fastest sets your latency. In three regions, write latency is the round trip to the nearer remote region.

Committed is not applied. A write is durable once a majority has it in their logs — before any of them has updated the data. A read at the leaseholder waits for the relevant entries to be applied.

Every replica fsyncs. Write latency has a floor set by your disks, which is why CockroachDB on storage with unpredictable flush latency behaves badly in ways that look mysterious. This is the same observation the etcd guide makes, for the same reason.

Where replicas live: the allocator

Nobody assigns replicas to nodes. A background process on each node — the allocator — continuously compares reality with policy and issues corrections. It juggles three goals:

Diversity. Spread a range’s replicas over as many distinct failure domains as possible, using the localities you gave each node at startup:

cockroach start --locality=region=eu-west,zone=eu-west-1a …

Caution

Get localities wrong and you can have three replicas in one rack with no warning at all. The cluster has no other way to learn your topology; it believes exactly what the flags say.

Balance. Even out disk usage, range count and lease count. New nodes attract replicas until they carry their share.

Constraints. Whatever the zone configuration demands:

ALTER TABLE users CONFIGURE ZONE USING
  num_replicas      = 5,
  constraints       = '{"+region=eu-west": 2, "+region=us-east": 2}',
  lease_preferences = '[[+region=eu-west]]';

SHOW ZONE CONFIGURATION FROM TABLE users;

Replication is configurable per cluster, database, table, index or partition. That one fact is what makes chapter 11’s multi-region features possible — they are a friendly vocabulary over these knobs.

Losing a node

This is the behaviour people adopt the product for, so the timeline is worth being precise about.

   t+0s    n8 dies. Its ranges are at 2/3 — still a
           majority, still serving. Leases it held move
           away within seconds.

   t+~10s  node liveness expires. The cluster marks it
           suspect and stops sending it work.

   t+5m    dead-node threshold passes. The allocator
           UP-REPLICATES every affected range: copy to
           a new node, add to the Raft group, remove
           the dead replica.

   later   back to 3/3. Nothing was promoted, nothing
           failed over, no runbook was opened.

The five-minute wait (server.time_until_store_dead) is deliberate: re-replicating a terabyte because somebody rebooted a machine is a self-inflicted incident. A planned removal should be a decommission, which moves replicas off first and shuts down after:

cockroach node decommission 8 --certs-dir=certs
cockroach node status --decommission --certs-dir=certs

The failure that actually hurts is losing a majority of a range’s replicas — two of three. That range becomes unavailable: not wrong, just unanswerable, exactly like the minority side of an etcd partition. Queries touching it hang rather than returning bad data. Recovery is genuinely unpleasant (chapter 14), and avoiding it is why data you cannot lose gets five replicas.

Voting and non-voting replicas

Not every copy needs a vote. A non-voting replica receives and applies the Raft log but does not count toward quorum, so it never slows a commit.

   us-east   voter      ─┐
   us-east   voter       ├─ quorum lives here; writes
   us-east   voter      ─┘  cost a us-east round trip

   eu-west   NON-VOTER     a local copy for fast stale
                           reads in Europe, costing the
                           write path nothing

That is how a distant region gets a readable local copy without every write paying for the distance. Combined with the follower reads of chapter 8, it is one of the most useful shapes available.

Counting replicas

   3 replicas   tolerate 1 loss
   4 replicas   tolerate 1 loss   ← buys nothing, costs a copy
   5 replicas   tolerate 2 losses

The arithmetic is per range, so the choice can be per table. A small tenant-metadata table everything depends on is a good candidate for five; a large table of historical events is not.

And the property etcd cannot offer, restated: replication factor is independent of cluster size. Three replicas on a fifty-node cluster means each range lives on three of the fifty; writes wait for two machines, never for fifty.

What to take from this chapter

• One Raft group per range — thousands per cluster, using etcd’s Raft library. This is the mechanism by which writes scale.
• Multiraft coalesces heartbeats and messages per node, deriving range liveness from node liveness — which makes the liveness range the most dangerous single dependency in the cluster.
• The leaseholder sits above Raft: an exclusive, time-bounded right to serve reads locally instead of through quorum. Leases are epoch-based for data ranges, expiration-based for system ranges.
• Lease and Raft leadership are normally co-located but deliberately separable, because leases move cheaply — for load and for geography.
• A write commits when a majority has fsynced it, so the second-fastest replica sets your latency; committed is not applied; and disk flush latency is the floor under everything.
• The allocator enforces diversity, balance and zone constraints continuously. Localities must be right at startup or diversity is a guess.
• Node loss is not a failover: ranges serve on at 2/3, and re-replication starts after five minutes. Planned removals decommission. Losing a majority of a range’s replicas makes it unavailable, not incorrect.
• Non-voting replicas give distant regions a local copy without adding their latency to every write.
• Three tolerates one, five tolerates two, even numbers buy nothing, and replication factor is independent of cluster size.

Next: transactions — write intents, the transaction record, parallel commits, and the single isolation level that shapes how you have to write your application.