2. The architecture

Run CockroachDB and you start one program. Run it on nine machines and you have started the same program nine times, with no configuration saying which of them is special, because none of them is.

# every node. the only difference is the address.
cockroach start \
  --certs-dir=certs \
  --advertise-addr=node3.internal:26257 \
  --join=node1.internal,node2.internal,node3.internal \
  --locality=region=eu-west,zone=eu-west-1c \
  --store=/mnt/data

No --role=router. No --config-server. No arbiter, no witness, no name node. Compare that with a sharded MongoDB deployment — mongos routers, config server replica set, shard replica sets, three failure behaviours to learn — or with Kafka before KRaft, which needed a whole other distributed system beside it.

   WHAT YOU ACTUALLY DEPLOY

   ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
   │  cockroach  │ │  cockroach  │ │  cockroach  │
   └─────────────┘ └─────────────┘ └─────────────┘
       node 1          node 2          node 3

   Identical binaries. No roles, no primary.
   The only difference between nodes is which
   ranges they happen to hold right now — and
   that changes by itself.

The five layers

Inside that one binary, a request falls through five layers. Every chapter of this guide lives in one of them, so this diagram is the table of contents for the rest of the book.

   ┌──────────────────────────────────────────────────┐
   │  SQL                                    ch 9, 10 │
   │  Parse, plan, optimise, execute.                 │
   │  Postgres wire protocol in, rows out.            │
   ├──────────────────────────────────────────────────┤
   │  TRANSACTIONAL KV                       ch 5, 6  │
   │  Serializable transactions over keys.            │
   │  Intents, timestamps, conflict resolution.       │
   ├──────────────────────────────────────────────────┤
   │  DISTRIBUTION                           ch 3     │
   │  One sorted key space, cut into ranges.          │
   │  "Which node holds /users/4718?"                 │
   ├──────────────────────────────────────────────────┤
   │  REPLICATION                            ch 4     │
   │  Each range replicated 3× or 5× by Raft.         │
   │  Leases, quorum, rebalancing.                    │
   ├──────────────────────────────────────────────────┤
   │  STORAGE                                ch 3, 6  │
   │  Pebble — an LSM tree on local disk.             │
   │  Keys, values, MVCC versions, compaction.        │
   └──────────────────────────────────────────────────┘

Read it as a translation chain. Each layer restates the request in the vocabulary of the layer below:

   SQL          SELECT balance FROM accounts WHERE id = 42
                     ↓
   TXN KV       read /Table/53/1/42/0 at timestamp T,
                and tell me about uncommitted writes there
                     ↓
   DISTRIBUTION that key is in range 137
                replicas n2 n5 n8, lease on n8
                     ↓
   REPLICATION  n8 holds the lease, its log is applied
                far enough, so it may answer locally
                     ↓
   STORAGE      Pebble: seek /Table/53/1/42/0 at ts ≤ T

The useful consequence: when something is slow or broken, the first question is always which layer. A slow query caused by a bad plan and a slow query caused by a cross-region leaseholder look nothing alike. Chapter 13 leans on this constantly.

The storage layer: Pebble

At the bottom of each node sits Pebble, a key-value store written by Cockroach Labs to replace RocksDB, which descends from LevelDB. It is an LSM tree: writes land in a memtable and a write-ahead log, get flushed to sorted files, and are merged by background compaction into progressively larger levels.

If that sounds familiar, it is because you have read it before — Cassandra’s storage engine is the same shape, and everything that guide says about write amplification, compaction debt and read amplification applies here.

Two CockroachDB-specific details:

Keys carry timestamps. A row version is not stored at /Table/53/1/42/0 alone but at that key with an MVCC timestamp suffix, so versions of one row sort together, newest first. Chapter 6 lives on this.

Pebble replaced RocksDB in 2020 because a Go program calling a C++ library through cgo paid a tax on every single operation, and because owning the engine meant it could understand MVCC natively. You will still see rocksdb in old metric names and old blog posts.

-- the LSM's health, per node
SELECT node_id, store_id, metrics->>'rocksdb.read-amplification'
FROM crdb_internal.kv_store_status;

Ranges, replicas, leaseholders

The distribution and replication layers together produce the three words you will use most when operating a cluster.

A range is a contiguous slice of the key space, about 512 MiB by default. It is the unit of everything: replication, rebalancing, splitting, lease ownership.

A replica is one copy of a range on one node. Three by default, on three different nodes, and — if you set localities — in three different failure domains.

The leaseholder is the one replica currently responsible for that range. It serves reads without consulting anyone and it coordinates writes. It is not a Raft concept; it sits above Raft, and chapter 4 explains why the separation matters.

   RANGE 137 — one range, three replicas, one lease

   node 2               node 5               node 8
   ┌────────────┐       ┌────────────┐       ┌────────────┐
   │ replica    │       │ replica    │       │ replica    │
   │ LEASEHOLDER│◄─────►│  follower  │◄─────►│  follower  │
   └────────────┘       └────────────┘       └────────────┘
        │
        └─ serves ALL reads for range 137
           coordinates ALL writes for range 137

   A node is leaseholder for SOME ranges and a plain
   follower for others. "The leader node" is not a thing.

A healthy nine-node cluster has every node holding leases for roughly a ninth of the ranges. Load spreads not because anyone balanced servers, but because leases spread.

-- how evenly, right now
SELECT node_id, ranges, leases FROM crdb_internal.kv_node_status;

The gateway node

Now the thing that trips people up.

The node your client connects to is the gateway. Your load balancer chose it, or your connection string did, or chance did. It has no particular relationship to your data.

   SELECT * FROM users WHERE id = 4718;

   client ──► node 3  (GATEWAY)
                │  parses, plans, optimises
                │  resolves key → range 137
                │  looks up leaseholder → node 8
                │
                └──► node 8  (LEASEHOLDER)
                        │  reads local Pebble
                        └──► rows ──► node 3 ──► client

The gateway does the SQL thinking and owns the client’s transaction; the leaseholder does the reading. For a single-row lookup that is one extra network hop — a fraction of a millisecond inside one datacentre, and very much not that between continents.

Three consequences

Any node serves any query. Point your load balancer at all of them. There is no write node to route to, which is why connection strings list every node:

postgresql://app@n1,n2,n3:26257/store?sslmode=verify-full

Locality matters more than you expect. A gateway in Frankfurt talking to leaseholders in Virginia pays the round trip at every step of the plan. Chapter 11 is largely about not doing that.

The gateway is a transaction’s home. It holds coordination state and drives the commit. If the gateway dies mid-transaction, that transaction aborts — the data is safe, the client sees an error and retries. Chapter 5.

Caution

Do not pin your application to one node “for consistency”. There is no consistency benefit — every node gives the same answers — and you have just created the single point of failure the product exists to remove. Balance across all of them, ideally to the nearest region.

How a node finds anything

If ranges move, how does a gateway know where range 137 lives?

The range map is itself stored in the key space, in a small number of meta ranges at the very beginning of it. They are ordinary ranges — replicated by Raft, subject to the same rules — that happen to hold the directory of every other range.

   /Meta2/Table/53/1/9000000  →  range 137 ends here;
                                 replicas n2 n5 n8; lease n8

   Every node caches this. A stale cache is NORMAL:
   the node tries the address it remembers, is told
   "not me any more", refreshes and retries. In logs
   that is NotLeaseHolderError — routine, not an error
   condition.

The payoff of storing the directory as data: there is no metadata service to deploy, scale, back up or lose. It gets consensus, replication and repair for free, from the machinery that was already there.

The background work

A cluster nobody is touching is still busy. Knowing these loops exist explains most “why is it doing that?” moments:

   ALLOCATOR          where replicas should live: spread
                      across failure domains, off full
                      nodes, restored after loss    → ch 4

   SPLITS / MERGES    keep ranges near target size, and
                      split hot ones by load         → ch 3

   COMPACTION         Pebble's LSM merging           → ch 13

   MVCC GC            drop row versions older than the
                      GC window (default 4h/25h)     → ch 6

   CLOSED TIMESTAMPS  publish "nothing will write below
                      T" so followers can serve reads → ch 8

None of these are jobs you schedule. All of them are jobs you can find in the DB Console when the cluster is unhappy.

Counting nodes

The arithmetic is Raft’s, and it applies per range rather than per cluster. Replication factor 3 tolerates one unavailable replica; 5 tolerates two.

Two differences from etcd matter enormously:

Adding nodes adds capacity. More nodes means ranges spread over more machines and more leaseholders sharing the work. etcd gets slower as it grows; this gets bigger.

Replication factor is not cluster size. Fifty nodes with replication factor three means every range lives on three of the fifty, and a write waits for two of them — never for fifty.

-- per table if you want it
ALTER TABLE accounts CONFIGURE ZONE USING num_replicas = 5;

What to take from this chapter

• One binary, symmetric nodes, no coordinator and no roles. The only difference between nodes is which ranges they currently hold, and that changes on its own.
• Five layers — SQL → transactional KV → distribution → replication → storage — each restating the request for the one below. Diagnosis starts with “which layer?”.
• Storage is Pebble, an LSM tree in the Cassandra mould, with MVCC timestamps built into the keys.
• A range is ~512 MiB of key space, a replica is one copy, and the leaseholder is the replica that serves reads and coordinates writes. Spreading leases is what spreads load.
• The gateway is whichever node you connected to; it plans the query and owns the transaction, but the data is almost certainly elsewhere. Balance across all nodes — pinning to one buys nothing and costs availability.
• The range directory lives in meta ranges inside the key space itself, so there is no metadata service. Stale caches and NotLeaseHolderError are routine.
• Continuous background loops: allocator, splits and merges, compaction, MVCC GC, closed timestamps.
• Replication factor is per range and independent of cluster size. Fifty nodes still commit with two of three.

Next: the key space itself — how a SQL row becomes a key, how ranges are cut, and why your primary key decides the shape of your cluster’s load.