8. Reads

Writes must reach a majority; that cost is unavoidable and chapter 7 accounted for it. Reads are where the design has room to be clever, and where most of the latency you can actually win back is hiding.

The default: read from the leaseholder

Because one replica holds an exclusive lease (chapter 4), it may answer from its own copy without asking anyone:

   SELECT balance FROM accounts WHERE id = 42;

   client ──► n3 gateway ──► n8 LEASEHOLDER ──► local Pebble
                                                seek key @ ts

No quorum, no consensus, no network beyond the one hop. A point lookup in a single region is typically well under a millisecond, and this is why CockroachDB’s read performance feels ordinary rather than distributed.

The leaseholder does have to wait until its Raft log is applied far enough to cover the read’s timestamp — usually already true — and it consults the lock table for intents left by uncommitted transactions (chapter 5).

Consistent reads cost geography, not consensus

The problem appears when the leaseholder is far away.

   gateway in eu-west, leaseholder in us-east

   client ──► eu-west gateway ──80 ms──► us-east leaseholder
          ◄────────────────── 80 ms ───────────────┘

   The data is replicated in Europe. You still crossed
   the Atlantic, because only the leaseholder may answer.

Three ways out, in increasing order of what you give up:

   MOVE THE LEASE        lease_preferences puts the lease
                         in the region that reads     → ch 11

   FOLLOWER READS        read a nearby replica, slightly
                         in the past                  → below

   NON-VOTING REPLICA    a local copy in a region that
                         never votes                  → ch 4

Closed timestamps make local reads safe

Chapter 6 introduced the closed timestamp: each range continuously publishes a promise that no write will ever commit below it.

   ───────────────────────────────────────────────────►
      ts_closed (≈3 s ago)            now

      ◄── frozen history ──►  ◄── still being written ──►

   Below ts_closed, EVERY replica has the same answer,
   because no new write can ever appear there.

That single guarantee is what lets a follower serve a read without violating anything: not a stale read in the Cassandra sense of “maybe wrong”, but a consistent read of a slightly earlier instant.

Exact-staleness follower reads

SELECT * FROM users
AS OF SYSTEM TIME follower_read_timestamp()
WHERE id = 4718;

follower_read_timestamp() returns the newest timestamp that any nearby replica is guaranteed to be able to serve — a few seconds ago. The gateway then routes the read to the closest replica, whatever its lease status.

   WITHOUT                        WITH follower read

   eu gateway ──80 ms──► us       eu gateway ──1 ms──►
   leaseholder                    eu replica
                                  (data ~3 s old)

The same syntax takes any timestamp, which is how reporting queries avoid contending with live traffic entirely:

SELECT count(*) FROM orders AS OF SYSTEM TIME '-1m' WHERE …;

BEGIN AS OF SYSTEM TIME follower_read_timestamp();
  -- a whole read-only transaction, locally served
COMMIT;

Bounded staleness

Exact staleness always pays the full staleness interval, even when fresher data is available locally. Bounded staleness asks for the freshest value a local replica can serve, as long as it is no older than a limit you set:

SELECT * FROM users
AS OF SYSTEM TIME with_max_staleness('10s')
WHERE id = 4718;

SELECT * FROM users
AS OF SYSTEM TIME with_min_timestamp(now() - '5s')
WHERE id = 4718;

   with_max_staleness('10s')

   "give me the newest version you can serve locally,
    but not older than 10 s. If no local replica can
    manage that, then go and ask the leaseholder."

Two restrictions make it narrower than it looks, and they are worth knowing before you design around it:

   • single-statement, read-only transactions only
   • the statement must touch ONE range
     (a point lookup, or a small scan)

So it fits session lookups, feature flags, profile reads and permission checks — the high-volume single-row reads that dominate most applications — and not analytical scans.

Caution

Follower reads are a correctness-preserving staleness: the data is a consistent snapshot of a real past instant, never a torn mixture. But it is still the past. Never use them for read-modify-write logic, for uniqueness checks, or for anything a user will immediately expect to see their own write reflected in.

Reading your own writes

The classic trap. A user updates their profile, the page reloads, and a follower read shows the old value — because the write was 200 ms ago and the follower is 3 seconds behind.

   WRITE  ──► leaseholder, committed at ts100
   READ   ──► follower at ts_closed = ts97  → old value

The rules that keep this out of your application:

   • inside one transaction, you always read your own
     writes — this is never a problem within a txn
   • after a write, route the user's next read to the
     LEASEHOLDER (plain SELECT), not a follower
   • use follower reads for data the user did not just
     change: catalogues, reference data, dashboards

Scans, and what the gateway does with them

A read touching many rows is not one request. DistSender splits it by range and fans it out:

   SELECT * FROM orders WHERE customer_id = 42;

   n3 gateway
     ├──► r88   n1   partial results
     ├──► r91   n5   partial results
     └──► r95   n2   partial results
                     ▼
              merged, ordered, returned

Where that work happens matters. For large scans, CockroachDB ships the computation to the data rather than the rows to the gateway — the distributed execution of chapter 9 — so a count(*) returns counts, not rows.

Reads inside a transaction

A read inside an explicit transaction is not the same as a standalone one, and the differences are the reason serializable isolation works at all.

   • every read is taken at the TRANSACTION's timestamp,
     not "now" — so a transaction sees one consistent
     instant from BEGIN to COMMIT

   • reads are REMEMBERED. the read set is what gets
     refreshed when the transaction is pushed forward
     in time (ch 5), and what makes a restart necessary
     when the refresh fails

   • you always see your own writes, including intents
     the rest of the cluster cannot see yet

The practical consequence is a rule worth putting in your code review checklist: a long transaction is expensive in proportion to how much it read, not only to how long it held locks. A transaction that scans a million rows and then writes one has a million-row read set to refresh, and a much higher chance of being told to start again.

-- bad: huge read set, tiny write, high restart odds
BEGIN;
  SELECT sum(total) FROM orders;              -- scans everything
  UPDATE stats SET order_total = …;
COMMIT;

-- better: read at a fixed past timestamp, outside the txn
SELECT sum(total) FROM orders AS OF SYSTEM TIME '-30s';
-- then a small, fast transaction to record it

The read latency budget

When a read is slower than expected, the time is in one of five places, and they are distinguishable:

   PLAN            reading far more rows than returned
                   → index          EXPLAIN ANALYZE rows
   CONTENTION      waiting on another txn's intents
                   → ch 5           "contention time"
   DISTANCE        the leaseholder is in another region
                   → ch 11          "network time"
   STORAGE         many SSTs or many MVCC versions
                   → ch 6, 13       KV time, MVCC steps
   QUEUEING        the node is overloaded and admission
                   control is shedding
                   → ch 13          admission metrics

Every one of those numbers is in EXPLAIN ANALYZE (DIST, VERBOSE). Reading the plan first takes a minute; guessing between five possibilities takes an afternoon.

Making reads fast

The levers, roughly in the order you should reach for them:

   1. an INDEX that covers the query      → ch 10
      (STORING the columns you select)
   2. lease placement near the readers     → ch 11
   3. follower or bounded-staleness reads  → above
   4. a non-voting replica in the reading
      region                               → ch 4
   5. AS OF SYSTEM TIME for reports, so
      they never contend with writes       → ch 6

Number 1 comes first for a reason: a query that reads 200 rows instead of 200,000 beats every distribution trick in the list, and it is the same advice any single-node database would give you.

Seeing it

EXPLAIN ANALYZE (DIST) SELECT * FROM users WHERE email = 'alice@example.com';

The output names the things this chapter described: distribution: local versus full, rows read versus rows returned, KV time, contention time, and which node did what.

-- is a query being served locally, or crossing regions?
SELECT * FROM crdb_internal.node_statement_statistics
ORDER BY service_lat_avg DESC LIMIT 10;

What to take from this chapter

• A default read is served entirely by the leaseholder from local storage — no quorum, no consensus. This is why reads feel like a normal database.
• The leaseholder waits for its apply to catch up and consults the lock table for intents, but never asks another node.
• Distance, not consensus, is what makes a read slow. The fixes are moving the lease, reading a follower, or placing a non-voting replica.
• Closed timestamps freeze history below a moving line, so any replica can serve a consistent read of a slightly earlier instant — stale in time, never inconsistent.
• follower_read_timestamp() gives exact staleness (a few seconds) routed to the nearest replica; with_max_staleness() gives the freshest local answer within a bound, but only for single-statement, single-range, read-only queries.
• Never use stale reads for read-modify-write, uniqueness checks, or read-your-own-writes. Inside a transaction you always see your own writes.
• Large scans are split by range and fanned out, with computation pushed to the data.
• The first lever is still a better index. Distribution tricks come after that.

Next: SQL — the optimiser, distributed execution, joins, and the plans that turn a fast query into a cluster-wide fan-out.