6. MVCC and clocks

Every transaction in the last chapter had a timestamp, and the whole serializability argument rested on comparing them. This chapter is about where those timestamps come from when there is no central clock, and why one of them can make your transaction restart for no visible reason.

MVCC, the familiar half

Nothing is ever overwritten. A write adds a new version of a key at a new timestamp:

   /Table/53/1/42/0 @ ts21   balance = 40
   /Table/53/1/42/0 @ ts14   balance = 60
   /Table/53/1/42/0 @ ts09   balance = 100
       ▲ newest first, because keys sort with the
         timestamp descending — a read at ts18 seeks
         to ts18 and takes the first version it finds

A read at a timestamp sees exactly the version that was current then, without blocking any writer and without being blocked by one. That is the same MVCC idea as PostgreSQL and MySQL, with one important difference: the version is selected by timestamp, not by transaction-id visibility rules. Time is the visibility rule here.

A DELETE writes a tombstone — a version marked as absent — exactly as Cassandra does, and with the same consequence: deleted data still occupies space and still has to be read past until it is collected.

Why clocks are the problem

Two transactions on two nodes. Which happened first?

   n1 commits a write   local clock says 10:00:00.000
   n4 reads the key     local clock says 09:59:59.998

   n4's clock is 4 ms behind. Should n4 see the write?

If you answer with wall clocks, you get the wrong answer whenever clocks disagree — and clocks always disagree. NTP keeps a fleet within a few milliseconds on a good day, and within seconds on a bad one.

Google solved this with TrueTime: atomic clocks and GPS receivers in every datacentre, giving every node an interval [earliest, latest] guaranteed to contain the true time, typically a few milliseconds wide. Spanner commits by waiting out that interval, so a transaction’s timestamp is definitely in the past before anyone can see it.

Cockroach Labs had to run on rented hardware with NTP, so they built something else.

Hybrid logical clocks

An HLC timestamp is a wall-clock reading plus a logical counter:

   1712558400.123456789 , 3
   └────── physical ─────┘ └┬┘
                            logical counter, for events
                            within the same physical tick

The rules are short:

   • every message between nodes carries the sender's HLC
   • on receipt, a node sets its HLC to
        max(own clock, received clock) + 1 tick
   • the physical part never runs backwards

The effect: causality is always respected. If A happened before B, and information flowed from A to B, then B’s timestamp is higher — with no atomic clocks anywhere. Drift between nodes that never talk to each other does not matter, because nothing depends on it.

What HLC does not give you is a guarantee about two events that never exchanged a message. Which brings us to the tax.

The uncertainty interval

A transaction starts on n3 at timestamp T. Somewhere else, a write committed at T+2ms. Did that write happen before this transaction began? With clocks that may differ by a few milliseconds, the honest answer is: unknowable.

CockroachDB refuses to guess. Every transaction carries an uncertainty interval of width --max-offset (500 ms by default, commonly tuned to 250 ms):

   txn timestamp T                  T + max_offset
   ├──────────────────────────────────────┤
              THE UNCERTAINTY WINDOW

   A value found in this window MIGHT have committed
   before the txn started. The txn cannot ignore it
   and cannot safely read it.

Meeting a value inside that window produces a ReadWithinUncertaintyIntervalError: the transaction pushes its timestamp above the offending value and refreshes its earlier reads. If the refresh succeeds, nothing is visible to you. If it fails, you get a 40001 retry.

ERROR: restart transaction: ReadWithinUncertaintyIntervalError:
       read at time 1712558400.123, encountered previous write
       with future timestamp 1712558400.125 within uncertainty
       interval (SQLSTATE 40001)

The window shrinks only once per transaction per node: after observing one node’s clock, the transaction remembers it, so repeated reads on that node stop being uncertain.

Danger

--max-offset must be identical on every node, and it can only be changed with a full cluster restart. Setting it too low is not a performance trick — it is a correctness risk, because a node whose clock drifts beyond the offset would be able to violate serializability. CockroachDB defends itself: a node that detects its clock is more than max-offset/2 from a majority of its peers shuts itself down immediately.

That self-destruct is deliberate and is the right behaviour, but it means NTP is a hard dependency. Chapter 14 describes what a badly synchronised fleet looks like from the outside: nodes dying one at a time, with clock synchronization error in the logs.

Closed timestamps

Each range continuously publishes a closed timestamp: a promise that no transaction will ever commit a write below it on that range.

   now ────────────────────────────────────────►
        │                        │
        closed timestamp         current time
        (typically ~3 s ago)

   Below the line: the history is frozen. Any replica,
   leaseholder or not, can serve a consistent read.

This is what makes follower reads possible in chapter 8, and it is also the mechanism changefeeds use in chapter 12 to emit ordered, gap-free checkpoints.

Time travel

Because old versions are kept, you can read the past directly:

SELECT * FROM users AS OF SYSTEM TIME '-10s' WHERE id = 4718;

SELECT * FROM users AS OF SYSTEM TIME '2026-09-21 09:15:00';

BEGIN AS OF SYSTEM TIME '-30s';
  SELECT … ; SELECT … ;   -- a whole consistent snapshot
COMMIT;

This is not a special feature bolted on; it is just naming a timestamp the storage layer already has. Uses worth knowing:

   '-10s'              a read that never contends with
                       live traffic — perfect for reports
   a fixed timestamp   "what did this row look like
                       before the bad deploy?"
   follower_read_ts()  the newest timestamp guaranteed
                       servable by a local replica  → ch 8

And the recovery that has saved people:

-- undo a mistaken DELETE, if it happened inside the GC window
INSERT INTO users
SELECT * FROM users AS OF SYSTEM TIME '-20m'
WHERE id NOT IN (SELECT id FROM users);

Garbage collection, and the limit on all of it

Old versions are removed by MVCC garbage collection once they fall outside the GC TTL — 4 hours by default in newer releases, 25 hours in older ones:

ALTER TABLE users CONFIGURE ZONE USING gc.ttlseconds = 86400;
SHOW ZONE CONFIGURATION FROM TABLE users;

The TTL is the single dial behind three different things, which is why it is worth setting consciously:

   LONGER TTL     time travel reaches further back
                  incremental backups can span more time
                  changefeeds survive longer outages
                  ── but more versions to store and scan

   SHORTER TTL    less space, faster scans over hot rows
                  ── but a shorter undo window, and
                     backups/changefeeds can fall off
                     the end and need a full restart

The failure this produces is specific and common enough to name: a table with heavy churn and a long TTL accumulates thousands of versions per row, and every read has to scan past them. The symptom is a point lookup that used to be fast slowly becoming slow, with high “MVCC step” counts in the statement profile. Chapter 13 is where you go looking.

Protected timestamps

Garbage collection is a problem for anything that reads history over a long period: a backup that takes two hours, a changefeed catching up after an outage, a restore streaming from one cluster to another. If GC ran on schedule during that work, the history would vanish underneath it.

So jobs take out a protected timestamp record: a claim on the key space saying “do not collect anything above time T while I am running”.

   gc.ttlseconds says     "collect below now − 4h"
   the backup job says    "but not below ts_start"
   ──────────────────────────────────────────────
   GC waits. The backup keeps the history it needs.

This is the mechanism behind a failure that surprises people: a stuck or forgotten job holds history forever, disk usage climbs, and reads slow down as every scan steps over versions that should have been collected months ago.

SELECT * FROM crdb_internal.kv_protected_ts_records;
SHOW JOBS WHERE status IN ('paused', 'running');

A paused changefeed from last quarter is the usual culprit. Cancel it, or resume it, but do not leave it paused indefinitely.

How this differs from PostgreSQL’s VACUUM

Both databases keep old versions and both must eventually remove them, but the failure modes are not the same, and instincts carried from one do not transfer.

   PostgreSQL            old versions live in the SAME
                         heap pages as live rows. VACUUM
                         must visit them; a long-running
                         transaction holds back the
                         horizon and BLOATS the table.
                         Failure mode: transaction
                         wraparound, table bloat.

   CockroachDB           old versions are separate KEYS
                         in an LSM tree, removed by a GC
                         queue per range and then by
                         compaction. A long transaction
                         does not hold back GC the way it
                         does in PostgreSQL — but a
                         PROTECTED TIMESTAMP does.
                         Failure mode: version pile-up on
                         churned rows; scans that step
                         over thousands of dead versions.

The practical difference: in PostgreSQL you watch autovacuum and the oldest transaction. Here you watch the GC TTL, protected timestamp records, and the MVCC step and seek counts in a statement profile.

-- the numbers that reveal version pile-up
EXPLAIN ANALYZE (VERBOSE) SELECT * FROM counters WHERE id = 7;
--   … MVCC step count (ext/int): 4,211 / 12,402
--   … MVCC seek count (ext/int): 3 / 9

A point lookup reading thousands of MVCC steps is reading a row’s entire recent history to find its current value. That is the signature of a hot counter row under a long TTL, and the fixes are the ones from chapter 10: shard the counter, shorten the TTL for that table, or stop updating one row thousands of times a second.

What to take from this chapter

• Nothing is overwritten: each write is a new version at a new timestamp, and deletes are tombstones. Visibility is decided by time, not by transaction ids.
• CockroachDB replaces Spanner’s TrueTime with hybrid logical clocks — a physical reading plus a logical counter, advanced on every message, which preserves causality without atomic clocks.
• What HLC cannot settle is handled by the uncertainty interval of width --max-offset (default 500 ms). Values found inside it trigger a push and a read refresh, and sometimes a visible 40001.
• --max-offset must be identical cluster-wide, and a node that detects excessive clock skew shuts itself down. NTP is a hard dependency, not a nicety.
• Each range publishes a closed timestamp — history below it is frozen — which underpins follower reads and changefeed checkpoints.
• AS OF SYSTEM TIME gives contention-free reads of the past, and an undo window for mistakes, bounded by the GC TTL.
• The GC TTL is one dial controlling time travel, incremental backups and changefeed resumption — against storage and scan cost. Version accumulation on churned rows is a classic slow-read cause.

Next: a write traced end to end — from the client’s INSERT through the gateway, the leaseholder, Raft and Pebble, and back again.