14. Failure modes

Symptom, cause, fix. The failures below are the ones that actually happen, roughly in order of how often you will meet them.

Retry errors flooding the application

   ERROR: restart transaction: TransactionRetryWith
   ProtoRefreshError: WriteTooOldError … (SQLSTATE 40001)

What it means. Transactions are conflicting on the same rows and the database is refusing to produce a non-serializable result (chapter 5).

When it is fine. A handful per thousand transactions, handled by a retry loop, is normal and not worth investigating.

When it is not. A climbing rate, or retries concentrated on one statement.

   FIND IT
     crdb_internal.cluster_contention_events
     Statements page → "Contention" column

   FIX IT
     • SELECT … FOR UPDATE to turn a retry into a wait
     • shorten transactions; never hold one open across
       an HTTP call, a queue publish, or user think-time
     • shard hot counter rows
     • consider READ COMMITTED for the specific workload

One node at 100% CPU while the rest idle

Cause. A hot range — almost always a monotonic key (chapter 3), sometimes a single hot row, sometimes a small table every query reads.

SHOW RANGES FROM TABLE events WITH DETAILS;   -- find it

Fix. Schema, not settings: gen_random_uuid(), unique_rowid(), or a hash-sharded index. Load-based splitting has already tried and cannot divide traffic that lands on one key.

Everything is slow, and nothing is obviously wrong

The multi-region tax (chapter 11), usually one of three arrangements:

   • table left at the default locality while its users
     are in another region
   • REGIONAL BY ROW queries that do not constrain the
     region column, so each one fans out globally
   • SURVIVE REGION FAILURE on a database that never
     needed it, taxing every write with a cross-region
     round trip

Find it. EXPLAIN ANALYZE (DIST) — look at network time and which nodes appear. High network time with low KV time is geography, every time.

The whole cluster stops accepting writes

The frightening one, and it usually has a short list of causes.

   ① NODE LIVENESS CANNOT BE WRITTEN
      a stalled disk, a saturated network, or an
      overloaded node means liveness heartbeats miss.
      Range leadership is in doubt everywhere at once.
      → check disk latency and liveness metrics (ch 4)

   ② DISK FULL
      a node at 100% capacity stops accepting writes
      for its ranges. Other nodes then take more load
      and may follow.
      → ballast file, delete something, add a node

   ③ CLOCK SKEW
      nodes shut themselves down when their clock is
      more than max-offset/2 from the majority.
      → "clock synchronization error" in the logs; fix NTP

Danger

CockroachDB writes a small ballast file on each store at startup precisely so a full disk can be recovered. If a node will not start because its store is full, removing the ballast buys you enough room to start it, delete data and rebalance. Do not delete random files from a store directory — they are the data.

Ranges unavailable

   symptom: queries touching certain tables hang forever
            (they do NOT return wrong data)
   cause:   a range has lost a MAJORITY of its replicas

Two replicas of three, gone at once: two nodes lost in the same rack, two zones lost when replicas were not diverse, or two nodes hard-removed in quick succession.

cockroach debug list-files …         # investigate
# and, only as a last resort, with data loss:
cockroach debug recover make-plan / apply-plan / verify

Unsafe range recovery reconstructs the range from a surviving replica and loses whatever that replica did not have. It is a genuine last resort, and the reason critical data gets five replicas and correct localities in the first place.

Under-replicated ranges that never recover

Normal for a few minutes after a node restart. A problem if it persists.

   • not enough nodes to satisfy the replication factor
     (3 replicas need 3 nodes; constraints may need more)
   • zone constraints that no node can satisfy — e.g.
     "+region=eu-west" with no live node there
   • every candidate node is out of disk

SELECT * FROM crdb_internal.replication_constraint_stats
WHERE violating_ranges > 0;

Reads are stale when they should not be

Almost always the read-your-own-writes trap of chapter 8: a follower read (AS OF SYSTEM TIME) used on a path where the user just wrote something.

Fix. Route post-write reads to the leaseholder with a plain SELECT, and reserve stale reads for data the user did not just change.

The LSM falls behind

   symptom: read latency climbs steadily; compaction
            backlog and read amplification rising
   cause:   sustained write rate above what the disk
            can compact; or very large bulk loads

Read amplification past roughly 10 means Pebble cannot keep up. The honest fixes are faster disks, more nodes, or less write volume; the temporary one is to pause the background job that caused it.

A schema change that will not finish

SHOW JOBS;                      -- state, fraction_completed, error
PAUSE JOB …;  RESUME JOB …;  CANCEL JOB …;

Backfills on huge tables take hours and are supposed to; that is not a failure. A job stuck at the same fraction for a long time, or repeatedly failing, usually means it is fighting foreground load or hit a constraint violation in existing data.

Caution

Cancelling a schema-change job rolls it back, which is itself a large background operation. Check SHOW JOBS until the rollback completes — an abandoned half-rolled-back change confuses every subsequent migration.

Backup or changefeed failures after an outage

   "protected timestamp record … has expired"
   "cannot resume changefeed: … below GC threshold"

Both are the same cause: the GC window passed while the job was not running (chapters 6 and 12). The incremental backup or changefeed needs history that has been collected.

Fix now. Take a new full backup; recreate the changefeed with an initial scan. Fix properly. Raise gc.ttlseconds above your incremental interval with margin, and alert on changefeed lag.

Connection storms after a node restart

A restarting node sheds its connections; every pool reconnects at once, and the survivors take the whole load.

   • drain before restarting: cockroach node drain <id>
   • health-check /health?ready=1 so the load balancer
     removes draining nodes before they go
   • cap pool sizes; a thundering pool is worse than
     a small queue

Certificates expired

   symptom: nodes cannot talk to each other, or clients
            cannot connect, all at once, with TLS errors
            — often weeks after anyone touched anything
   cause:   node or CA certificates reached their expiry

A secure cluster is entirely dependent on its certificates, and the failure arrives on a date rather than under load. Rotation is routine (cockroach cert create-node, then a rolling restart), but only if somebody knows the expiry date.

cockroach cert list --certs-dir=certs     # check the dates NOW, not later

Put the expiry in the same calendar as your domain renewals. This failure has taken down more clusters than any algorithmic subtlety in this guide.

Out of memory

   "memory budget exceeded"
   "root: memory budget exceeded: … bytes requested"

A query asked for more than the SQL memory pool allows — usually a hash join or aggregation over far more rows than the optimiser expected, which points back at statistics (chapter 9). The knobs exist (--max-sql-memory, sql.distsql.temp_storage.workmem), but raising them to make one bad query fit usually means the next one takes the node down instead.

The right order: fix the plan, then reconsider the limits.

Too many small ranges

   symptom: high CPU spent on Raft and heartbeats with
            little user traffic; slow rebalancing
   cause:   hundreds of thousands of tiny ranges — many
            small tables, or manual SPLIT AT never
            unsplit

ALTER TABLE t UNSPLIT ALL;        -- let merges reclaim them

Range merges handle this automatically once the manual split points are removed, but they will not override a split you asked for and forgot about.

A node that will not rejoin

   • wiped data directory, same node ID → it must be
     decommissioned and re-added as a NEW node, not
     restarted into its old identity
   • clock outside max-offset → fix NTP first
   • certificates regenerated for the cluster but not
     for that node → see above
   • cluster version already finalised past this
     binary's version → upgrade the binary

The logs are explicit about which of these it is; the mistake is assuming it is a network problem and restarting repeatedly.

The most dangerous non-failure

Everything works, latency is fine, no alerts fire — and nobody has ever tested a restore. The cluster survives every hardware failure it was designed for, and the first DELETE without a WHERE clause takes down the business for as long as it takes to learn how RESTORE works.

What to take from this chapter

• 40001 retries are normal in small numbers. Climbing rates mean contention: FOR UPDATE, shorter transactions, sharded counters.
• One hot node is a hot range from a monotonic key. The fix is schema design, not tuning.
• Unexplained slowness with high network time is multi-region geography: wrong locality, unconstrained REGIONAL BY ROW queries, or unnecessary region survival.
• A cluster-wide write stall is liveness, a full disk, or clock skew. The ballast file exists to rescue the full-disk case.
• Unavailable ranges mean a lost majority — queries hang, data is never wrong, and recovery is unsafe and lossy. Prevent it with correct localities and, for critical data, five replicas.
• Persistent under-replication means the allocator cannot satisfy your constraints: too few nodes, impossible zone configs, or no disk space.
• Unexpectedly stale reads are follower reads used where read-your-own-writes was required.
• Rising LSM read amplification means compaction is losing; it is a hardware or volume problem, not a knob.
• Backup and changefeed failures after an outage are the GC window expiring. Size gc.ttlseconds for your worst realistic outage.
• Drain before restarting, and cap connection pools, to avoid reconnection storms.
• The most dangerous state is a healthy cluster with an untested restore.

Next: when to use it — the honest comparison against PostgreSQL, Vitess, Spanner and the rest, including when the answer is “not this”.