14. Failure modes and war stories

Every mechanism in this guide has a failure mode. Here they are, roughly in order of how often they bite, each with the symptom you will actually see.

1. The wide partition

Symptom. One node has much higher latency than its peers. Long GC pauses, then OutOfMemoryError and a restart. Reads on one table time out while everything else is fine.

Cause. A partition grew without bound (chapter 4). PRIMARY KEY ((sensor_id), reading_time) with a sensor reporting every second is 31 million rows in one partition, on three nodes, read as a unit.

Diagnose.

nodetool tablehistograms my_keyspace.readings
# Look at the Max row for Partition Size, not the mean.

nodetool tablestats my_keyspace | grep -i "partition maximum"

Fix now. There is no quick fix. You must change the primary key, which means creating a new table and migrating. You cannot alter a primary key in place.

Interim relief: reduce read_request_timeout pressure by making the application query narrow clustering ranges instead of whole partitions, and raise heap slightly to survive.

Prevent. Bucket the partition key at design time. Turn on partition_size_warn_threshold. Ask “what is the maximum this partition can reach in three years?” for every table.

2. Tombstone accumulation

Symptom.

WARN  Read 10 live rows and 98234 tombstone cells for query ...
ERROR Scanned over 100000 tombstones during query; query aborted

Reads on a table get gradually slower over weeks, then start failing.

Cause. Chapter 7. Delete-heavy patterns, a queue built on Cassandra, TTL’d data on the wrong compaction strategy, or — very commonly — an ORM writing null for unset columns on every write.

Diagnose.

nodetool tablestats my_keyspace | grep -Ei "table:|tombstone"
sstablemetadata /var/lib/cassandra/data/ks/tbl-*/nb-*-Data.db | grep -i droppable

Fix now.

# Rewrite SSTables purely to drop tombstones
nodetool garbagecollect my_keyspace my_table

# If gc_grace has passed and you need the space, temporarily:
ALTER TABLE t WITH gc_grace_seconds = 3600;   # then run compaction
# ← only after confirming repair is current

Prevent. Use UNSET_VALUE rather than null. Use TTLs with TWCS so expiry drops whole files. Delete whole partitions rather than rows. Do not build queues on Cassandra.

3. Compaction falls behind

Symptom. SSTable count climbs steadily. Read latency degrades. Disk usage grows faster than data. pending tasks in nodetool compactionstats never returns to near zero.

Cause. Write rate exceeds compaction throughput; or LCS on a write-heavy table (it cannot keep up, by design); or the throttle is too low for the hardware; or the disk is too full to compact.

Diagnose.

nodetool compactionstats           # pending tasks climbing?
nodetool tablehistograms ks.tbl    # SSTables per read rising?
nodetool getcompactionthroughput
df -h

Fix now.

nodetool setcompactionthroughput 256      # live, no restart
# and raise concurrent_compactors if CPU allows

Prevent. Right compaction strategy per table (chapter 8). compaction_throughput tuned for SSD. Keep 50% disk free. Alert on pending compactions.

4. Disk full

Symptom. Compaction stops. Writes start failing. The node may refuse to start.

Cause. Cassandra needs free space to compact — it writes the new SSTable before deleting the old ones. Past roughly 70–80% full, STCS in particular can no longer merge its largest tables, so the situation compounds.

Contributing causes, and it is usually more than one: forgotten snapshots (hard links keeping deleted SSTables alive), incremental backups that were never cleaned up, no nodetool cleanup after adding nodes, and bloat from a bad compaction strategy.

Diagnose.

nodetool listsnapshots            # very often the answer
du -sh /var/lib/cassandra/data/*/*/backups/
nodetool tablestats | grep -i "space used"

Fix now.

nodetool clearsnapshot            # frees space instantly
rm -rf /var/lib/cassandra/data/*/*/backups/*   # after copying them off
nodetool cleanup                  # if a topology change happened

Prevent. Alert at 50% disk, not 80%. A snapshot-retention routine. cleanup after every topology change.

5. Repair never runs, and deleted data comes back

Symptom. Rows you deleted weeks ago reappear. Replicas return different answers for the same query.

Cause. Chapter 7 and 10. Repair did not complete within gc_grace_seconds, so a tombstone was dropped on some replicas while another replica still held the original data — which then got repaired outward.

Diagnose.

SELECT keyspace_name, columnfamily_name, status, finished_at
FROM system_distributed.repair_history ORDER BY id DESC LIMIT 20;

If that is empty or stale, this is your answer.

Fix now. Delete the resurrected data again, and get repair running. There is no way to undo a resurrection.

Prevent. Run Reaper. Alert on repair completion, not just repair start. Treat “repair has not completed on table X in 10 days” as a page-worthy condition.

6. Clock skew corrupts data

Symptom. Writes silently do nothing. A user updates a value, gets success, reads it back, and sees the old value. No errors anywhere.

Cause. Chapter 7. A node’s clock is ahead, so writes it coordinates get future timestamps and beat genuinely newer writes forever.

Diagnose.

# On every node
timedatectl status
chronyc tracking

# In CQL — is the writetime in the future?
SELECT writetime(col) FROM t WHERE ...;

Fix now. Fix the clock. Then find and rewrite the affected data with a higher timestamp — there is no automatic repair for this.

Prevent. NTP or chrony on every node, monitored, with an alert on drift above a few tens of milliseconds. This is a data-integrity control, not sysadmin hygiene.

7. Multi-DC QUORUM

Symptom. After adding a second data centre, latency jumps from 3 ms to 180 ms on every query. Or reads start failing with UnavailableException the moment the WAN link degrades.

Cause. Chapter 9. With 'dc_east': 3, 'dc_west': 3, total RF is 6 and QUORUM is 4 — which cannot be satisfied locally.

Fix now. Change the client consistency level to LOCAL_QUORUM. It is a one-line change and the effect is immediate.

Prevent. Never use bare QUORUM or ONE. Switch clients to LOCAL_* before adding a data centre, not after.

8. GC pauses and node flapping

Symptom. Nodes intermittently marked down and then up again, with no network problem. Latency spikes correlated across the cluster. system.log full of long GC entries.

Cause. A stop-the-world pause longer than the failure detector’s tolerance makes peers conclude the node is dead. When it resumes, it comes back — and gossip churns.

The underlying trigger is nearly always one of: wide partitions read into heap, tombstone-heavy reads, large IN clauses or multi-partition batches, or a heap over 32 GB.

Diagnose.

nodetool gcstats
grep -i "GC in" /var/log/cassandra/system.log | tail -50
grep -i "gossip" /var/log/cassandra/system.log | tail

Fix now. Raise phi_convict_threshold to 10–12 as a stopgap so nodes stop flapping. Then find the query causing the allocation.

Prevent. Heap 16–31 GB, G1GC, Xms == Xmx. Guardrails on partition size, tombstones, and batch size. Do not treat this as a GC tuning problem — it is almost always a data model or query problem.

9. Hot partitions

Symptom. Three nodes at 90% CPU while the other seventeen idle. Latency bad only for one table.

Cause. A partition key with low cardinality or extreme skew. PRIMARY KEY ((country), ...) gives 200 partitions. PRIMARY KEY ((status), ...) gives about five, and one of them holds most of the rows.

Diagnose.

nodetool status         # is Load uneven across nodes?
nodetool toppartitions my_keyspace my_table 10000   # sample the hottest keys

Fix now. Nothing quick. Add a bucket to the partition key, which means a new table and a migration.

Prevent. At design time, ask how many distinct partition keys this table will have and whether traffic is even across them. You want far more partitions than nodes, with comparable traffic.

10. A node down longer than gc_grace_seconds

Symptom. A node has been out for two weeks. Someone restarts it. Deleted data reappears cluster-wide.

Cause. The returning node holds rows whose tombstones were already dropped elsewhere. Read repair and repair propagate its stale data outward as though it were the truth.

Fix now. Do not repair it. Wipe its data directories and rebuild:

# Stop the node, delete data/commitlog/saved_caches, then start with:
-Dcassandra.replace_address_first_boot=<its_own_ip>

Prevent. A policy: any node down longer than gc_grace_seconds gets rebuilt, never restarted. Alert on nodes that have been DN for hours.

11. Schema disagreement

Symptom. nodetool describecluster lists two or more schema versions. Queries fail unpredictably. Some nodes do not know about a new table.

Cause. Concurrent DDL from multiple clients, DDL during a rolling upgrade, or a table created and dropped with the same name in quick succession.

Diagnose.

nodetool describecluster
# Schema versions:
#   a1b2...: [10.0.1.10, 10.0.1.11]
#   c3d4...: [10.0.1.12]      ← disagreement

Fix now.

nodetool resetlocalschema     # on the minority node
# if that fails, a rolling restart of the affected nodes

Prevent. DDL from one client, one statement at a time, verifying convergence between statements. Never during an upgrade.

12. The queue that ate the cluster

Symptom. A table used as a job queue gets slower every day, then starts throwing tombstone errors, then takes the cluster down.

Cause. Chapter 7’s worked example. Deletes at the head of a partition that reads always start from. Every element of Cassandra’s design compounds the problem.

Fix now. TRUNCATE the table if you can afford to (it removes the SSTables wholesale rather than writing tombstones), and move the workload off Cassandra.

Prevent. Use a queue. Kafka, SQS, RabbitMQ, Pulsar. This is not a Cassandra workload.

An incident checklist

# 1. Is the cluster healthy?
nodetool status
nodetool describecluster        # one schema version?

# 2. Is anything overloaded or dropping work?
nodetool tpstats                # dropped MUTATION must be 0
nodetool compactionstats        # pending tasks climbing?

# 3. Where is the latency?
nodetool proxyhistograms
nodetool tablehistograms <ks>.<tbl>

# 4. Disk?
df -h
nodetool listsnapshots          # very often the answer

# 5. GC?
nodetool gcstats
grep -i "GC in" /var/log/cassandra/system.log | tail -20

# 6. Data model smells?
grep -iE "tombstone|batch|large partition" /var/log/cassandra/system.log | tail -30

# 7. Clocks?
chronyc tracking

# 8. Repair current?
# → check Reaper, or system_distributed.repair_history

And on a single slow query:

TRACING ON;
-- run it

The pattern behind all of these

Read back through the twelve. Eight of them are data model problems that surfaced months after the model was written — wide partitions, tombstones, hot partitions, the queue, the compaction strategy mismatch, null writes.

Three are operational hygiene that was never scheduled: repair, snapshots, NTP.

Only one — schema disagreement — is really about the software behaving unexpectedly.

That distribution is the honest summary of running Cassandra. The database is remarkably reliable. The failure mode is a data model that was correct at 10 GB and wrong at 10 TB, discovered when changing it requires a migration.

Which is why chapter 4 is the most important chapter in this guide, and why it is worth spending a day on the model before writing the first row.

Next: whether you should be using Cassandra at all.