16. Cheat sheet

Everything from the previous fifteen chapters, condensed.

Incident response

# 1. Is the cluster healthy?
nodetool status                 # all UN?
nodetool describecluster        # ONE schema version?

# 2. Overloaded? Dropping work?
nodetool tpstats                # dropped MUTATION must be 0
nodetool compactionstats        # pending tasks climbing = losing

# 3. Where is the latency?
nodetool proxyhistograms                    # coordinator + network
nodetool tablehistograms <ks>.<tbl>         # SSTables/read, p99

# 4. Disk
df -h
nodetool listsnapshots          # very often the answer

# 5. GC
nodetool gcstats
grep -i "GC in" /var/log/cassandra/system.log | tail -20

# 6. Data model smells
grep -iE "tombstone|batch|large partition" \
     /var/log/cassandra/system.log | tail -30

# 7. Clocks (a correctness issue, not hygiene)
chronyc tracking

# 8. Which nodes hold this key?
nodetool getendpoints <ks> <tbl> '<partition-key>'

TRACING ON;    -- then run the slow query. Shows the whole read path.

Health checks

nodetool status                 # UN / DN / UJ / UL, load, ownership %
nodetool info                   # heap, caches, uptime
nodetool ring                   # every token
nodetool netstats               # streaming progress
nodetool gossipinfo             # this node's view of everyone
nodetool toppartitions <ks> <tbl> 10000    # find hot partitions

-- Virtual tables (4.0+)
SELECT * FROM system_views.clients;
SELECT * FROM system_views.thread_pools;
SELECT * FROM system_views.sstable_tasks;
SELECT * FROM system_views.settings WHERE name = 'compaction_throughput';

-- Repair history
SELECT keyspace_name, columnfamily_name, status, finished_at
FROM system_distributed.repair_history ORDER BY id DESC LIMIT 20;

Reading tablehistograms

Percentile  SSTables  Write Latency  Read Latency  Partition Size  Cell Count
50%             1.00          35.43        124.00            1109          50
99%             8.00          73.46       2346.80         3379391       73457
Max            24.00         182.79      20924.30       186563160     1629722
                    ▲                                         ▲
      SSTables/read p99 > ~10                      Max partition > 100 MB
      → compaction problem                         → data model problem

Always read Max for partition size, and p99 for latency. The mean hides everything.

Data model rules

PRIMARY KEY ( (partition_key, bucket), clustering_1, clustering_2 )
--            └── decides the NODE ──┘  └── decides DISK ORDER ──┘

✓ Every query supplies the FULL partition key
✓ Clustering columns used left to right; range on the last one only
✓ ORDER BY can only follow or reverse the clustering order
✓ Keep partitions under ~100 MB and ~100k rows  ← THE number
✓ One table per access pattern; denormalise freely
✓ Bucket by time or hash to bound partition size
✗ No joins. No aggregates across partitions.
✗ ALLOW FILTERING in application code is a bug
✗ Never write null — it creates a tombstone. Use UNSET.
✗ Do not build queues on Cassandra

-- The time-series shape
CREATE TABLE readings (
  sensor_id uuid, day date, reading_time timestamp, value double,
  PRIMARY KEY ((sensor_id, day), reading_time)
) WITH CLUSTERING ORDER BY (reading_time DESC)
  AND compaction = {'class':'TimeWindowCompactionStrategy',
                    'compaction_window_unit':'DAYS',
                    'compaction_window_size':1}
  AND default_time_to_live = 2592000
  AND gc_grace_seconds = 3600;

Consistency

              R + W > RF   ⇒  the read sees the latest write

   quorum = floor(RF/2) + 1
   RF=3 → 2       RF=5 → 3       RF=2 → 2  (no fault tolerance!)

RF=3	W	R	Consistent?
ONE / ONE	1	1	no
QUORUM / QUORUM	2	2	yes — the standard
ONE / ALL	1	3	yes
ALL / ONE	3	1	yes

   MULTI-DC:  'dc_east': 3, 'dc_west': 3  ⇒ RF = 6
              QUORUM       = 4  ← crosses the WAN. Never use.
              LOCAL_QUORUM = 2  ← use this.

   ALWAYS use LOCAL_QUORUM / LOCAL_ONE in multi-DC.

   UnavailableException   → definitely NOT applied. Safe to retry.
   WriteTimeoutException  → UNKNOWN. Retry only if idempotent.
   ReadTimeoutException   → safe to retry.

Compaction

Workload	Strategy
Time-series with TTL	TWCS — expired files are deleted whole
Write-heavy, append-only	STCS (or UCS T4)
Read-heavy or overwrite-heavy	LCS (or UCS L4)
Disk-constrained	LCS / UCS (STCS needs 50% free)
Cassandra 5.0, general purpose	UCS

ALTER TABLE t WITH compaction = {'class':'LeveledCompactionStrategy'};
ALTER TABLE t WITH compaction = {'class':'UnifiedCompactionStrategy',
                                 'scaling_parameters':'T4',
                                 'target_sstable_size':'1GiB'};

nodetool compactionstats
nodetool setcompactionthroughput 128        # MiB/s, live
nodetool garbagecollect <ks> <tbl>          # rewrite to drop tombstones
nodetool cleanup <ks>                       # AFTER adding nodes
# nodetool compact  ← usually a mistake with STCS

Keep 30–50% disk free. Compaction writes the new file before deleting the old ones.

Repair — not optional

   Run repair on every table at least once every gc_grace_seconds.
   Default gc_grace = 10 days  ⇒  repair weekly.

   Skip it and deleted data comes back, permanently.

nodetool repair -pr <ks>              # scheduled: run on EVERY node
nodetool repair -pr -local <ks>       # stay in the local DC
nodetool repair -full <ks>            # explicitly full, not incremental

Use Cassandra Reaper. Hand-rolled cron is how repair silently stops happening.

   node down < 3h            → hints replay automatically
   node down 3h – gc_grace   → MUST repair
   node down > gc_grace      → WIPE AND REBUILD. Do not repair.

Topology operations

# Add a node: correct seeds + cluster_name, then start. Auto-bootstraps.
nodetool status                    # UJ while joining
nodetool cleanup                   # then on every OTHER node, one at a time

nodetool decommission              # remove a live node gracefully
nodetool removenode <host_id>      # remove a dead one
nodetool rebuild -- <source_dc>    # stream data from another DC

# Replace a dead node (start the new one with):
-Dcassandra.replace_address_first_boot=<old_ip>

✓ One node at a time
✓ nodetool cleanup after every topology change
✗ Never put a joining node in its own seed list

Adding a data centre

1. Switch clients to LOCAL_QUORUM / LOCAL_ONE   ← MUST BE FIRST
2. New nodes: dc=dc_west, auto_bootstrap: false, seeds from both DCs
3. Start them
4. ALTER KEYSPACE ... {'dc_east':3, 'dc_west':3}
5. nodetool rebuild -- dc_east   on each new node
6. Point clients at the new DC with local_dc=dc_west

Backups

nodetool snapshot -t <name> <ks>       # hard links; instant
nodetool listsnapshots                 # check this when disk is full
nodetool clearsnapshot -t <name> <ks>
nodetool refresh <ks> <tbl>            # pick up restored SSTables live
sstableloader -d <host> /path/to/snapshot/<ks>/<tbl>

auto_snapshot: true          # snapshot before DROP/TRUNCATE — keep this ON
incremental_backups: true    # you must copy off and clean up yourself

Replication is not backup. TRUNCATE replicates in milliseconds. Store backups off-cluster. Test restores.

Configuration essentials

cluster_name: 'production'
endpoint_snitch: GossipingPropertyFileSnitch
num_tokens: 16
allocate_tokens_for_local_replication_factor: 3

commitlog_directory: /mnt/commitlog/cassandra   # separate device
commitlog_sync: periodic
commitlog_sync_period: 10000ms

memtable_allocation_type: offheap_objects
file_cache_size: 2GiB
key_cache_size: 200MiB
row_cache_size: 0MiB                # leave OFF

compaction_throughput: 128MiB/s
concurrent_compactors: 4
concurrent_reads: 32
concurrent_writes: 64

hinted_handoff_enabled: true
max_hint_window: 3h

# Guardrails — turn ALL of these on
tombstone_warn_threshold: 1000
tombstone_failure_threshold: 100000
batch_size_warn_threshold: 5KiB
batch_size_fail_threshold: 50KiB
partition_size_warn_threshold: 100MiB

# Security — defaults are wide open
authenticator: PasswordAuthenticator
authorizer: CassandraAuthorizer

cassandra-rackdc.properties

dc=dc_east
rack=rack1

# JVM
-Xms16G
-Xmx16G                 # Xms == Xmx, ALWAYS. Never exceed 31G with G1.
-XX:+UseG1GC
-XX:MaxGCPauseMillis=300
-Xlog:gc*:file=/var/log/cassandra/gc.log:time,uptime,level,tags

# OS
swapoff -a                                    # never swap
echo never > /sys/kernel/mm/transparent_hugepage/enabled
# ulimits: nofile 1048576, memlock unlimited, nproc 32768
# vm.max_map_count = 1048575

Client configuration

from cassandra.cluster import Cluster
from cassandra.policies import TokenAwarePolicy, DCAwareRoundRobinPolicy
from cassandra import ConsistencyLevel

cluster = Cluster(
    contact_points=['10.0.1.10', '10.0.1.11'],
    load_balancing_policy=TokenAwarePolicy(
        DCAwareRoundRobinPolicy(local_dc='dc_east')),
)
session = cluster.connect('app')

stmt = session.prepare("SELECT * FROM users WHERE id = ?")   # ALWAYS prepare
stmt.consistency_level = ConsistencyLevel.LOCAL_QUORUM
session.execute(stmt, (user_id,))

# Never write null — use UNSET_VALUE
from cassandra.query import UNSET_VALUE

✓ Prepared statements everywhere (needed for token-aware routing)
✓ Token-aware + DC-aware load balancing with an explicit local_dc
✓ LOCAL_QUORUM / LOCAL_ONE
✓ Idempotent writes, so a timeout can be retried
✗ Never large IN lists — issue parallel async queries instead
✗ Never multi-partition batches

LWT, batches, counters

	Atomic	Isolated	Cost	Use when
Normal write	per partition	no	1×	almost always
Single-partition batch	yes	yes	~1×	several statements, one partition
Logged batch	eventually	no	~3×	a few denormalised tables in sync
LWT	one partition	yes	~10–20×	rare compare-and-set
Counter	yes	no	~2×	approximate counting only

INSERT INTO users (username) VALUES ('alice') IF NOT EXISTS;
-- SERIAL CONSISTENCY LOCAL_SERIAL;  in multi-DC

Counters are not idempotent — never retry a timed-out counter write.

Upgrades

1. Read NEWS.txt for every intervening version
2. nodetool drain, stop, install, start — ONE NODE AT A TIME
3. NO repair, NO topology changes, NO DDL during the upgrade
4. After all nodes: nodetool upgradesstables (one at a time)
5. One major version at a time: 3.11 → 4.0 → 4.1 → 5.0

Rules of thumb

Partition size            < 100 MB and < 100k rows
SSTables per read (p99)   1-3 good, > 10 is a problem
Replication factor        3 per DC, NetworkTopologyStrategy always
Consistency               LOCAL_QUORUM / LOCAL_ONE
Racks                     1, or exactly RF
num_tokens                16
Heap                      16-31 GB, G1GC, never cross 32 GB
Disk free                 keep 50%
Tables per cluster        low hundreds, not thousands
Repair                    weekly (within gc_grace_seconds)
Dropped mutations         must be 0
Pending compactions       must return to ~0
NTP drift                 alert at tens of milliseconds

The mental model, in ten lines

1. Every node is identical. There is no primary and no failover.
2. Consistent hashing places data; walk clockwise for RF replicas.
3. Nothing on disk is ever modified. Writes append; SSTables are immutable.
4. A write is: append to the commit log, insert into the memtable. That is all.
5. A read must merge the memtable and every SSTable that might hold the key.
6. Highest timestamp wins. That is the entire conflict-resolution model.
7. A delete is a write — a tombstone — and it makes your data bigger.
8. Compaction is the rent you pay for cheap writes.
9. R + W > RF is the only consistency guarantee you get.
10. The data model is permanent. Design it from the queries.

Where to go next

• The official documentation — much improved in recent years.
• The Dynamo and BigTable papers — Cassandra is their marriage, and reading them makes the design feel inevitable rather than arbitrary.
• CEP proposals on the Apache wiki — where the future (Accord, transactions) is being designed in public.
• The cassandra-user and dev mailing lists, and the ASF Slack.
• DataStax’s Cassandra data modelling material — the best free treatment of chapter 4’s subject.

You now understand what happens between INSERT and the bytes hitting the disk on three machines. Open nodetool tablehistograms on your own cluster and see how much of it you recognise.