13. Tuning and day-to-day operations

Cassandra has fewer knobs than PostgreSQL and a smaller number that genuinely matter. But it runs on the JVM, which adds a whole dimension of tuning that has no equivalent in a C database — and garbage collection is behind a large share of real-world Cassandra latency problems.

The JVM

The heap size question has a counter-intuitive answer: smaller is usually better.

(And in a container, set it explicitly rather than letting Cassandra’s startup script compute it — that script reads the host’s RAM, since a cgroup limit does not change /proc/meminfo. See the Docker guide’s chapter on what a container actually is.)

Cassandra deliberately keeps most of its memory off-heap — bloom filters, compression metadata, the chunk cache, and (optionally) memtables. The heap holds working state, not bulk data. A huge heap does not make Cassandra faster; it makes garbage collection pauses longer.

# jvm-server.xml (4.x+) or jvm.options
-Xms16G
-Xmx16G            # ALWAYS set Xms == Xmx. A resizing heap causes
                   # unpredictable pauses.

Guidance by collector:

Heap	Collector	Notes
8–16 GB	G1GC	the standard choice for most clusters
16–31 GB	G1GC	stay under 32 GB — see below
31 GB+	ZGC or Shenandoah	sub-millisecond pauses, more CPU
under 8 GB	G1 struggles; CMS was once used	avoid heaps this small on real workloads

The 32 GB cliff

The JVM uses compressed object pointers — 32-bit references into a heap up to about 32 GB — which saves a large amount of memory. Cross 32 GB and every pointer becomes 64 bits, so a 33 GB heap holds less usable data than a 31 GB heap while collecting more slowly.

If you want more than 31 GB, go substantially more (48 GB+) with ZGC, or you are strictly worse off. In practice, most clusters should sit at 16–31 GB with G1.

# G1GC — the standard configuration
-XX:+UseG1GC
-XX:MaxGCPauseMillis=300         # a target, not a guarantee
-XX:G1RSetUpdatingPauseTimePercent=5
-XX:InitiatingHeapOccupancyPercent=70

# Always keep these on. GC logs are the only way to diagnose pauses.
-Xlog:gc*:file=/var/log/cassandra/gc.log:time,uptime,level,tags:filecount=10,filesize=10M

Diagnosing GC

nodetool gcstats
grep -i "GC in" /var/log/cassandra/system.log | tail -20
# INFO  G1 Young Generation GC in 412ms.  G1 Eden Space: ...

Occasional young-generation pauses of tens of milliseconds are normal. Repeated pauses over a second, or any full GC, mean trouble. The usual causes, in order:

• Large partitions being read into heap (chapter 4). By far the most common.
• Tombstone-heavy reads materialising huge numbers of cells (chapter 7).
• Large IN clauses or multi-partition batches holding state on one coordinator (chapter 11).
• Too many tables, each with memtable and metadata overhead.
• A heap that is genuinely too small for the workload.

Notice that four of those five are data model problems. GC tuning rarely fixes a GC problem in Cassandra — it usually just moves it. Find the query or the partition that is causing it.

Off-heap memory

memtable_allocation_type: offheap_objects   # keeps memtables off-heap
file_cache_size: 2GiB                       # chunk cache, off-heap

Plan total memory as: heap + off-heap structures + OS page cache. Leave the OS at least 25–30% of RAM for page cache — Cassandra relies on it heavily for SSTable reads.

cassandra.yaml: the settings that matter

Most of the file can be left alone. These are the ones worth deciding deliberately.

# ─── Identity and topology ──────────────────────────────────────────
cluster_name: 'production'          # must match on every node
endpoint_snitch: GossipingPropertyFileSnitch
num_tokens: 16
allocate_tokens_for_local_replication_factor: 3

# ─── Storage ────────────────────────────────────────────────────────
data_file_directories:
  - /mnt/data1/cassandra
commitlog_directory: /mnt/commitlog/cassandra   # separate device if you can

commitlog_sync: periodic
commitlog_sync_period: 10000ms      # up to 10s of writes at risk per node
commitlog_total_space: 8GiB

# ─── Memory ─────────────────────────────────────────────────────────
memtable_allocation_type: offheap_objects
file_cache_size: 2GiB
key_cache_size: 200MiB
row_cache_size: 0MiB                # leave OFF unless you have measured

# ─── Compaction ─────────────────────────────────────────────────────
compaction_throughput: 128MiB/s     # default 64 is often too low on SSD
concurrent_compactors: 4            # ~ min(cores, number of data disks)

# ─── Concurrency ────────────────────────────────────────────────────
concurrent_reads: 32                # ~ 16 × number of data drives
concurrent_writes: 64               # ~ 8 × cores
concurrent_counter_writes: 32

# ─── Timeouts ───────────────────────────────────────────────────────
read_request_timeout: 5000ms
write_request_timeout: 2000ms
range_request_timeout: 10000ms
counter_write_request_timeout: 5000ms

# ─── Hints ──────────────────────────────────────────────────────────
hinted_handoff_enabled: true
max_hint_window: 3h

# ─── Guardrails (4.1+) — turn these ON ──────────────────────────────
tombstone_warn_threshold: 1000
tombstone_failure_threshold: 100000
batch_size_warn_threshold: 5KiB
batch_size_fail_threshold: 50KiB
partition_size_warn_threshold: 100MiB
partition_tombstones_warn_threshold: 1000

# ─── Security — the defaults are open ───────────────────────────────
authenticator: PasswordAuthenticator
authorizer: CassandraAuthorizer

Guardrails are the cheapest safety you can buy

Cassandra’s 4.1+ guardrails turn silent data-model mistakes into loud warnings and, at the failure thresholds, into errors. A cluster that refuses a 200 MB partition write in staging is far better than one that accepts it and dies in production six months later. Turn them on in every environment.

Operating system

# Disable swap entirely. A swapping Cassandra node is worse than a dead one:
# it stays "up" while responding at disk speed, so the cluster keeps
# routing traffic to it.
swapoff -a

# Generous limits
cassandra - memlock unlimited
cassandra - nofile  1048576
cassandra - nproc   32768
cassandra - as      unlimited

# sysctl
vm.max_map_count = 1048575
net.ipv4.tcp_keepalive_time = 60

# Disable transparent huge pages — a known source of latency spikes
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

# Filesystem: XFS or ext4. Use noatime.

Storage: local NVMe or SSD. Cassandra does a lot of small random reads during compaction and merges. Network-attached storage adds latency to every one of them, and EBS-style volumes with burst credits will exhaust them during a repair or a large compaction and then throttle you to a crawl at exactly the wrong moment.

What to monitor

Seven things. These catch nearly everything. Cassandra exposes them over JMX; the usual path into the Prometheus guide is the JMX exporter as a Java agent.

1. Node state

nodetool status
# every node UN. Anything else needs attention.

2. Thread pool backlogs and dropped messages

nodetool tpstats
# Pool Name              Active  Pending  Completed  Blocked  All time blocked
# MutationStage               0        0   48219342        0                0
# ReadStage                   2        0   19203847        0                0
# CompactionExecutor          2        4     182934        0                0
#
# Message type           Dropped
# MUTATION                     0     ← MUST be zero
# READ                         0

Dropped MUTATION messages mean writes were silently discarded because the node could not keep up. The coordinator may still have met the consistency level from other replicas, and repair will eventually fix it — but it is an unambiguous overload signal. Alert on any non-zero value.

Sustained Pending counts mean that stage is a bottleneck.

3. Compaction

nodetool compactionstats
# pending tasks: 3          ← climbing and staying high = losing

4. Latency

nodetool tablehistograms my_keyspace.my_table
nodetool proxyhistograms          # coordinator-level, includes network

Watch SSTables per read (chapter 6) and the p99, not the mean.

5. Partition sizes

nodetool tablestats my_keyspace | grep -i "partition maximum"

The max, not the mean.

6. Tombstones

nodetool tablestats my_keyspace | grep -i tombstone
grep -i "tombstone" /var/log/cassandra/system.log | tail

7. Repair

Did it complete, on every table, within gc_grace_seconds? This is the one people forget to monitor and the one with the worst consequences.

SELECT keyspace_name, columnfamily_name, status, finished_at
FROM system_distributed.repair_history ORDER BY id DESC LIMIT 20;

Virtual tables (4.0+)

You can query much of this over CQL, which makes it far easier to feed into dashboards:

SELECT * FROM system_views.clients;
SELECT * FROM system_views.settings WHERE name = 'compaction_throughput';
SELECT * FROM system_views.thread_pools;
SELECT * FROM system_views.sstable_tasks;
SELECT * FROM system_views.local_read_latency;

Tooling

Prometheus with the JMX exporter or the Cassandra Exporter, and one of the community Grafana dashboards. Metrics are exposed over JMX in great detail. Cassandra Reaper for repair scheduling and visibility (chapter 10) — treat it as required infrastructure, not optional.

Rolling upgrades

Cassandra upgrades in place, one node at a time, with no downtime. The rules are strict.

   1. Read the NEWS.txt for every version between yours and the target.
      It is the authoritative list of breaking changes.

   2. nodetool drain          ← flushes memtables and stops accepting
                                writes. ALWAYS do this before stopping.
   3. stop the node
   4. install the new version
   5. start the node
   6. Wait for it to be UN and for hints to drain.
   7. Next node.

During the upgrade, do not:

• Run repair. Streaming between mixed versions is not supported.
• Add, remove, or decommission nodes.
• Change the schema. Mixed-version schema propagation causes disagreement.

Upgrade the whole cluster, then do those things.

# After upgrading every node, rewrite SSTables into the new format
nodetool upgradesstables        # heavy — one node at a time

Upgrade one major version at a time. 3.11 → 4.0 → 4.1 → 5.0, not 3.11 → 5.0. Cassandra only supports streaming and schema compatibility across adjacent versions.

Schema changes

DDL propagates through gossip, and this is one place Cassandra is genuinely fragile.

ALTER TABLE users ADD nickname text;      -- instant, metadata only
ALTER TABLE users DROP nickname;          -- instant, data removed by compaction

Adding and dropping columns is cheap — there is no table rewrite, because SSTables store cells by name and a missing column is simply absent.

What you cannot do:

• Change a column’s type (with narrow exceptions).
• Change the primary key. Ever. Make a new table and migrate.
• Rename a column that is part of the primary key.

The rules that prevent schema disagreement:

   ✓ Make schema changes from ONE node, one statement at a time.
   ✓ Wait for the schema version to converge before the next change.
   ✗ Never run concurrent DDL from multiple clients.
   ✗ Never change schema during a rolling upgrade.
   ✗ Never create and drop a table with the same name in quick
     succession — this is a known source of stuck schema disagreement.

nodetool describecluster
# Schema versions:
#   a1b2c3d4-...: [10.0.1.10, 10.0.1.11, 10.0.1.12]
#   ← ONE version listed = converged. Two or more = disagreement.

A persistent schema disagreement is a real incident. The usual fix is nodetool resetlocalschema on the odd node out, and in stubborn cases a rolling restart.

Multi-tenancy and table count

Each table costs memory: a memtable, bloom filters, index summaries, metadata, and its own compaction bookkeeping.

Keep the total under a few hundred tables per cluster. At a thousand, memtables become tiny, flushes become frequent, SSTables proliferate, and compaction never rests. At several thousand, nodes struggle to start.

A pattern of “one keyspace per customer” scales badly for exactly this reason. Put the tenant in the partition key instead:

PRIMARY KEY ((tenant_id, entity_id), ...)

A checklist for a new cluster

□ cluster_name set and identical everywhere
□ GossipingPropertyFileSnitch, with dc/rack set in cassandra-rackdc.properties
□ num_tokens: 16 + allocate_tokens_for_local_replication_factor
□ 2-3 seeds per DC; no node seeds itself
□ NetworkTopologyStrategy, RF=3 per DC — even with one DC
□ system_auth replicated (NOT RF=1)
□ PasswordAuthenticator + CassandraAuthorizer; default password changed
□ internode and client TLS enabled
□ Xms == Xmx, 16-31 GB, G1GC, GC logging on
□ swap off, THP off, generous ulimits
□ commitlog on its own device if possible
□ compaction_throughput raised for SSD (128 MiB/s)
□ all guardrail thresholds set
□ compaction strategy chosen per table (TWCS for time-series)
□ clients: prepared statements + token-aware + DCAware(local_dc)
□ clients: LOCAL_QUORUM / LOCAL_ONE, never bare QUORUM / ONE
□ NTP running and monitored on every node — a correctness requirement
□ Reaper (or equivalent) scheduling repair inside gc_grace_seconds
□ backups: snapshots off-cluster, auto_snapshot on, restore tested
□ monitoring: node state, dropped mutations, pending compactions,
  p99 latency, SSTables per read, max partition size, tombstones,
  repair completion, disk free
□ disk free alert at 50% — compaction needs the headroom

What to take from this chapter

• Heap 16–31 GB with G1GC. Never cross 32 GB. Set Xms == Xmx. Keep GC logs on.
• GC problems are usually data model problems — large partitions, tombstones, big batches. Tuning the collector moves the pain rather than removing it.
• Turn on guardrails. They catch data-model mistakes before they become incidents.
• Disable swap and transparent huge pages. Use local NVMe.
• Monitor seven things; dropped mutations and pending compactions are the two earliest overload signals.
• Rolling upgrades: nodetool drain first, one node at a time, one major version at a time, and no repair, topology change, or DDL during the upgrade.
• Make schema changes from one node, serially, and verify convergence with nodetool describecluster.
• Keep table count in the low hundreds. Multi-tenancy goes in the partition key.

Next: the ways all of this goes wrong in production.