8. Compaction

Compaction is the price of the LSM tree. Writes are cheap because they append; reads are expensive because they must merge. Compaction is the background process that continuously merges SSTables so reads stay cheap — and choosing the wrong strategy for your workload is one of the few configuration mistakes that can genuinely ruin a Cassandra cluster.

What compaction does

Take several SSTables, merge them into fewer, delete the originals.

   BEFORE                                    AFTER
   ┌─────────────────────────┐
   │ SSTable 1               │
   │  alice: name='Alice'    │               ┌─────────────────────────┐
   │         email='a@old'   │               │ SSTable 5               │
   ├─────────────────────────┤   compact     │  alice: name='Alice'    │
   │ SSTable 2               │  ──────────▶  │         email='a@new'   │
   │  alice: email='a@new'   │               │  bob:   name='Bob'      │
   ├─────────────────────────┤               │  (carol dropped —       │
   │ SSTable 3               │               │   tombstone was older   │
   │  bob:   name='Bob'      │               │   than gc_grace)        │
   │  carol: TOMBSTONE       │               └─────────────────────────┘
   └─────────────────────────┘

Four jobs, all of them essential:

1. Reduce SSTables per read. The number that predicts your read latency (chapter 6).
2. Discard superseded cells. Only the highest-timestamp version survives.
3. Drop expired tombstones, once past gc_grace_seconds and once it can be proven safe.
4. Reclaim disk. Deleted and overwritten data actually goes away.

Because SSTables are immutable and sorted, compaction is a merge of sorted streams into a new file. Readers keep using the old files until the new one is complete and registered; then the old ones are deleted. Nothing blocks.

Compaction needs free disk

The new SSTable is written before the old ones are deleted, so you need room for both. Keep at least 30–50% of each data disk free. A full disk stops compaction, which causes SSTable counts to climb, which makes reads slower and disk usage worse. This spiral is one of the classic Cassandra outages (chapter 14).

The trade-off every strategy makes

You cannot have all three of these at once, and each strategy picks two:

              read amplification
              (SSTables per read)
                      ▲
                      │
                     LCS
                      │
                      │
   write ◀────────────┼────────────▶ space
   amplification     STCS          amplification
   (rewrites per                   (wasted disk)
    piece of data)

• Read amplification — how many files a read must touch.
• Write amplification — how many times the same data gets rewritten over its life.
• Space amplification — how much extra disk is held by obsolete versions.

STCS — Size-Tiered Compaction Strategy

The default. Group SSTables of similar size; when enough have accumulated, merge them into one bigger one.

   many small ──▶ ┌──┐┌──┐┌──┐┌──┐  4 similar-sized tables
                  └──┘└──┘└──┘└──┘
                        │ merge
                        ▼
                  ┌────────────┐    one bigger table
                  └────────────┘
                        │  ... which eventually joins 3 others
                        ▼          of ITS size, and so on
                  ┌──────────────────────────┐
                  └──────────────────────────┘

ALTER TABLE t WITH compaction = {
  'class': 'SizeTieredCompactionStrategy',
  'min_threshold': 4,      -- merge when 4 similar tables exist
  'max_threshold': 32
};

Good: low write amplification — data is rewritten only when its tier fills. Simple, predictable, cheap. Excellent for write-heavy, append-mostly workloads.

Bad: two serious problems.

Space amplification. Merging four 100 GB tables needs 400 GB free while it runs. Worse, a table can end up with one enormous SSTable that almost never gets compacted again, because there are no other tables its size to pair with. Obsolete data inside it is never reclaimed. Plan for up to 50% free disk.

Read amplification. A row that has been updated over months has versions in several tiers, so reads touch several files. Overwrite-heavy workloads suffer.

Use it for: append-only or write-heavy tables, tables where rows are written once, and as the default when you have no reason to choose otherwise.

LCS — Levelled Compaction Strategy

Organise SSTables into levels. Within each level, SSTables have non-overlapping key ranges — so any given partition key appears in at most one SSTable per level.

   L0  ┌──┐┌──┐┌──┐        newly flushed, overlapping, unsorted
       └──┘└──┘└──┘
        │ compact into L1
        ▼
   L1  ┌────┬────┬────┬────┐    ~160 MB total, non-overlapping
       └────┴────┴────┴────┘
        │ when L1 exceeds its budget, push into L2
        ▼
   L2  ┌────┬────┬────┬────┬────┬────┬────┬────┐   ~1.6 GB (10×)
       └────┴────┴────┴────┴────┴────┴────┴────┘
        │
   L3  ...  10× again

Each level holds ten times the previous. A read must check L0 (a few tables) plus at most one table per level, so a four-level table means roughly 4–7 SSTables in the worst case — usually far fewer, because bloom filters eliminate most of them.

ALTER TABLE t WITH compaction = {
  'class': 'LeveledCompactionStrategy',
  'sstable_size_in_mb': 160
};

Good: excellent and predictable read latency — typically 1–2 SSTables per read. Low space amplification, usually around 10% overhead. Tombstones and obsolete data are reclaimed promptly.

Bad: very high write amplification. A piece of data is rewritten every time it is promoted a level — commonly 10–30× over its lifetime. That is a large amount of continuous disk I/O and CPU.

Use it for: read-heavy tables, overwrite-heavy tables where the same rows are updated repeatedly, tables where read latency must be consistent, and tables where you cannot afford STCS’s space overhead. Do not use it for write-heavy or append-only tables — compaction will never catch up, and you will watch pending compactions climb forever.

TWCS — Time-Window Compaction Strategy

Purpose-built for time-series data with a TTL. Group SSTables by the time window their data was written in, and compact only within a window. Once a window is complete, its SSTable is never compacted again.

   ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
   │ Aug 17   │ │ Aug 18   │ │ Aug 19   │ │ Aug 20   │
   │ (sealed) │ │ (sealed) │ │ (sealed) │ │ (active) │
   └──────────┘ └──────────┘ └──────────┘ └──────────┘
        │
        │  when everything inside has passed its TTL,
        │  the WHOLE FILE is deleted
        ▼
      *poof*   ← no merging, no tombstone scanning, no I/O

CREATE TABLE readings (...)
WITH compaction = {
  'class': 'TimeWindowCompactionStrategy',
  'compaction_window_unit': 'DAYS',
  'compaction_window_size': 1
}
AND default_time_to_live = 2592000        -- 30 days
AND gc_grace_seconds = 3600;              -- see the note below

This is the magic of TWCS: expired data is removed by deleting whole files. There is no tombstone merge, no compaction pass, no read amplification from expiry. It is the cheapest possible way to age out data, and it is why TWCS is not just “a bit better” for time-series — it is categorically different.

It also helps reads enormously. Because each SSTable covers a known time window, the min/max timestamp check (chapter 6) eliminates every file outside your query’s range before reading a byte.

Sizing the window: aim for 20–30 windows in the table’s total lifetime. 30 days of retention → daily windows. A year of retention → weekly or monthly. Too many windows means too many SSTables to check; too few means large files and delayed expiry.

The rules, and they are strict:

• Every write must have a TTL, and ideally the same one. A row without a TTL sits in its window forever and stops that file from ever being dropped.
• Do not delete explicitly. A tombstone written today applies to data in an old window, which forces a cross-window compaction and defeats the entire design.
• Do not write out-of-order timestamps. Backfilling last month’s data into today’s window breaks the time-bounded property that makes reads fast.
• gc_grace_seconds can be low, often 3600 or even 0, because you are not generating tombstones — but only if the previous two rules genuinely hold.

Use it for: metrics, sensor readings, logs, events, audit trails — any append-only, TTL’d, time-ordered data. It is the correct answer for that workload and there is rarely a reason to choose otherwise.

DTCS is dead

DateTieredCompactionStrategy was TWCS’s predecessor. It was fragile and hard to tune, and TWCS replaced it. If you find DTCS on an old cluster, migrate to TWCS.

UCS — Unified Compaction Strategy (Cassandra 5.0)

New in 5.0, and it is the most significant compaction change in years. UCS unifies tiered and levelled compaction into one strategy with a single parameter that slides between them.

ALTER TABLE t WITH compaction = {
  'class': 'UnifiedCompactionStrategy',
  'scaling_parameters': 'T4',    -- tiered, 4 SSTables per level
  'target_sstable_size': '1GiB'
};

The scaling_parameters value controls the shape:

   T4    tiered, fan-in 4        ≈ STCS      write-optimised
   T2    tiered, fan-in 2                    even more write-optimised
   N     the neutral middle
   L4    levelled, fan-out 4     ≈ LCS       read-optimised
   L10   levelled, fan-out 10                ≈ classic LCS

   You can even specify per level:
   'T8, T4, N, L4'   ← write-optimised at the bottom,
                       read-optimised at the top

Two things make this genuinely better than choosing between STCS and LCS:

You can change your mind without a rewrite. Moving from STCS to LCS on an existing table forces a complete recompaction of everything. Adjusting a UCS scaling parameter reshapes gradually.

It fixes STCS’s worst failure. UCS produces SSTables of a bounded target_sstable_size rather than letting one file grow without limit, so the “giant SSTable that never gets compacted again” problem disappears. Sharding by token range also means compactions are smaller and more parallel.

If you are on Cassandra 5.0, UCS is a reasonable default for new tables — with the exception of time-series, where TWCS’s whole-file expiry remains unbeatable.

Choosing

Workload	Strategy
Time-series with TTL	TWCS — no contest
Write-heavy, append-only	STCS (or UCS T4)
Read-heavy	LCS (or UCS L4)
Overwrite-heavy, same rows updated repeatedly	LCS
Disk-constrained	LCS or UCS — STCS needs 50% free
No idea yet	STCS, then measure
On Cassandra 5.0, general purpose	UCS

Compaction strategy is per table, and it should be. A cluster typically has TWCS on its event tables, LCS on its lookup tables, and STCS on its write-heavy ones.

Watching compaction

# Is it keeping up? "pending tasks" is the number that matters.
nodetool compactionstats
# pending tasks: 4
#    id       compaction type   keyspace   table     completed   total   unit
#    abc-123  Compaction        ks         messages  4.21 GiB   9.8 GiB  bytes

# SSTables per read — the outcome metric
nodetool tablehistograms ks.messages

# Per-table state
nodetool tablestats ks.messages | grep -Ei "sstable count|space used|compaction"

pending tasks climbing and never returning to near zero is the alarm. It means compaction is losing to your write rate. What follows is predictable: SSTable counts rise, reads get slower, disk fills, and eventually the node cannot recover without intervention.

The responses, roughly in order:

Give compaction more throughput. The default throttle is often far too low for SSD.

nodetool getcompactionthroughput
nodetool setcompactionthroughput 128        # MiB/s, live, no restart

cassandra.yaml

compaction_throughput: 128MiB/s     # default 64 in 4.x; 16 in older versions
concurrent_compactors: 4            # ~= min(cores, data disks). Not too high.

Setting throughput to 0 means unthrottled. That is occasionally the right emergency move and a bad steady state — it will starve your read and write path of I/O.

Check the strategy is right for the workload. LCS on a write-heavy table can never keep up, by design. This is the most common root cause.

Check for oversized partitions. A single 200 GB partition must be compacted as one unit.

Add nodes. If compaction cannot keep up on adequate hardware with the right strategy, the cluster is simply undersized.

Manual compaction

nodetool compact my_keyspace my_table

Usually a mistake. With STCS, a major compaction merges everything into one enormous SSTable which then has no peers of its size and will never be compacted again — so obsolete data inside it is never reclaimed. You have traded a temporary problem for a permanent one.

The legitimate uses:

# After a huge delete, to actually reclaim the space
nodetool compact --split-output ks tbl   # STCS: produce several files,
                                         # not one giant one

# Rewrite specific SSTables that are mostly tombstones
nodetool garbagecollect ks tbl

# Remove data this node no longer owns, after a topology change
nodetool cleanup ks

nodetool cleanup is the one people forget. After adding a node, the existing nodes still hold data for ranges they no longer own. It is not served, but it occupies disk. Run cleanup on every pre-existing node after a topology change — one node at a time, since it is I/O intensive.

Compaction and repair interact

Two settings worth knowing:

unchecked_tombstone_compaction (default false) lets compaction rewrite a single SSTable purely to drop tombstones, even when normal thresholds are not met. Turning it on helps tombstone-heavy tables, at the cost of extra I/O.

Incremental repair (chapter 10) marks data as repaired, and repaired and unrepaired data are kept in separate compaction pools so they never merge. This is correct but it means an incrementally-repaired table effectively has two sets of SSTables, which raises SSTable counts. It is one reason many operators prefer full repairs.

What to take from this chapter

• Compaction merges SSTables to keep reads fast, drop obsolete data, and reclaim disk. It needs 30–50% free disk.
• Every strategy trades read amplification against write amplification against space.
• STCS: cheap writes, needs lots of free disk, worse reads. The default.
• LCS: 1–2 SSTables per read, low space overhead, very expensive writes. Read-heavy and overwrite-heavy tables only.
• TWCS: for TTL’d time-series. Expired data is removed by deleting whole files. Do not delete explicitly, do not write without a TTL, do not backfill.
• UCS (5.0) unifies tiered and levelled behind one tunable parameter and fixes STCS’s giant-SSTable problem.
• Strategy is per table. Set it per table.
• Watch nodetool compactionstats pending tasks. Climbing and staying high is the alarm.
• Avoid nodetool compact. Remember nodetool cleanup after topology changes.

Next: the arithmetic of consistency levels.