3. The storage engine on disk

This is the foundation chapter, the counterpart to “what the bytes on disk look like” in the PostgreSQL guide. Almost everything about Cassandra’s behaviour — why writes are fast, why reads are not, why deletes create data, why compaction exists — comes from the structure described here.

The idea: never overwrite anything

PostgreSQL uses a B-tree: data is kept sorted in place, and writing means finding the right page and modifying it. Reads are simple; writes require a seek and a read-modify-write.

Cassandra uses a log-structured merge tree, or LSM tree. The rule is:

Never modify anything on disk. Only ever append new files.

Writes become pure sequential appends — no seek, no read, no in-place modification. That is why Cassandra absorbs writes at rates that make a B-tree engine cry.

The bill arrives at read time. Since a row’s history is spread across however many files were written since it first appeared, a read may have to look in many places and merge what it finds. Everything difficult about Cassandra reads (chapter 6) and the entire existence of compaction (chapter 8) is the cost of this one decision.

   B-TREE (PostgreSQL)              LSM TREE (Cassandra)
   ─────────────────────            ─────────────────────
   write: seek, read page,          write: append to log,
          modify, write back               insert into RAM
          → random I/O                     → sequential I/O

   read:  descend the tree          read:  check RAM, then check
          → one lookup                     every file that might
                                           hold this key, then merge

The three pieces

                        write
                          │
              ┌───────────┴────────────┐
              ▼                        ▼
     ┌─────────────────┐      ┌──────────────────┐
     │   COMMIT LOG    │      │     MEMTABLE     │
     │   append-only   │      │  sorted, in RAM  │
     │   on disk       │      │  one per table   │
     │   for crash     │      └────────┬─────────┘
     │   recovery only │               │ when full: FLUSH
     └─────────────────┘               ▼
                              ┌──────────────────┐
                              │     SSTABLE      │
                              │  sorted file     │
                              │  on disk         │
                              │  IMMUTABLE       │
                              └──────────────────┘
                                       │
                                       │ many SSTables accumulate
                                       ▼
                                  COMPACTION
                              merges them into fewer

Three structures, and the whole engine is how they interact.

The commit log

An append-only file whose only purpose is crash recovery. Every mutation is appended to it before being acknowledged.

It is never read during normal operation. Not once. It exists solely so that if the machine loses power with a memtable full of unflushed data, that data can be replayed on restart.

   /var/lib/cassandra/commitlog/
   ├── CommitLog-7-1724150400000.log     ← being written
   ├── CommitLog-7-1724150300000.log
   └── CommitLog-7-1724150200000.log

Segments are 32 MB by default. Once every memtable containing data from a segment has been flushed, that segment is discarded — the data is safely in SSTables and the log is redundant.

The durability setting that matters

commitlog_sync: periodic
commitlog_sync_period: 10000ms       # fsync every 10 seconds

This is the default, and it is worth understanding exactly what it means: a write is acknowledged as soon as it is in the operating system’s page cache, without waiting for it to reach the physical disk. If the machine loses power, up to 10 seconds of acknowledged writes on that node can be lost.

That sounds alarming and mostly is not, because of replication. With RF=3, the same write exists on three machines whose power supplies are independent. Losing all three simultaneously in the same 10-second window is a different class of event. Cassandra’s durability strategy is “many copies,” not “one very careful copy.”

If that is not enough:

commitlog_sync: batch          # fsync before acknowledging. Slower.
# or
commitlog_sync: group          # batch up writes, fsync every N ms
commitlog_sync_group_window: 2ms

group is the sensible middle ground: it amortises the fsync across concurrent writes rather than doing one per write.

The commit log is pure sequential write. SSTable flushes and compaction are large sequential reads and writes. On separate devices they do not interfere; on the same device they do.

commitlog_directory: /mnt/commitlog/cassandra
data_file_directories:
  - /mnt/data1/cassandra
  - /mnt/data2/cassandra

This matters much less on NVMe than it did on spinning disks, but it is still measurable under heavy write load.

The memtable

An in-memory, sorted structure — one per table — holding recent writes. Rows are kept sorted by partition token and then by clustering columns, so the memtable is always in the order it will eventually be written to disk.

Since Cassandra 5.0 the default implementation is a trie-based memtable, which is more memory-efficient and faster to look up than the previous skip-list-like structure.

A memtable is flushed to disk when any of these happen:

Its memory footprint crosses the threshold (memtable_cleanup_threshold against memtable_heap_space / memtable_offheap_space).
The commit log grows past commitlog_total_space and needs to recycle a segment.
Someone runs nodetool flush or nodetool drain.
The table’s memtable_flush_period_in_ms elapses.

Flushing writes the memtable out as one new SSTable, sequentially, in sorted order — no seeking, no merging, no reading anything. Then the memtable is emptied and the relevant commit log segments become eligible for recycling.

A single row can exist in the memtable and in several SSTables at once, with different columns written at different times. Nothing reconciles them at write time. That reconciliation happens on every read.

SSTables

A Sorted String Table is an immutable file of rows, sorted by partition token then by clustering columns. Once written it is never modified — only read, and eventually deleted by compaction.

Immutability is what makes everything else safe: no locking, no torn writes, no coordination between readers and writers, and backups can be taken by hard-linking files.

An SSTable is not one file. It is a set of components sharing a generation identifier:

   /var/lib/cassandra/data/my_keyspace/users-a1b2c3.../
   ├── nb-1-big-Data.db              ← the actual rows
   ├── nb-1-big-Index.db             ← partition key → offset in Data.db
   ├── nb-1-big-Summary.db           ← sampled index, held in memory
   ├── nb-1-big-Filter.db            ← bloom filter, held in memory
   ├── nb-1-big-CompressionInfo.db   ← compressed chunk offsets
   ├── nb-1-big-Statistics.db        ← metadata: timestamps, TTLs, histograms
   ├── nb-1-big-Digest.crc32         ← checksum
   └── nb-1-big-TOC.txt              ← list of components

Data.db

The rows themselves. Partitions in token order; within a partition, rows in clustering order; within a row, the cells.

Every cell — one column of one row — carries its own metadata:

   ┌──────────────────────────────────────────────────┐
   │ column name (or an id from the schema)           │
   │ flags       is it deleted? does it have a TTL?   │
   │ timestamp   MICROSECONDS since epoch  ◀── the    │
   │             this is the conflict resolver        │
   │ ttl + local deletion time  (if it has a TTL)     │
   │ value                                            │
   └──────────────────────────────────────────────────┘

That per-cell timestamp is the single most important field in Cassandra. There is no version chain, no transaction ID, no locking. When two replicas disagree about a value, the one with the higher timestamp wins. That is the entire conflict-resolution algorithm, and chapter 7 is about its consequences.

Modern versions pack this efficiently — shared row-level timestamps, delta encoding, column subsets described once per partition — so the overhead is far lower than the diagram suggests. But conceptually, every cell knows when it was written.

Filter.db — the bloom filter

The read path’s first line of defence. A bloom filter is a probabilistic structure that answers one question about a set very cheaply: is this key definitely not here?

   bloom filter answers:
     "definitely NOT in this SSTable"   ← always correct. Skip the file.
     "probably in this SSTable"         ← might be wrong (false positive)

   It never says "definitely not" about a key that IS present.
   So it never causes a missed read — only occasional wasted lookups.

Each SSTable has one, held in memory. Without them, a read would have to check the index of every SSTable on the node. With them, most files are eliminated with a handful of bit tests.

ALTER TABLE users WITH bloom_filter_fp_chance = 0.01;

The default is 0.01 (1% false positives) for most compaction strategies and 0.1 for levelled compaction — LCS guarantees at most one SSTable per level holds a given key, so it needs less help. Lowering the value means fewer wasted lookups and more memory; raising it does the opposite.

nodetool tablestats my_keyspace.users | grep -i bloom
#   Bloom filter false positives: 4213
#   Bloom filter false ratio: 0.00312
#   Bloom filter space used: 4.21 MiB

A false ratio well above your configured fp_chance usually means the filter has been sized for fewer keys than the table actually holds.

Summary.db and Index.db

If the bloom filter says “probably here,” Cassandra needs the byte offset of that partition inside Data.db.

Index.db maps every partition key to its offset, but it is too large to hold in memory. So Summary.db samples it — by default every 128th entry — and that is held in memory.

   Summary.db (in RAM)          Index.db (on disk)         Data.db
   ┌──────────────────┐         ┌──────────────────┐      ┌──────────┐
   │ key_0    → 0     │────────▶│ key_0    → 41200 │─────▶│ partition│
   │ key_128  → 4096  │         │ key_1    → 41890 │      │ bytes    │
   │ key_256  → 8192  │         │ ...              │      │          │
   └──────────────────┘         │ key_128  → 98430 │      └──────────┘
                                └──────────────────┘
   binary search in RAM         one disk seek, then       one disk read
   narrows to a 128-key         scan forward
   window

Two lookups, one of them in memory. This is the structure the BTI format replaces with a trie, which finds the offset directly rather than in two hops.

CompressionInfo.db

SSTables are compressed by default, in fixed-size chunks.

ALTER TABLE users WITH compression = {
  'class': 'LZ4Compressor',
  'chunk_length_in_kb': 16
};

CompressionInfo.db maps logical offsets to compressed chunk boundaries, so Cassandra can decompress just the chunk it needs.

The chunk length is a real trade-off that people leave at the default without thinking:

Large chunks (64 KB) compress better — more context for the algorithm — but reading one small row means decompressing 64 KB.
Small chunks (4–16 KB) waste less work per read at the cost of a worse ratio and a bigger offset map.

The default is 16 KB in Cassandra 4.0+ (it was 64 KB before). For a table of small rows read one at a time, 4 KB can measurably cut read latency and I/O. For an analytics table scanned in bulk, 64 KB compresses better.

LZ4Compressor is the right default — very fast, decent ratio. ZstdCompressor compresses substantially better at more CPU cost and is worth considering for cold or archival tables. DeflateCompressor is slow and rarely the right answer.

Statistics.db

Per-SSTable metadata: minimum and maximum timestamps, minimum and maximum clustering values, estimated row counts, tombstone histograms, TTL information.

This is quietly load-bearing. The min/max timestamp lets a read skip an entire SSTable if it cannot possibly contain a relevant version. Time-window compaction (chapter 8) depends on it. And the tombstone histograms drive compaction’s decisions about when a file is worth rewriting.

Putting it together: the shape of a table on disk

   users (one table, one node)

   MEMORY                              DISK
   ┌────────────────────┐              ┌──────────────────────────────┐
   │ memtable           │              │ SSTable 1  (oldest)          │
   │  alice → {age:30}  │              │  alice → {name:'Alice'}      │
   │  carol → {age:25}  │              │  bob   → {name:'Bob'}        │
   └────────────────────┘              ├──────────────────────────────┤
   ┌────────────────────┐              │ SSTable 2                    │
   │ bloom filters      │              │  alice → {email:'a@b.c'}     │
   │ index summaries    │              ├──────────────────────────────┤
   │ key cache          │              │ SSTable 3  (newest)          │
   └────────────────────┘              │  alice → {age:29}  ← older   │
                                       │                     than the │
                                       │                     memtable │
                                       └──────────────────────────────┘

   The full picture of alice = merge memtable + SSTable 1 + 2 + 3,
   taking the highest timestamp for each column:
     name  from SSTable 1
     email from SSTable 2
     age   from the memtable (newer than SSTable 3's copy)

No single file holds the whole truth about alice. That is the LSM tree in one image, and it is why chapter 6 is the hard chapter.

Caches

Three caches sit in front of this, and their defaults are chosen carefully.

Key cache — maps a partition key directly to its offset in an SSTable, skipping the summary and index lookups. Enabled by default, sized at 5% of heap up to 100 MB. Cheap and effective; leave it on and consider raising it.

Chunk cache (file_cache_size_in_mb) — decompressed SSTable chunks, held off-heap. This is the main read cache and does most of the work.

Row cache — caches entire partitions. Disabled by default, and that is usually correct. It caches the whole partition, so a single wide partition can evict everything else, and any write to a partition invalidates its entire cache entry. It is a narrow tool for small, hot, read-mostly, rarely-written partitions. Most people who enable it make things worse.

nodetool info | grep -i cache

Inspecting real files

# Dump an SSTable as JSON — the single best way to believe this chapter
sstabledump /var/lib/cassandra/data/ks/tbl-*/nb-1-big-Data.db | head -50

# Metadata: timestamps, tombstone ratios, level, repaired status
sstablemetadata /var/lib/cassandra/data/ks/tbl-*/nb-1-big-Data.db

# Which SSTables would a read of this key touch?
nodetool getsstables my_keyspace users 'alice'

# Per-table state
nodetool tablestats my_keyspace.users

sstabledump on a table you have just written to, deleted from, and updated is genuinely illuminating. You will see the tombstones, the per-cell timestamps, and the fact that your UPDATE created a new cell rather than changing an old one.

What to take from this chapter

Nothing on disk is ever modified. Writes append; SSTables are immutable.
Commit log for crash recovery only, memtable in RAM sorted, SSTables on disk sorted and immutable.
The default commitlog_sync: periodic can lose up to 10 seconds of writes on one node in a power loss. Replication, not fsync, is Cassandra’s durability story.
Every cell carries a microsecond timestamp. Highest timestamp wins. That is the whole conflict-resolution model.
Bloom filters eliminate most SSTables cheaply; summary and index narrow to a byte offset; compression works in chunks.
A row’s data can be spread across the memtable and many SSTables, and every read must merge them.
Row cache is off by default and should usually stay off.

Next: the data model — because in Cassandra, how you lay out your partitions decides everything.

Written by Sepuri Sai Krishna