6. A read, traced end to end

Writes were cheap because they never looked at existing data. The read path is where that bill comes due: a row’s truth is scattered across the memtable and an unknown number of immutable files, and the read has to find all of it and merge it.

Our query:

SELECT sender, body FROM messages
WHERE room_id = ? AND bucket = ? AND sent_at > ?
LIMIT 50;

The whole journey

   client
     │
     ▼
   coordinator
     │  ① find replicas for the partition
     │  ② send a full read to the closest, digest reads to the others
     ▼
   ┌────────────────────────────────────────────────────────────┐
   │  on each replica:                                          │
   │    ③ check the row cache          (usually off)            │
   │    ④ read the memtable                                     │
   │    ⑤ for each SSTable:                                     │
   │         a. min/max timestamp and clustering — can we skip? │
   │         b. bloom filter — definitely not here?             │
   │         c. key cache — do we already know the offset?      │
   │         d. summary → index → offset                        │
   │         e. read and decompress the chunk                   │
   │    ⑥ merge everything by timestamp, apply tombstones       │
   └────────────────────────────────────────────────────────────┘
     │
     ▼
   coordinator
     │  ⑦ compare digests; if they disagree, read repair
     │  ⑧ return the merged result
     ▼
   client

① and ② The coordinator’s part

The coordinator computes the token, finds the replicas, and — this is the clever bit — does not ask all of them for the full data.

It asks the closest replica (by the dynamic snitch’s latency measurements) for the actual rows, and the others for a digest: a hash of the data they would have returned. Digests are tiny, so the network cost of a QUORUM read is roughly one full response plus some hashes rather than three full responses.

If all digests match the full response, the coordinator returns it. If any digest differs, the replicas disagree, and the coordinator issues a second round of full reads to reconcile — this is blocking read repair (chapter 10).

Speculative retry covers the case where one replica is slow rather than down:

ALTER TABLE messages WITH speculative_retry = '99p';

If the chosen replica has not responded by the 99th-percentile latency for this table, the coordinator quietly sends the request to another replica and takes whichever answers first. This turns a single slow node from a source of tail latency into a non-event, and 99p is a good default. ALWAYS sends to an extra replica every time — more load, lower tail latency. NONE disables it.

③ The row cache

Off by default, and usually correctly so. It caches whole partitions; a write to any part of a partition invalidates the entire entry, and one wide partition can evict everything else. Skip it unless you have small, hot, read-heavy, rarely-written partitions and you have measured a benefit.

④ The memtable

Read the in-memory sorted structure for any cells in this partition. Fast, and it holds the newest data.

⑤ The SSTables — the interesting part

Now the replica must consider every SSTable that could contain part of this partition. Four filters run before any real work happens.

a. Metadata pruning

Each SSTable’s Statistics.db records its minimum and maximum write timestamps, and its minimum and maximum clustering values. If the query asks for sent_at > '2026-08-01' and an SSTable’s maximum clustering value is 2026-07-15, that file is skipped without a single byte being read.

This is why time-window compaction (chapter 8) is so effective for time-series: it deliberately groups data by time window so that this check eliminates almost everything.

b. The bloom filter

For each surviving SSTable, test the partition key against its bloom filter, held in memory.

   "definitely not here"  → skip the file entirely. Always correct.
   "probably here"        → continue. Might be a false positive.

This is the workhorse. On a node with 40 SSTables for a table, the bloom filters typically eliminate all but one or two in microseconds.

nodetool tablestats ks.messages | grep -i bloom
#   Bloom filter false positives: 1204
#   Bloom filter false ratio: 0.00081

A false ratio much higher than the configured bloom_filter_fp_chance means wasted lookups. It usually means the filters were sized for fewer keys than the table now holds; a compaction rewrites them.

c. The key cache

Maps (SSTable, partition key) straight to a byte offset, skipping the summary and index. On by default, small, and very effective on hot keys.

nodetool info | grep -i "key cache"
# Key Cache  : entries 98234, size 24.1 MiB, capacity 100 MiB,
#              1204321 hits, 1301442 requests, 0.926 recent hit rate

A hit rate below about 0.9 on a read-heavy workload is a reason to raise key_cache_size_in_mb.

d. Summary, then index, then data

On a key cache miss: binary-search the in-memory Summary.db to find a 128-key window, seek into Index.db and scan forward to the exact entry, then read the offset it gives you and decompress the relevant chunk of Data.db.

That is one memory lookup, one disk seek, one disk read. In Cassandra 5.0’s BTI format, the trie in Partitions.db collapses the first two into a single traversal.

⑥ The merge

Now the replica has candidate cells for this partition from the memtable and from n SSTables. It merges them.

The rule is the same rule as everywhere else in Cassandra: for each cell, the version with the highest timestamp wins.

   memtable      : (alice, email, 'a@new.com', ts=3000)
   SSTable 3     : (alice, age,   29,          ts=2500)
   SSTable 2     : (alice, email, 'a@mid.com', ts=2000)
   SSTable 1     : (alice, name,  'Alice',     ts=1000)
                   (alice, email, 'a@old.com', ts=1000)
                   (alice, age,   30,          ts=1000)

   merged result:
     name  = 'Alice'        (only one version)
     email = 'a@new.com'    (ts=3000 beats 2000 and 1000)
     age   = 29             (ts=2500 beats 1000)

Because SSTables are sorted, this is a merge of sorted streams, not a random-access nightmare. But every additional SSTable is another stream to open, seek into, and read from.

Tombstones participate in the merge as deletion markers with their own timestamps. A tombstone at ts=4000 suppresses every cell for that column with a lower timestamp. This is chapter 7, and it is where reads go wrong most dramatically.

The number that predicts your read latency

Everything above reduces to one metric: how many SSTables does a read have to touch?

nodetool tablehistograms my_keyspace.messages
# Percentile  SSTables  Write Latency  Read Latency  Partition Size  Cell Count
# 50%             1.00          35.43        124.00            1109          50
# 95%             3.00          51.01        545.79          126934        2299
# 99%             8.00          73.46       2346.80         3379391       73457
# Max            24.00         182.79      20924.30       186563160     1629722

Read that table as a diagnosis:

50% at 1 SSTable — most reads hit a single file. Healthy.
99% at 8 — the slow tail is reading eight files and merging them.
Max at 24 — some reads touch 24 files. Those are your worst latencies, and the correlation with Read Latency in the same row is not a coincidence.

Rules of thumb: a p99 of 1–3 SSTables per read is good. Above about 10, compaction is not keeping up or the compaction strategy is wrong for the workload. Chapter 8 is the fix.

nodetool proxyhistograms       # coordinator view, includes network + CL waiting
nodetool tablestats ks.tbl | grep -Ei "sstable count|read latency|tombstone"

Range queries and paging

Our query used sent_at > ?, a range within one partition. That is the good case: the rows are physically adjacent in clustering order, so it is a sequential scan of a contiguous byte range.

Cross-partition queries are a different animal.

SELECT * FROM messages;            -- every partition, every node

The coordinator walks token ranges across the whole cluster. Cassandra pages this automatically — the driver’s fetch_size (default 5,000) controls how many rows come back per round trip, with a paging state token to resume. So it will not blow up memory, but it is a full cluster scan and its cost scales with your data, not your result.

For genuine bulk export, drive it by token range so each query hits one node’s data:

SELECT * FROM messages
WHERE token(room_id, bucket) > ? AND token(room_id, bucket) <= ?;

Get the ranges from nodetool describering (or use dsbulk, or Spark with the Cassandra connector, both of which do exactly this for you and parallelise it).

Never use IN with a large list of partition keys

SELECT * FROM users WHERE id IN (?, ?, ?, ... 500 more);

One coordinator becomes responsible for 500 partitions scattered across the cluster. It holds all the state, waits for the slowest, and if it dies you lose the whole thing.

Issue 500 separate async queries instead. The driver spreads them across coordinators, each is token-routed to its own replicas, and a slow one does not block the rest. This is consistently faster and far more resilient. The same argument applies to unlogged batches (chapter 11).

The two reads that are genuinely expensive

Reading the start of a partition full of tombstones. Covered properly in chapter 7. The short version: deleted rows are still read and merged before being discarded. A partition with 100,000 tombstones and 10 live rows costs you 100,010 rows of work.

WARN  Read 10 live rows and 98234 tombstone cells for query
      SELECT * FROM ks.queue WHERE id = ? LIMIT 100

That warning in your log is a data model problem, not a tuning problem.

Reading a large partition. A multi-hundred-megabyte partition must be traversed, and depending on the query, substantial parts of it held in heap. This is where garbage collection pauses and OutOfMemoryError come from.

Consistency level and read latency

The consistency level determines how many replicas must respond:

   CL = ONE           1 replica          fastest, may be stale
   CL = LOCAL_ONE     1 in local DC      fastest cross-DC-safe option
   CL = QUORUM        2 of 3             may cross data centres — beware
   CL = LOCAL_QUORUM  2 of 3 in local DC the usual production choice
   CL = ALL           all 3              slowest; one slow node stalls it

Your latency is the latency of the slowest replica you wait for. QUORUM with RF=3 waits for the second-fastest of three. ALL waits for the slowest of three, every time, which is why it has such bad tail behaviour.

QUORUM in a multi-data-centre cluster is a common and painful mistake. With 'dc_east': 3, 'dc_west': 3, RF is 6 and a global quorum is 4 — which cannot be satisfied without crossing the ocean. You wanted LOCAL_QUORUM. Chapter 9 does the arithmetic.

Tracing: seeing all of this for real

TRACING ON;
SELECT * FROM messages WHERE room_id = ? AND bucket = ?;

The output is the read path, annotated with microsecond timings:

 activity                                              | source    | elapsed
-------------------------------------------------------+-----------+---------
 Execute CQL3 query                                    | 10.0.1.10 |       0
 Parsing SELECT * FROM messages...                     | 10.0.1.10 |     123
 Preparing statement                                   | 10.0.1.10 |     245
 Sending READ message to /10.0.1.11                    | 10.0.1.10 |     512
 Sending READ_DIGEST message to /10.0.1.12             | 10.0.1.10 |     534
 Executing single-partition query on messages          | 10.0.1.11 |    1204
 Acquiring sstable references                          | 10.0.1.11 |    1289
 Bloom filter allows skipping sstable 41               | 10.0.1.11 |    1301
 Bloom filter allows skipping sstable 39               | 10.0.1.11 |    1308
 Key cache hit for sstable 42                          | 10.0.1.11 |    1350
 Merged data from memtables and 2 sstables             | 10.0.1.11 |    2104
 Read 47 live rows and 0 tombstone cells               | 10.0.1.11 |    2150
 Digest mismatch: org.apache.cassandra.service.Diges...| 10.0.1.10 |    3011
 Request complete                                      | 10.0.1.10 |    4230

Every concept in this chapter appears in that output. Turn tracing on for one slow query and read it line by line — it is far more informative than any amount of guessing.

nodetool settraceprobability 0.001    # trace 0.1% of queries in production
                                      # (never leave this high — it writes
                                      #  to system_traces on every request)

Making reads faster

In rough order of impact:

Fix the data model. Single-partition reads with the full key. Bounded partitions. Clustering order matching your query’s ORDER BY. Nothing else on this list will rescue a bad model.

Reduce SSTables per read. Choose the right compaction strategy (chapter 8). Check nodetool tablehistograms.

Eliminate tombstones. Chapter 7. Often the single biggest win on a table that has degraded over time.

Use LOCAL_QUORUM or LOCAL_ONE, never plain QUORUM, in a multi-DC cluster.

Turn on speculative_retry = '99p' to hide slow nodes.

Select only the columns you need. SELECT * on a wide table fetches and decompresses everything.

Tune chunk length. chunk_length_in_kb: 4 for small-row point lookups; 64 for bulk scans.

Raise the key cache if its hit rate is below ~0.9.

Use token-aware routing and prepared statements to skip a hop.

What to take from this chapter

The coordinator asks one replica for data and the rest for digests; mismatches trigger read repair.
Each replica checks: metadata, bloom filter, key cache, summary, index, data — then merges everything by timestamp.
SSTables per read is the number that predicts read latency. p99 above ~10 means compaction is not doing its job.
Cross-partition scans are cluster-wide. Page by token range for bulk work; never use large IN lists.
Tombstone-heavy partitions and oversized partitions are the two genuinely expensive reads, and both are data model problems.
Use LOCAL_QUORUM, not QUORUM, across data centres.
TRACING ON shows you the entire path with timings. Use it.

Next: timestamps, last-write-wins, and the deletes that create data.

Written by Sepuri Sai Krishna