5. A write, traced end to end

The write path is the easy half, and it is where Cassandra earns its reputation. There is no read, no seek, no lock, and no coordination beyond counting acknowledgements.

Our statement:

INSERT INTO messages (room_id, bucket, sent_at, message_id, sender, body)
VALUES (?, ?, ?, ?, ?, ?);

The whole journey

   client
     │  ① pick a coordinator (token-aware, if the driver is good)
     ▼
   coordinator (node E)
     │  ② hash the partition key → token → replicas B, C, D
     │  ③ send the mutation to all of them, in parallel
     ▼
   ┌────────┐  ┌────────┐  ┌────────┐
   │ node B │  │ node C │  │ node D │
   └───┬────┘  └───┬────┘  └───┬────┘
       │           │           │      each replica, independently:
       │           │           │        ④ append to commit log
       │           │           │        ⑤ insert into memtable
       │           │           │        ⑥ acknowledge
       └───────────┴───────────┘
                   │
                   ▼
   coordinator waits for as many acks as the
   consistency level requires, then replies
                   │
                   ▼
   client                    ⑦ later, asynchronously: memtable → SSTable

① Choosing a coordinator

The client sends the statement to one node, which becomes the coordinator for this request. Any node will do.

A token-aware driver (chapter 2) hashes the partition key itself and sends the request directly to one of the replicas, saving a network hop. This requires a prepared statement, because the driver needs to know which bound parameter is the partition key:

stmt = session.prepare(
    "INSERT INTO messages (room_id, bucket, sent_at, message_id, sender, body) "
    "VALUES (?, ?, ?, ?, ?, ?)")
session.execute(stmt, (room_id, bucket, sent_at, msg_id, sender, body))

Prepared statements also mean the CQL is parsed once per node rather than on every request, and they make injection structurally impossible. Use them for everything. There is no case where a non-prepared statement in application code is the right choice.

② Finding the replicas

The coordinator hashes the partition key with Murmur3 to get a token, walks the ring clockwise, and collects the RF nodes responsible — plus, if NetworkTopologyStrategy is in use, the replicas in every other data centre.

This is pure local computation against gossip state. No lookup service, no metadata round trip.

nodetool getendpoints my_keyspace messages 'room-42:2026-08-20'
# 10.0.1.11
# 10.0.1.12
# 10.0.2.10

③ Sending to replicas

The coordinator sends the mutation to all live replicas in parallel — not just enough to satisfy the consistency level. Consistency level controls how many acknowledgements it waits for, not how many nodes it writes to.

This distinction matters. With RF=3 and CL=ONE, all three replicas receive the write; the coordinator simply replies to the client after the first acknowledgement. The other two are still being written, and almost always succeed.

Replicas that are known to be down do not get the mutation. Instead the coordinator stores a hint — a note saying “node C missed this, deliver it when C returns” (chapter 10).

④ The commit log append

Now we are on a replica. It does two things, and neither involves reading anything.

First, append the mutation to the commit log. Sequential write to an open file. With the default commitlog_sync: periodic, this lands in the OS page cache and is fsynced within 10 seconds; with batch or group, the replica waits for the fsync before acknowledging.

The commit log is shared by all tables on the node — one log, not one per table — which keeps it purely sequential no matter how many tables are being written.

⑤ The memtable insert

Second, insert into the table’s memtable: an in-memory sorted structure (a trie, in Cassandra 5.0), keyed by partition token then clustering columns.

That is the entire write. No read of existing data, no check for whether the row already exists, no index maintenance, no lock.

This is the crucial difference from a B-tree engine. In PostgreSQL, an UPDATE must find the existing row, mark it dead, write a new version, and update every index. In Cassandra, an update is just a write of the new cells with a new timestamp. Whether a previous version exists is not Cassandra’s problem right now — it becomes the read path’s problem later, and compaction’s problem eventually.

Insert and update are the same operation

INSERT INTO users (id, name) VALUES (1, 'Alice');
UPDATE users SET name = 'Alice' WHERE id = 1;

These produce identical writes. Both are “upserts.” There is no existence check, so INSERT never fails because the row is already there, and UPDATE never fails because it is not.

The exception is IF NOT EXISTS / IF conditions, which do require reading and coordinating, and cost roughly an order of magnitude more (chapter 11).

⑥ Acknowledging

The replica replies to the coordinator. Once enough replicas have replied, the coordinator replies to the client.

“Enough” is the consistency level:

   RF = 3, CL = ONE       ──▶ reply after 1 ack     (fastest)
   RF = 3, CL = QUORUM    ──▶ reply after 2 acks
   RF = 3, CL = ALL       ──▶ reply after 3 acks    (slowest, least available)
   RF = 3, CL = ANY       ──▶ reply after a HINT is stored, even if no
                              replica took the write  (avoid this)

Chapter 9 covers consistency levels properly. The thing to notice here is that the write went to all three replicas regardless. CL is about how long you wait, not about how many copies you get.

Writes do not fail cleanly

An important and often-misunderstood property: if the coordinator returns a timeout, the write may still have been applied.

Suppose CL=QUORUM with RF=3, and only one replica acknowledges before the timeout. The client gets WriteTimeoutException. But that one replica did apply the write, and it will spread to the others through hints and repair. The write is not rolled back — there is no rollback.

This is why idempotent writes matter so much in Cassandra. A retry after a timeout must be safe. Writing the same cells with the same values is naturally idempotent; incrementing a counter is not. The rule that follows: prefer immutable, fully-specified rows over read-modify-write, and be careful with counters.

⑦ The flush

Eventually the memtable is full and is written out as a new SSTable.

Triggers:

• The memtable’s share of memtable_heap_space / memtable_offheap_space crosses memtable_cleanup_threshold.
• Commit log space exceeds commitlog_total_space, forcing the oldest segments to be freed.
• memtable_flush_period_in_ms elapses (0 = disabled, the default).
• nodetool flush or nodetool drain.

The flush writes the memtable’s contents in sorted order — sequentially, in one pass — producing Data.db plus its bloom filter, index, summary, compression and statistics components (chapter 3). No merging, no reading of existing SSTables.

Once every memtable containing data from a commit log segment has been flushed, that segment is recycled.

nodetool tpstats | grep -i flush
#   MemtableFlushWriter   0   0   4821   0   0

nodetool flush my_keyspace my_table    # force it

If flushes cannot keep up, memtables back up, then writes back up. MemtableFlushWriter pending counts above zero for sustained periods point at memtable_flush_writers being too low or disks being too slow.

What an UPDATE really writes

This is worth making concrete, because it explains chapters 6, 7 and 8 in one image.

INSERT INTO users (id, name, email) VALUES (1, 'Alice', 'a@old.com');
-- ... later ...
UPDATE users SET email = 'a@new.com' WHERE id = 1;

   SSTable 1 (from the first flush)
   ┌───────────────────────────────────────────────────┐
   │ id=1  name='Alice'      ts=1000                   │
   │       email='a@old.com' ts=1000                   │
   └───────────────────────────────────────────────────┘

   SSTable 2 (from a later flush)
   ┌───────────────────────────────────────────────────┐
   │ id=1  email='a@new.com' ts=2000                   │
   └───────────────────────────────────────────────────┘
       ▲
       only the changed column. Nothing else was written,
       and SSTable 1 was not touched.

The UPDATE wrote exactly one cell. That is why writes are cheap.

The read now has to merge both files and pick email by highest timestamp. That is why reads are not.

Partial writes are normal

Because every cell is written independently with its own timestamp, a row can be a genuine mixture of different writes:

   name  written at ts=1000, still the latest
   email written at ts=2000
   phone written at ts=1500

There is no notion of “the row as of a point in time.” A SELECT reconstructs the row cell by cell, each from whichever write had the highest timestamp. This is exactly what you want when two data centres write concurrently to different columns — both changes survive. It is confusing if you expect row-level versioning.

Write performance in practice

What makes writes fast: sequential I/O only, no read-before-write, no locks, no index maintenance, and all replicas written in parallel.

What slows them down, in rough order of how often it is the cause:

Large partitions. Not directly a write cost, but they make compaction expensive and compaction competes with writes for I/O.

Too many tables. Each table has its own memtable, so memory is divided. A cluster with 2,000 tables flushes tiny memtables constantly, producing tiny SSTables and endless compaction. Keep it in the low hundreds.

Compaction falling behind. Writes create SSTables faster than compaction merges them. Check nodetool compactionstats — a growing pending count is the warning sign, and chapter 8 is the response.

Garbage collection. Memtables live on or off the heap; large heaps with the wrong collector produce pauses that show up as write latency spikes. Chapter 13.

Counters and lightweight transactions. Both require a read and, for LWT, several coordination round trips. They are not on this fast path at all.

Wide rows with many collection elements. Each element is a cell with its own overhead.

# Where is write latency actually going?
nodetool tablehistograms my_keyspace.messages
nodetool proxyhistograms          # coordinator-level, includes network
nodetool tpstats                  # thread pool backlogs and dropped messages

nodetool tpstats showing dropped MUTATION messages means replicas could not keep up and silently discarded writes — the coordinator may still have satisfied the consistency level from other replicas, and repair will eventually fix it, but it is a clear signal the cluster is overloaded.

What to take from this chapter

• Coordinator hashes the key, finds replicas, sends to all of them; CL decides how many acks it waits for.
• Each replica: append to the commit log, insert into the memtable. No read, no lock, no index update.
• INSERT and UPDATE are the same upsert. Neither checks whether the row exists.
• An UPDATE writes only the changed cells, each with its own timestamp. Older cells stay in older SSTables.
• A write timeout does not mean the write did not happen. Make your writes idempotent.
• Flushing turns a memtable into one new SSTable, sequentially. Commit log segments are recycled after the data is safe.
• Slow writes usually mean compaction is behind, too many tables, or GC — rarely the write path itself.

Next: the read path, where all of this cheapness gets paid for.