9. Consistency levels

This is where Cassandra’s central promise lives: consistency is a per-query decision, not a property of the database. You choose, on each statement, how many replicas must respond — trading latency and availability for how fresh an answer you get.

The arithmetic

One inequality governs everything:

              R + W > RF

   RF = replication factor      (how many copies exist)
   W  = replicas that must ack a write
   R  = replicas that must respond to a read

   If R + W > RF, the read set and the write set must OVERLAP
   by at least one replica. That replica has the latest write.
   Since Cassandra takes the highest timestamp, the read sees it.

Why the overlap works:

   RF = 3.  Write at W=2 (QUORUM), read at R=2 (QUORUM).
   2 + 2 = 4 > 3.  ✓

   ┌───┐ ┌───┐ ┌───┐
   │ A │ │ B │ │ C │
   └───┘ └───┘ └───┘
     ▲     ▲
     └──write acked here──┘
           ▲     ▲
           └──read from here──┘
             │
             └── B is in both sets. B has the new value.
                 Highest timestamp wins. The read is correct.

There is no arrangement of two nodes out of three that misses the overlap. That is the whole guarantee.

Worked combinations, RF = 3

W	R	Sum	Overlap?	What you get
ONE (1)	ONE (1)	2	no	fastest; reads may be stale
ONE (1)	ALL (3)	4	yes	fast writes, slow reads
QUORUM (2)	QUORUM (2)	4	yes	the standard choice
ALL (3)	ONE (1)	4	yes	slow writes, fast reads
ALL (3)	ALL (3)	6	yes	slowest, least available

The middle three all give you consistency. They differ in where you pay for it.

• Write-heavy system? Write at ONE, read at ALL. Cheap writes.
• Read-heavy system? Write at ALL, read at ONE. Cheap reads.
• Balanced? QUORUM both ways.

QUORUM/QUORUM is the usual answer because it is the only combination that tolerates the loss of a node for both reads and writes. ALL on either side means one dead replica breaks that operation entirely.

Consistency here means one thing only

R + W > RF guarantees you read the most recent successful write of that cell. It does not give you transactions, isolation, or atomicity across rows.

Two clients writing the same cell concurrently still resolve by timestamp, not by ordering. A read during a write may see the old value or the new one. Multi-row operations are not atomic. If you need compare-and-set, that is chapter 11.

Every level

Write levels

Level	Waits for	Notes
ANY	a hint being stored, if no replica is up	avoid — a write can “succeed” with zero replicas holding it
ONE	1 replica	fast
TWO / THREE	2 / 3 replicas	explicit counts, rarely used
LOCAL_ONE	1 replica in the local DC	never crosses the WAN
QUORUM	(RF/2)+1 across all DCs	see the multi-DC trap below
LOCAL_QUORUM	(local RF/2)+1 in the local DC	the multi-DC default
EACH_QUORUM	a quorum in every DC	strong, slow, and fragile
ALL	every replica	one node down = writes fail

Read levels

Same list, minus ANY and EACH_QUORUM (which is write-only), plus:

Level	Meaning
SERIAL	read the latest value including any in-flight Paxos state (chapter 11)
LOCAL_SERIAL	the same, confined to the local data centre

What “quorum” means

   quorum = floor(RF / 2) + 1

   RF=1 → 1      RF=2 → 2      RF=3 → 2
   RF=4 → 3      RF=5 → 3      RF=6 → 4

Notice RF=2 has a quorum of 2, which is all of them. RF=2 gives you no fault tolerance at quorum whatsoever — one node down and quorum operations fail. RF=3 is the minimum sensible replication factor, and RF=2 is a trap that looks like it saves a third of your disk.

The multi-data-centre trap

This is the most expensive misunderstanding in Cassandra, and it is easy to make.

CREATE KEYSPACE app WITH replication = {
  'class': 'NetworkTopologyStrategy',
  'dc_east': 3,
  'dc_west': 3
};

Total RF is 6. So:

   QUORUM       = floor(6/2)+1 = 4 replicas

   Your application is in dc_east, which holds only 3 replicas.
   A QUORUM of 4 CANNOT be satisfied without reaching dc_west.

   Every single read and write now crosses the ocean.
   Latency goes from 2 ms to 150 ms.
   And if the WAN link drops, both data centres stop working.

You have thrown away the entire point of a multi-region deployment.

   LOCAL_QUORUM = floor(3/2)+1 = 2 replicas, IN THE LOCAL DC ONLY

   dc_east requests are satisfied by dc_east nodes.
   Latency stays at 2 ms.
   A WAN outage does not stop either side.
   Cross-DC replication continues asynchronously in the background.

In any multi-data-centre cluster, use LOCAL_QUORUM and LOCAL_ONE. Never bare QUORUM or ONE.

ONE is dangerous for a subtler reason: it means “any one replica, anywhere,” so the driver may happily satisfy it from the far data centre. LOCAL_ONE pins it locally.

Does LOCAL_QUORUM still give consistency?

Within the local data centre, yes: LOCAL_QUORUM + LOCAL_QUORUM gives 2 + 2 > 3 among the three local replicas.

Across data centres, no. A write acknowledged by a LOCAL_QUORUM in dc_east replicates to dc_west asynchronously. A read in dc_west immediately afterwards may return the old value. Convergence typically takes milliseconds to a second or two, but it is not guaranteed.

This is almost always the right trade, and it is the standard architecture for global deployments. But your application must tolerate it — a user who writes in one region and reads in another may briefly see stale data.

If you genuinely cannot tolerate that, EACH_QUORUM writes plus LOCAL_QUORUM reads give cross-DC consistency. The cost is that a write fails if any data centre is unreachable, which reintroduces exactly the fragility you built multiple data centres to avoid. Think hard before choosing it.

Availability: how many nodes can you lose?

   RF=3, CL=QUORUM (2 of 3)
     lose 1 replica  →  still fine
     lose 2 replicas →  operations on those ranges fail

   RF=3, CL=ONE
     lose 2 replicas →  still fine
     lose 3 replicas →  those ranges are unavailable

   RF=3, CL=ALL
     lose 1 replica  →  operations on those ranges fail

With vnodes, “losing a replica” is per token range rather than per whole node — a given failure affects the ranges that node participated in, not the entire keyspace. But the arithmetic per range is exactly as above.

   dc_east: 3, dc_west: 3   with LOCAL_QUORUM in dc_east
     lose 1 node in dc_east      → fine
     lose 2 nodes in dc_east     → dc_east requests fail for those ranges
     lose ALL of dc_west         → dc_east completely unaffected  ✓

That last line is the payoff. A whole-region outage is invisible to the other region.

Setting it

-- cqlsh, for the session
CONSISTENCY LOCAL_QUORUM;

# Per statement — the right way
from cassandra import ConsistencyLevel
from cassandra.query import SimpleStatement

stmt = session.prepare("SELECT * FROM users WHERE id = ?")
stmt.consistency_level = ConsistencyLevel.LOCAL_QUORUM
session.execute(stmt, (user_id,))

SimpleStatement.newInstance("SELECT ...")
    .setConsistencyLevel(DefaultConsistencyLevel.LOCAL_QUORUM);

Set a sensible default on the session, then override per query. Not every query needs the same level, and that is the point:

   billing ledger write     LOCAL_QUORUM   correctness matters
   read a user's profile    LOCAL_QUORUM   they just edited it
   count page views         LOCAL_ONE      stale is fine
   analytics batch read     LOCAL_ONE      stale is fine
   write an audit record    LOCAL_QUORUM   must not be lost

The failure exceptions

   UnavailableException
     The coordinator knows, before trying, that not enough replicas
     are alive. Nothing was attempted. The write definitely did not
     happen. Safe to retry.

   WriteTimeoutException
     The request was sent but not enough acks arrived in time.
     ⚠ THE WRITE MAY HAVE BEEN APPLIED on some replicas.
     Safe to retry ONLY if the write is idempotent.

   ReadTimeoutException
     Not enough replicas responded to a read in time.
     Always safe to retry.

The distinction between Unavailable and WriteTimeout matters and is worth encoding in your retry logic. Unavailable means “definitely not applied.” WriteTimeout means “unknown.”

Because a timed-out write may have partially applied — and because there is no rollback — idempotent writes are close to mandatory in Cassandra. Writing the same cells with the same values is naturally idempotent. Incrementing a counter is not. IF NOT EXISTS is not, in the sense that a retry may report failure for a write that actually succeeded.

The drivers ship reasonable retry policies. Understand yours before relying on it, and never blindly retry a non-idempotent statement.

from cassandra.policies import ExponentialReconnectionPolicy
cluster = Cluster(
    reconnection_policy=ExponentialReconnectionPolicy(1.0, 60.0),
)

Common mistakes

Bare QUORUM in a multi-DC cluster. The big one. Covered above.

CL=ONE and then wondering about stale reads. With ONE/ONE, 1 + 1 is not greater than 3. You will read stale data. That is the deal you made.

CL=ALL to “be safe.” It makes you strictly less available than QUORUM while giving no additional consistency guarantee that matters. One slow node sets your latency; one dead node breaks the operation.

RF=2. Quorum is 2, so quorum operations have zero fault tolerance. Use 3.

CL=ANY for writes. The write can be satisfied by storing a hint with no replica actually holding the data. If the coordinator then dies, the write is gone — and it was reported as successful.

Assuming a write timeout means the write failed. It does not.

Mixing levels carelessly. Writing at LOCAL_ONE and reading at LOCAL_QUORUM gives 1 + 2 = 3, which is not greater than 3. No overlap guarantee.

Choosing, in practice

For most production systems:

   Replication:  NetworkTopologyStrategy, RF=3 per data centre
   Writes:       LOCAL_QUORUM
   Reads:        LOCAL_QUORUM
   Tolerates:    one node down per DC, and an entire DC down

Then relax specific queries to LOCAL_ONE where staleness is genuinely acceptable, and reach for SERIAL only where you truly need compare-and-set.

The fact that consistency is a per-query choice here is the whole design, and it is worth contrasting with a system that made the opposite decision. etcd has no tunable consistency: a write is committed by a majority or it fails, and on the minority side of a partition it serves nothing at all rather than serving something stale. That buys guarantees strong enough to build a lock on, and costs availability and scale — a few thousand writes per second, total, for the whole cluster, with more members making it slower. Cassandra chose availability and spends its operational effort on repair afterwards; etcd chose to stop instead, so every one of its failure modes is a variant of “stopped” and none is “gave two different answers”. Neither is wrong. They are answers to different questions, and knowing which question you have is the skill.

What to take from this chapter

• R + W > RF guarantees the read set overlaps the write set, so you see the latest write.
• QUORUM = floor(RF/2) + 1. RF=3 is the minimum sensible value; RF=2 has no fault tolerance at quorum.
• In multi-DC clusters, always LOCAL_QUORUM / LOCAL_ONE. Bare QUORUM with two DCs at RF=3 each means every request crosses the WAN.
• LOCAL_QUORUM gives consistency within a data centre and eventual consistency across them. That is usually the right trade.
• UnavailableException = definitely not applied. WriteTimeoutException = unknown. Make writes idempotent.
• Avoid ANY and ALL. Set the level per query, not once for the application.

Next: the three mechanisms that make “eventual” actually happen.