12. Multi-datacenter, topology, and backups

Multi-region is Cassandra’s flagship capability and the thing it does better than almost anything else. There is no failover, no promotion, and no split brain — because every data centre is already fully writable.

Snitches: teaching Cassandra your topology

A snitch tells each node which data centre and rack every other node is in. NetworkTopologyStrategy uses this to place replicas (chapter 2), and the dynamic snitch uses latency history to route reads to the fastest replica.

Get it wrong and replicas land in the wrong failure domains — which you discover during an outage.

Snitch	Use
SimpleSnitch	single-DC development only
GossipingPropertyFileSnitch	the right default for almost everyone
Ec2Snitch	AWS, single region: region = DC, AZ = rack
Ec2MultiRegionSnitch	AWS multi-region, using public IPs between regions
GoogleCloudSnitch	GCP equivalent
PropertyFileSnitch	explicit file listing every node — do not use

Use GossipingPropertyFileSnitch (GPFS) unless you have a specific reason not to. Each node declares its own location in a small file, and gossip spreads it:

cassandra-rackdc.properties

dc=dc_east
rack=rack1

cassandra.yaml

endpoint_snitch: GossipingPropertyFileSnitch

GPFS works identically in any environment, including cloud, and it does not require every node to know about every other node in advance (which is PropertyFileSnitch’s fatal flaw — the file must be updated everywhere on every topology change).

Changing the snitch on a live cluster

Changing the snitch changes replica placement, which means data is now on the wrong nodes. Doing this carelessly on a running cluster loses data.

If you must: keep the DC and rack names identical across the change, so placement does not actually move. If names must change, the safe path is to add a new data centre with the new topology, replicate to it, and decommission the old one.

The dynamic snitch

Wrapping whichever snitch you choose, the dynamic snitch tracks recent latency from each replica and steers reads toward the fastest ones, routing around a node that is struggling but not yet marked down.

dynamic_snitch_badness_threshold: 0.1   # prefer a different replica if the
                                        # preferred one is >10% worse

This is on by default and is one reason Cassandra tolerates a degraded node gracefully.

Adding a data centre to a live cluster

The procedure is genuinely online — no downtime, no read impact — but the order matters and one step is easy to get wrong.

   Step 1. Make sure clients use LOCAL_ONE / LOCAL_QUORUM, not
           ONE / QUORUM.

           ⚠ THIS MUST BE DONE FIRST. If clients use bare QUORUM,
           the moment you raise RF the quorum size grows to include
           the new DC — which has no data yet. Reads start failing
           or returning nothing.

   Step 2. Stand up the new nodes with:
             - the same cluster_name
             - dc=dc_west in cassandra-rackdc.properties
             - auto_bootstrap: false      ← important
             - seeds including nodes from BOTH data centres

   Step 3. Start them. They join the ring but stream no data.

   Step 4. Alter the keyspace to replicate into the new DC:
             ALTER KEYSPACE app WITH replication = {
               'class': 'NetworkTopologyStrategy',
               'dc_east': 3,
               'dc_west': 3
             };
           From this moment, NEW writes go to both DCs.
           Existing data does not.

   Step 5. On each new node, stream the historical data:
             nodetool rebuild -- dc_east

   Step 6. Verify with nodetool status, then point some clients
           at the new DC with local_dc=dc_west.

auto_bootstrap: false plus an explicit rebuild is deliberate: it gives you control over when the (heavy) streaming happens, and it lets you rebuild one node at a time rather than having several bootstrap simultaneously.

Active-active

Once two data centres both hold full replicas, both accept writes at local latency.

   ┌────────── dc_east (RF=3) ──────────┐   ┌────── dc_west (RF=3) ──────┐
   │                                    │   │                            │
   │  app servers ──▶ local nodes       │   │  local nodes ◀── app       │
   │                  LOCAL_QUORUM      │   │  LOCAL_QUORUM     servers  │
   │                       │            │   │        │                   │
   └───────────────────────┼────────────┘   └────────┼───────────────────┘
                           │                          │
                           └────── async replication ─┘
                                  (a coordinator forwards to
                                   ONE node in the remote DC,
                                   which fans out locally)

Two properties worth stating precisely:

Cross-DC traffic is efficient. A coordinator does not send the mutation to all three remote replicas. It sends it once to a single node in the remote data centre, which then forwards it locally. One WAN message per DC per write, not RF.

Conflicts resolve by timestamp. Two regions writing the same cell concurrently is not an error — the higher timestamp wins (chapter 7). This is why clock synchronisation across regions is a correctness requirement, not a nicety.

Losing an entire region

nodetool status
# Datacenter: dc_east
# UN  10.0.1.10  ...
# Datacenter: dc_west
# DN  10.1.1.10  ...    ← all down
# DN  10.1.1.11  ...

With LOCAL_QUORUM, dc_east does not notice. No failover, no promotion, no election, no split brain. It keeps serving at full speed with local replicas.

When dc_west returns, hints cover short outages and repair covers the rest. If it was down longer than gc_grace_seconds, rebuild it rather than repairing it (chapter 10).

Compare that to a leader-based system, where losing the region containing the primary means a failover window, a fencing problem, and a real risk of divergence. This is the single strongest argument for Cassandra.

Other topology patterns

An analytics data centre. Add a DC that holds a full replica set but takes no application traffic, and point Spark or your reporting jobs at it. Heavy scans never touch production nodes.

ALTER KEYSPACE app WITH replication = {
  'class': 'NetworkTopologyStrategy',
  'dc_prod': 3,
  'dc_analytics': 2
};

A cheaper standby. A DC at a lower RF as a disaster-recovery target, with less hardware than production.

Migration. Adding a DC and decommissioning the old one is the standard way to move a cluster between regions, cloud providers, or hardware generations — with no downtime.

Backups

Cassandra’s replication protects against node and region loss. It does not protect against TRUNCATE, a bad migration, a bug, or a malicious actor — all of which replicate faithfully to every replica in milliseconds.

Snapshots

A snapshot is a set of hard links to the current SSTable files.

nodetool snapshot -t before_migration my_keyspace
# → data/my_keyspace/table-<uuid>/snapshots/before_migration/

nodetool listsnapshots
nodetool clearsnapshot -t before_migration my_keyspace

Because SSTables are immutable, hard-linking is instant and consumes no extra disk at first. Space is only consumed as compaction deletes the original files — the links keep the old data alive.

This is a real operational trap. Forgotten snapshots silently consume disk, and it looks like unexplained growth. Two defences:

auto_snapshot: true              # snapshot before DROP or TRUNCATE
                                 # (keep this on — it has saved people)
snapshot_before_compaction: false

and a routine that lists and clears old snapshots. Check nodetool listsnapshots when investigating disk usage; it is a common answer.

A snapshot is per node. A cluster-wide backup means snapshotting every node and copying the files off — the snapshot itself is local and dies with the node.

Incremental backups

incremental_backups: true

Every time an SSTable is flushed, a hard link is created in a backups/ directory. Combined with periodic full snapshots, this gives you a much finer restore granularity than snapshots alone.

Cassandra never cleans these up. You must copy them off and delete them yourself, or they will fill the disk.

Commit log archiving

For point-in-time recovery finer than your snapshot interval:

commitlog_archiving.properties

archive_command=/bin/cp %path /mnt/backups/commitlog/%name
restore_command=/bin/cp -f %from %to
restore_directories=/mnt/backups/commitlog
restore_point_in_time=2026:08:20 14:30:00

Rarely used in practice, and worth knowing exists.

Restoring

   1. Stop the node.
   2. Delete the current SSTables for the affected tables.
   3. Copy the snapshot files into the table directory.
   4. Start the node.
   5. nodetool refresh my_keyspace my_table
        ← picks up new SSTable files without a restart
   6. nodetool repair my_keyspace
        ← reconcile with the other replicas

For a full cluster restore where token assignments must be preserved, sstableloader streams SSTables into a cluster and handles redistribution:

sstableloader -d 10.0.1.10 /path/to/snapshot/my_keyspace/my_table

This is the right tool when restoring into a cluster with a different topology.

Tools

Medusa (from the Cassandra Reaper authors) handles cluster-wide backup and restore to S3, GCS, or Azure, with orchestration and verification. Priam is Netflix’s equivalent for AWS. Managed services (Astra, Amazon Keyspaces, DataStax) handle it for you.

Hand-rolled nodetool snapshot plus rsync works and is a reasonable starting point, but it will not orchestrate a cluster-wide restore for you at 3 a.m.

The rules

A backup you have never restored is not a backup. Practise on a separate cluster, with the clock running.

Keep auto_snapshot: true. It is the reason a TRUNCATE typed against the wrong keyspace is recoverable.

Store backups off-cluster. A snapshot living on the node it protects is not a backup.

Watch snapshot disk usage. It is a top cause of unexplained disk growth.

Security, briefly

Two settings that ship insecure and should not stay that way.

authenticator: PasswordAuthenticator      # default is AllowAllAuthenticator
authorizer: CassandraAuthorizer           # default is AllowAllAuthorizer
role_manager: CassandraRoleManager

server_encryption_options:
  internode_encryption: all               # node-to-node TLS
client_encryption_options:
  enabled: true                           # client-to-node TLS

Defaults allow anyone who can reach port 9042 to do anything. The first thing to do on a new cluster is change the default cassandra/cassandra superuser password and raise the system_auth keyspace’s replication factor — if system_auth is at RF=1 and that node is down, nobody can log in.

ALTER KEYSPACE system_auth WITH replication = {
  'class': 'NetworkTopologyStrategy', 'dc_east': 3, 'dc_west': 3
};

What to take from this chapter

• Use GossipingPropertyFileSnitch. Changing snitches on a live cluster moves data — be very careful.
• Adding a data centre is online: switch clients to LOCAL_* first, then auto_bootstrap: false, alter the keyspace, then nodetool rebuild.
• Active-active is the default outcome. Each write crosses the WAN once per DC; conflicts resolve by timestamp, so clock sync is a correctness requirement.
• Losing a whole data centre with LOCAL_QUORUM is a non-event. No failover, no split brain.
• Replication is not backup. Snapshots are hard links — instant, but they hold disk as compaction proceeds. Clean them up.
• Keep auto_snapshot: true. Store backups off-cluster. Practise restores.
• Cassandra ships with authentication and encryption off. Turn them on, and raise system_auth replication.

Next: running the thing day to day.