11. Multi-region

Everything so far works in one datacentre, and in one datacentre CockroachDB is a competent distributed SQL database with a latency penalty. Across regions it becomes something no single-node database can imitate — and the difference between a good deployment and a miserable one is entirely in this chapter.

The only number that matters

The speed of light in fibre is about 200,000 km/s, and the internet is not a straight line. So:

   eu-west-1  ↔  eu-central-1      ~10 ms round trip
   us-east-1  ↔  us-west-2         ~60 ms
   us-east-1  ↔  eu-west-1         ~75 ms
   us-east-1  ↔  ap-southeast-1    ~230 ms

A write commits when a majority of its replicas has it (chapter 4). Put three replicas in three continents and every write waits for the second-nearest continent:

   replicas in us-east, eu-west, ap-southeast
   write from us-east:
       us-east   0 ms   ✓
       eu-west  75 ms   ✓  ← majority reached here
       ap-se   230 ms      (irrelevant)
   ───────────────────────────────────────────
   commit ≈ 75 ms, every single write, forever

No configuration removes that. Multi-region design is the art of arranging for the quorum to be near the client that writes, and for reads to avoid the quorum entirely.

Telling the cluster where it is

cockroach start --locality=region=eu-west,zone=eu-west-1a …

ALTER DATABASE store SET PRIMARY REGION "eu-west";
ALTER DATABASE store ADD REGION "us-east";
ALTER DATABASE store ADD REGION "ap-southeast";
SHOW REGIONS FROM DATABASE store;

Once a database has regions, the multi-region abstractions become available. They are a friendly vocabulary over the zone configurations of chapter 4 — you can express all of it by hand, and you will be glad not to.

Survival goals

The first decision, and it is a real trade rather than a preference:

ALTER DATABASE store SURVIVE ZONE FAILURE;    -- default
ALTER DATABASE store SURVIVE REGION FAILURE;

   SURVIVE ZONE FAILURE
     3 replicas, all in the primary region, spread
     across its zones.
     ✓ local writes are FAST (intra-region quorum)
     ✗ losing the whole region makes data unavailable

   SURVIVE REGION FAILURE
     ≥3 regions, replicas spread across them.
     ✓ an entire region can burn; the cluster serves on
     ✗ every write to that data pays a cross-region
       round trip, because quorum spans regions

That is the whole choice, and it is measurable in advance: region survival costs you the round trip to the second-nearest region on every write. If your business needs it, pay it deliberately. If it does not, zone survival with a well-placed primary region is dramatically faster.

Table localities

The second decision is per table, and it is where the product earns its reputation.

REGIONAL BY TABLE — the default

ALTER TABLE orders SET LOCALITY REGIONAL BY TABLE IN "eu-west";

All of this table’s leases and (under zone survival) its replicas live in one region. Fast for clients there; a round trip for everyone else. Correct for tables that belong to one market.

REGIONAL BY ROW — the reason people buy it

ALTER TABLE users SET LOCALITY REGIONAL BY ROW;

-- or with an explicit column you control
ALTER TABLE users ADD COLUMN region crdb_internal_region
  AS (CASE WHEN country IN ('DE','FR') THEN 'eu-west'
           ELSE 'us-east' END) STORED;
ALTER TABLE users SET LOCALITY REGIONAL BY ROW AS region;

Each row is homed in a region. German rows live in Europe, American rows live in America, in one table, with one schema, under one set of transactions.

   users
   ┌──────────────────────────────────────────────┐
   │ id  … region='eu-west'  ──► replicas in EU   │
   │ id  … region='us-east'  ──► replicas in US   │
   └──────────────────────────────────────────────┘

   A German user's read and write: entirely inside
   the EU. An American's: entirely inside the US.
   Same query, same table, different physics.

This is simultaneously the latency answer and the data-residency answer, which is why it exists. Two things to watch: a query that does not constrain the region column may fan out to every region, and a transaction touching rows homed in two regions pays for both.

GLOBAL — fast reads everywhere

ALTER TABLE products SET LOCALITY GLOBAL;

For tables read constantly everywhere and written rarely: catalogues, currency tables, feature flags, permissions.

   GLOBAL tables use non-blocking transactions: writes
   commit slightly in the FUTURE, so every replica can
   serve a consistent PRESENT-TIME read locally.

   reads   ~1 ms, in every region, always current
   writes  slower than normal — they wait out the
           future interval before acknowledging

That inversion — pay on write, read free everywhere — is exactly right for reference data and exactly wrong for anything hot.

Putting the lease where the readers are

Under zone survival you can also simply move the lease (chapter 4):

ALTER TABLE orders CONFIGURE ZONE USING
  lease_preferences = '[[+region=eu-west], [+region=us-east]]';

Leases are cheap to move, so this is the lightest-weight geographic tuning available, and it is often all a read-heavy table needs.

Reads that never leave the region

Chapter 8’s tools are how you stop cross-region reads without moving any data:

SELECT * FROM users AS OF SYSTEM TIME follower_read_timestamp() WHERE …;
SELECT * FROM users AS OF SYSTEM TIME with_max_staleness('10s') WHERE id = $1;

And chapter 4’s non-voting replicas put a local copy in a region without adding it to the quorum — which is the standard way to serve a distant region’s reads without taxing every write.

A worked deployment

Three regions, an application with EU and US users, a shared product catalogue:

   DATABASE store, regions eu-west (primary), us-east,
   ap-southeast. SURVIVE REGION FAILURE.

   users, orders, payments   REGIONAL BY ROW
       homed by the customer's region. EU customers'
       traffic stays in the EU.

   products, currencies      GLOBAL
       read everywhere at local speed, written by an
       admin job a few times a day.

   audit_log                 REGIONAL BY TABLE IN eu-west
       written from everywhere, read almost never.
       Accept the write latency; nobody is waiting.

-- confirm what you think you configured
SHOW CREATE TABLE users;
SELECT * FROM [SHOW RANGES FROM DATABASE store WITH DETAILS]
WHERE table_name = 'users' LIMIT 5;

Danger

Multi-region misconfiguration usually shows up as a mysteriously slow application, not as an error. The three classic causes: a table left at the default locality while its users are elsewhere; queries that do not constrain the region column of a REGIONAL BY ROW table, so every one fans out globally; and SURVIVE REGION FAILURE chosen reflexively on a database whose business never asked for it. All three are visible in EXPLAIN ANALYZE as network time, and invisible in the logs.

The arithmetic, done in advance

You can predict every one of these numbers before deploying anything. Three regions — us-east, eu-west, ap-southeast — and a client in eu-west:

   CONFIGURATION                      WRITE      READ
   ────────────────────────────────────────────────────
   zone survival, primary eu-west     ~3 ms     ~1 ms
     quorum inside eu-west                      (local
     lease in eu-west                            lease)

   zone survival, primary us-east    ~155 ms   ~75 ms
     every statement crosses the
     Atlantic twice                  ◄── the accident
                                         people deploy

   region survival, 3 regions         ~85 ms    ~1 ms
     quorum spans eu-west + us-east             (lease
                                                 local)

   region survival + REGIONAL BY ROW  ~12 ms    ~1 ms
     EU rows' quorum: eu-west +
     eu-central                       ◄── the goal

   GLOBAL table                       ~90 ms    ~1 ms
     writes wait out the future
     interval; reads are local
     everywhere

The second row is what a careless deployment produces: correct, durable, and four hundred times slower than it needed to be, with nothing in the logs to say so.

Note what the fourth row requires — more than one region per continent. Region survival with exactly three far-apart regions forces every quorum to cross an ocean. Adding a nearby second region (eu-west plus eu-central) lets EU data keep its majority in Europe while still surviving the loss of either. That is usually the single highest-value change available in a multi-region deployment.

Zones, before you reach for regions

Inside one region, spreading across availability zones costs almost nothing and buys real fault tolerance:

   eu-west-1a  replica
   eu-west-1b  replica       quorum latency ~1–2 ms
   eu-west-1c  replica       survives losing a zone

cockroach start --locality=region=eu-west,zone=eu-west-1b …

This is the default survival goal for a reason, and for a large number of businesses it is the correct final answer: a zone is a building, and losing a building is the failure that actually happens. Reach for multi-region when residency or global latency demands it — not because three regions sound more serious than three zones.

The cost of getting it right

Multi-region is the deepest part of the product and the most expensive to operate. Before adopting it, be honest about which of these you actually need:

   ✓ regulatory data residency          → REGIONAL BY ROW
   ✓ single-digit-ms reads worldwide    → GLOBAL, or
                                          follower reads
   ✓ survive losing an entire region    → SURVIVE REGION
                                          FAILURE, and pay
   ✗ "we might go global one day"       → one region now;
                                          add regions later,
                                          which is supported

The last line matters: regions can be added to a running cluster, and localities changed as online schema changes. You do not have to buy the complexity before the requirement arrives.

What to take from this chapter

• The only number that matters is the round trip to the second-nearest replica. Multi-region design is arranging for the quorum to sit near the writer.
• Nodes declare --locality; databases declare regions. The multi-region syntax is a vocabulary over chapter 4’s zone configurations.
• Survival goal is the first decision: zone failure keeps quorum local and fast; region failure survives losing a whole region and taxes every write with a cross-region round trip.
• Table locality is the second: REGIONAL BY TABLE for market-specific data, REGIONAL BY ROW to home each row where its user is — the latency and data-residency answer — and GLOBAL for read-everywhere reference data, which pays on writes so reads are free.
• Lease preferences move reads closer with no data movement, and non-voting replicas plus follower reads serve distant regions without taxing the write path.
• Misconfiguration presents as unexplained slowness, never an error: wrong locality, unconstrained REGIONAL BY ROW queries, or region survival nobody asked for. EXPLAIN ANALYZE network time is where you see it.
• Regions can be added later. Do not buy this complexity before the requirement exists.

Next: getting data in and out — backups, restores, changefeeds and imports, and how they interact with the GC window.