4. Raft and consistency modes

Gossip decides who is alive. Raft decides what is true.

Everything durable in Consul — the catalog, the KV store, sessions, ACL tokens, configuration entries, mesh certificates — lives in a Raft log replicated across the servers.

Raft in one page

   THREE SERVERS

   ┌──────────┐      ┌──────────┐      ┌──────────┐
   │  LEADER  │─────►│ follower │      │ follower │
   └──────────┘      └──────────┘      └──────────┘
        │                                    ▲
        └────────────────────────────────────┘

   every write:
     1. client sends it to the leader (agents forward automatically)
     2. leader appends it to its log
     3. leader replicates to followers
     4. once a MAJORITY have it durably, it is COMMITTED
     5. leader applies it and answers the client

The majority requirement is the whole guarantee. Two overlapping majorities of the same set must share at least one member, so a newly elected leader is guaranteed to have seen every committed entry. That single property is why a leader change cannot lose an acknowledged write.

Chapter 3 of the etcd guide works through the safety argument in full; it is the same algorithm and the same reasoning, and Consul uses HashiCorp’s own Raft implementation rather than etcd’s library.

Losing quorum

   3 servers, 1 fails      2 remain, majority is 2  →  FINE
   3 servers, 2 fail       1 remains, majority is 2  →  NO LEADER

With no leader:

   WRITES        fail. Registration, deregistration, KV puts, ACL changes.
   CONSISTENT    fail.
   READS
   DEFAULT       still work, from any server's local state.
   READS
   STALE READS   still work.
   DNS QUERIES   still work — they are stale reads by default.
   AGENTS        keep running, keep health-checking, keep answering
                 from cache.

That last block is the important one. A Consul cluster that has lost quorum does not stop serving discovery. Existing services keep resolving, health checks keep running locally, and traffic keeps flowing. What stops is change: nothing new can register, nothing can deregister, and the catalog freezes as it was.

This is a genuinely good degradation story and it is worth knowing during an incident, because the instinct is to panic about quorum loss when the immediate user impact may be zero.

The three consistency modes

Every read in Consul picks one, and the default is not the strongest.

   ┌──────────────────────────────────────────────────────────┐
   │ "consistent"                                              │
   │   leader confirms it is still leader (a round trip to a   │
   │   majority) before answering                              │
   │   → definitely current. Slowest. Fails with no quorum.    │
   ├──────────────────────────────────────────────────────────┤
   │ "default"                            ← THE DEFAULT        │
   │   the leader answers from its own state, without          │
   │   confirming leadership                                   │
   │   → current unless there is a deposed leader that has     │
   │     not noticed yet — bounded by the leader lease         │
   ├──────────────────────────────────────────────────────────┤
   │ "stale"                                                   │
   │   ANY server answers from its own state                   │
   │   → may lag. Scales reads across all servers.             │
   │     Works with no leader at all.                          │
   └──────────────────────────────────────────────────────────┘

curl "http://localhost:8500/v1/health/service/orders?consistent"
curl "http://localhost:8500/v1/health/service/orders"          # default
curl "http://localhost:8500/v1/health/service/orders?stale"

The default mode’s window is small and real. A leader that has been partitioned away but has not yet noticed will answer from state that may be behind the new leader. Consul bounds this with a leader lease (leader_lease_timeout, 500 ms by default), so the staleness is at most that — but it is not zero, and default is not linearizable.

Stale reads are the right choice for service discovery, and it is worth being explicit about why. A read that is 50 ms behind tells you a service instance was healthy 50 milliseconds ago. That is exactly as useful as one that was healthy right now, because by the time your request arrives the instance may have died anyway. Service discovery is inherently an optimistic answer, and paying for linearizability to get it is buying a guarantee that the physics of the network takes away again immediately.

Every stale response carries how stale it is:

X-Consul-KnownLeader: true
X-Consul-LastContact: 12          ← ms since this server heard from the leader
X-Consul-Index: 128401

Check X-Consul-LastContact. A large value means the server answering you is behind, and max_stale in the agent configuration sets a limit past which it will forward to the leader instead rather than serve very old data.

When you genuinely need `consistent`

   leader election through the KV store        → consistent
   a lock's holder                             → consistent
   a check-and-set loop                        → consistent
   reading an ACL you just wrote                → consistent
   "is this service healthy?"                   → stale is fine
   DNS resolution                               → stale is fine

The rule: if you are going to act on the answer in a way that would be unsafe if the answer were stale, use consistent. Otherwise do not pay for it.

DNS and staleness

dns_config {
  allow_stale = true        # the default
  max_stale   = "87600h"    # effectively unlimited, by default
  node_ttl    = "0s"
  service_ttl = { "*" = "0s" }
  use_cache   = true
}

allow_stale = true by default is what lets DNS scale: every server can answer, so DNS throughput grows with the server count. max_stale at ten years is effectively “answer even if this server has been isolated for a long time” — which is a deliberate availability choice and worth lowering if you would rather fail than serve very old data:

  max_stale = "10s"

Leader election and elections you did not want

   follower hears nothing from the leader for `election_timeout`
        │
        ▼
   becomes a candidate, increments the term, requests votes
        │
        ▼
   a majority votes for it → it is the leader for that term

Elections are fast — typically under a second on a healthy network — and the cluster is unavailable for writes while one is in progress.

Elections you did not want come from three sources, in order of frequency:

raft_multiplier left at 5. The development default makes every timeout five times longer, which sounds safer and means a leader failure takes five times as long to detect. Set it to 1 in production.

Slow disks. Every Raft commit is an fsync on a majority of servers. A server on network storage with a 50 ms fsync makes every write take 50 ms, and under load the leader can fall behind its own heartbeat schedule. Put Consul’s data directory on local SSD, and watch consul.raft.fsm.apply and consul.raft.leader.lastContact.

Servers doing other work. A Consul server sharing a machine with a busy application will occasionally be starved of CPU at exactly the wrong moment. Give servers their own machines.

Autopilot

Autopilot is Consul’s automatic cluster maintenance, and it is on by default.

consul operator autopilot get-config

autopilot {
  cleanup_dead_servers      = true
  last_contact_threshold    = "200ms"
  max_trailing_logs         = 250
  server_stabilization_time = "10s"
}

cleanup_dead_servers removes a failed server from the Raft peer set once a replacement joins. Without it, a cluster where you replaced a dead server has four peers — three alive and one permanently dead — and its quorum is 3 of 4, so you have reduced your fault tolerance by adding a machine.

server_stabilization_time makes a new server sit as a non-voting member until it has been healthy for that long and has caught up on the log. This prevents the classic mistake of adding a server that immediately becomes a voter while still replaying a gigabyte of Raft log — raising the quorum requirement while being unable to participate.

Autopilot health is worth checking during any server operation:

consul operator autopilot state

   Healthy:                      true
   Failure Tolerance:            1
   Optimistic Failure Tolerance: 1
   Servers:
      srv-1  Healthy: true  Voter: true  StableSince: 2026-09-08T09:12:03Z
      srv-2  Healthy: true  Voter: true
      srv-3  Healthy: true  Voter: true

Failure Tolerance is the number to read. If it says 0, you cannot lose a server without losing quorum, whatever the server count says.

Replacing a server

The order matters, exactly as it does in etcd:

   1. Is the cluster healthy?     consul operator autopilot state
   2. REMOVE the dead server      consul operator raft remove-peer -id=<id>
   3. THEN add the replacement    start it with retry_join
   4. Verify it became a voter    consul operator autopilot state

Remove before you add. Adding first means momentarily having four peers where one is dead — quorum 3 of 4, with only 3 alive, so any further failure is fatal. Autopilot’s cleanup_dead_servers usually handles this for you; doing it explicitly means you are not depending on it.

Two commands to know:

consul operator raft list-peers
consul operator raft remove-peer -id=<node-id>

Snapshots

Raft logs are compacted into snapshots so they do not grow forever, and snapshots are also your backup.

consul snapshot save backup.snap
consul snapshot inspect backup.snap
consul snapshot restore backup.snap

A snapshot contains the catalog, the KV store, sessions, ACL tokens and configuration entries. It does not contain agent configuration — restoring it into a fresh cluster gives you back all the state, and the agents re-register their own services anyway through anti-entropy.

Take them on a schedule, store them off the machine, and test a restore. An untested backup is a hypothesis.

# a reasonable cron
consul snapshot save /backups/consul-$(date +%Y%m%d-%H%M).snap

Enterprise has an automatic snapshot agent; open source has cron.

The metrics that matter

consul_raft_leader_lastContact              # ms since a follower heard the leader
consul_raft_state_leader                    # 1 on exactly one server, always
consul_raft_apply                           # write throughput
consul_raft_commitTime                      # commit latency — includes fsync
consul_raft_fsm_lastRestoreDuration
consul_serf_member_failed                   # gossip-detected failures
consul_kvs_apply                            # KV write rate
consul_autopilot_failure_tolerance          # ALERT IF 0
consul_autopilot_healthy                    # ALERT IF 0

Alert on consul_autopilot_failure_tolerance == 0 and on more than one server reporting raft_state_leader == 1. The first means you are one failure from an outage; the second means something is very wrong indeed.

Rising consul_raft_commitTime is the earliest warning of a disk problem, and it precedes the elections that follow from it.

What to take from this chapter

Raft holds the catalog, KV, sessions, ACLs and config entries. Gossip holds membership. Different mechanisms for different needs.
Losing quorum stops writes, not discovery. Existing services keep resolving; the catalog simply freezes.
Three read modes, and the default is not the strongest — it is bounded by the leader lease, not linearizable.
Stale reads are correct for service discovery. A 50 ms-old answer is as useful as a current one, because the instance can die between the answer and your request.
Check X-Consul-LastContact, and lower max_stale if you would rather fail than serve very old data.
raft_multiplier = 1, local SSD, dedicated machines. Those three prevent most unwanted elections.
Autopilot’s Failure Tolerance is the number to read, not the server count.
Remove a dead server before adding its replacement.
Snapshots are the backup. Test a restore.

Next: registration and the catalog — service definitions, tags, metadata, and the reconciliation that puts back what you deleted.

Written by Sepuri Sai Krishna