3. Gossip and failure detection

Ten thousand machines. How does the cluster notice when one of them dies?

The obvious answer is heartbeats to a central server, and it does not scale.

   HEARTBEAT TO A SERVER

   10,000 nodes × 1 heartbeat/sec  = 10,000 packets/sec at the server
   plus 10,000 timers to track
   plus: the server is now a single point of failure for
         knowing anything at all
   plus: nodes learn about each other only by asking the server

Consul does something else, inherited from Serf, which implements a protocol called SWIM — Scalable Weakly-consistent Infection-style process group Membership. The name is a mouthful and the idea is simple: nodes check on each other, and tell each other what they found.

The probe

Every node, on a timer (probe_interval, 1 second by default in the LAN pool), picks one other node at random and pings it.

   node A picks node F at random

   A ──── ping ────► F
   A ◄─── ack ────── F        F is alive. Done.

That is the entire steady state. One packet out, one packet back, per node per second, regardless of cluster size. A ten-thousand-node cluster generates the same per-node load as a ten-node cluster, which is the property that makes gossip work where heartbeats do not.

The indirect probe

Now F does not answer. It might be dead. It might be a dropped packet, a moment of GC, or a network hiccup between A and F specifically.

Declaring a node dead on one missed packet would produce constant false positives. So A asks for help.

   A ──── ping ────►  F        (no ack within probe_timeout)

   A picks k other nodes (indirect_checks, default 3):

   A ──► B ── ping ──► F
   A ──► C ── ping ──► F
   A ──► D ── ping ──► F

   if ANY of them gets an ack, they tell A, and F is alive.
   if none of them do, A suspects F.

This is the part that makes SWIM robust. A failure of the A–F network path is not a failure of F, and asking three other nodes distinguishes the two. It also means a node behind a partial partition is not evicted by whichever peer happened to be on the wrong side of it.

Consul adds a TCP fallback to the same probe, which catches the case where UDP specifically is being dropped — a firewall allowing TCP but not UDP on port 8301 is common enough that this exists to survive it, though the result is a cluster whose failure detection is much slower than it should be.

Suspicion

A node is not marked dead immediately. It is marked suspect, and that state is gossiped.

   ALIVE ──► SUSPECT ──► DEAD ──► LEFT
                │
                └── if the node refutes it: back to ALIVE

The refutation is the elegant part. When F hears — through gossip, from any node — that it is suspected, it broadcasts an “alive” message with a higher incarnation number, a counter each node keeps for itself.

   B gossips:  "F is suspect, incarnation 7"
      ...
   F hears it: "I am alive, incarnation 8"

   higher incarnation wins. Everyone marks F alive again.

So a node that was merely slow, or briefly unreachable, corrects the record itself. Only a node that is genuinely gone stays silent and eventually transitions to dead.

The suspicion timeout scales with cluster size, logarithmically: in a larger cluster the message takes longer to reach everyone, so the node is given proportionally longer to object. Consul computes this rather than using a fixed value, which is why you cannot state a single “time to detect a failure” number without knowing the cluster size.

Dissemination

How does everyone learn what A found out? Piggybacking.

   Every ping and ack carries a few pending state updates.

   A ──ping + "F is suspect", "G joined"──► B
   A ◄─ack + "H left", "I is alive"────── B

There is no separate broadcast channel and no flood. Updates ride along on the probe traffic that was happening anyway, and they spread the way a rumour does — each node that learns something tells the next few nodes it happens to talk to.

   round 1:      1 node knows
   round 2:      3 know
   round 3:      9 know
   round 4:     27 know
   ...
   ~log(N) rounds to reach everyone

For 10,000 nodes that is roughly a dozen rounds — a handful of seconds — and the total traffic is bounded because each update is only piggybacked a limited number of times before being retired.

Consul also runs a slower full state sync (push-pull, every 30 seconds by default) where a node exchanges its entire membership list with one random peer. Gossip is probabilistic and can, rarely, leave a node with a stale view; the full sync guarantees convergence rather than merely making it very likely.

Why “weakly consistent” is the right choice

The W in SWIM is weakly-consistent, and it means exactly what it says: at any instant, different nodes may have different opinions about who is alive.

For membership, that is fine and it is deliberate:

   node A thinks F is alive       (has not heard yet)
   node B thinks F is suspect
   node C thinks F is dead

   → they converge within seconds
   → meanwhile the WORST case is a few requests sent to a dead node,
     which every client already has to handle

Contrast with the catalog, which is in Raft and strongly consistent. The split is deliberate: membership is high-frequency, tolerant of brief disagreement, and must scale; the catalog is lower-frequency, must be correct, and does not need to scale the same way. Using consensus for membership would make a ten-thousand-node cluster impossible; using gossip for the catalog would mean two clients getting different answers about a service’s configuration.

If you have read chapter 3 of the etcd guide, this is the contrast worth holding: Raft refuses to answer when it cannot be certain; gossip always answers and lets the answer be briefly stale. Consul uses both, for the things each is right for.

Tuning

performance {
  raft_multiplier = 1
}

The single most useful knob is not a gossip setting but the network profile:

consul agent -config-file=...

# in the agent config
performance {
  raft_multiplier = 1     # 1 = production, 5 = the dev default
}

raft_multiplier scales every Raft and gossip timing constant. The default of 5 is deliberately slack so that a laptop or a small VM does not trigger constant elections; in production on real hardware, set it to 1, which makes leader election and failure detection roughly five times faster. This is the most commonly missed production setting in Consul.

For the gossip layer itself, Consul ships presets rather than making you tune individual values:

consul agent -config-file=... -node-meta=...

# choose a profile matched to the network
# default is "lan"; use "wan" for high-latency links,
# and "local" only for a laptop

Direct tuning is possible and rarely wise:

gossip_lan {
  probe_interval  = "1s"     # how often each node probes
  probe_timeout   = "500ms"  # how long to wait for an ack
  suspicion_mult  = 4        # scales the suspicion timeout
  retransmit_mult = 4        # how many times an update is piggybacked
}

Making these more aggressive trades false positives for speed. A probe_timeout shorter than your p99 network latency plus a GC pause will mark healthy nodes suspect constantly, and the resulting gossip storm makes everything worse. If failure detection feels slow, check raft_multiplier and your UDP connectivity before touching these.

Encryption

Gossip carries membership information and it is unauthenticated by default. Anyone who can send packets to port 8301 can join your cluster.

consul keygen
# 3Qc0dLxJ1LEnaTBRnrPVL/BsFBs4gDLXCaLDF9nq6/Y=

encrypt = "3Qc0dLxJ1LEnaTBRnrPVL/BsFBs4gDLXCaLDF9nq6/Y="
encrypt_verify_incoming = true
encrypt_verify_outgoing = true

Every agent needs the same key. Rotating it is a rolling operation with a keyring:

consul keyring -install=<new-key>    # every node now accepts both
consul keyring -use=<new-key>        # every node now sends with the new one
consul keyring -remove=<old-key>     # retire the old
consul keyring -list

Do the three steps in that order, with a pause between each, and verify with -list that every node has the key before moving on. Skipping to -use before every node has installed the key partitions your cluster into two halves that cannot hear each other — which is the most self-inflicted outage available in Consul.

Gossip encryption is symmetric and separate from the TLS that protects the RPC and HTTP layers (chapter 12). You need both; one does not imply the other.

Watching it work

consul members                    # the membership list, from gossip
consul members -detailed
consul monitor -log-level=debug   # live agent log, gossip included

consul operator raft list-peers   # the Raft peers, a DIFFERENT list

consul members and consul operator raft list-peers answer different questions, and comparing them is a genuinely useful diagnostic: members is the gossip view of every node; raft list-peers is the consensus view of the servers. A server that appears in one and not the other is a specific, diagnosable problem.

consul rtt web-1 web-2     # estimated round-trip time between two nodes

consul rtt uses network tomography — Consul continuously estimates coordinates for every node from gossip round-trip times, so it can tell you the approximate latency between any two nodes without probing them. It is a small, unusual feature, and it powers PreparedQuery failover to the nearest datacentre, which chapter 7 covers.

What to take from this chapter

• Each node probes one random peer per second. Load per node is constant regardless of cluster size, which is why gossip scales where heartbeats do not.
• Indirect probes through three other nodes distinguish “the node is dead” from “the path between us is broken”.
• Suspicion is refutable. A node that hears it is suspected broadcasts a higher incarnation number and is marked alive again.
• Updates piggyback on probe traffic and spread in about log(N) rounds; a slower full state sync guarantees convergence.
• Membership is weakly consistent by design; the catalog is Raft. Each mechanism is used for what it is right for.
• raft_multiplier = 1 in production. The default of 5 is for laptops and it makes everything five times slower to detect.
• Gossip is unauthenticated without encrypt. Rotate the key in three ordered steps, or you split the cluster.
• Compare consul members with raft list-peers — they answer different questions and the difference is diagnostic.

Next: Raft and the consistency modes — what “default”, “consistent” and “stale” actually cost, and when each is the right choice.