2. Agents, servers and datacentres

Consul is one binary that runs in two modes, and understanding which mode holds what explains almost everything else.

   ┌────────────── DATACENTRE dc1 ─────────────────────────────┐
   │                                                            │
   │   SERVERS (3 or 5)                                         │
   │   ┌────────┐  ┌────────┐  ┌────────┐                       │
   │   │ leader │──│follower│──│follower│   Raft: the catalog,   │
   │   └───┬────┘  └────┬───┘  └────┬───┘   the KV store,        │
   │       │            │           │       ACLs, config entries │
   │       └────────────┴───────────┘                            │
   │                    │                                        │
   │            LAN gossip pool — every node, servers included   │
   │       ┌────────────┼────────────┬─────────────┐             │
   │   ┌───▼───┐   ┌────▼──┐    ┌────▼──┐    ┌─────▼─┐          │
   │   │ agent │   │ agent │    │ agent │    │ agent │          │
   │   │  +app │   │  +app │    │  +app │    │  +app │          │
   │   └───────┘   └───────┘    └───────┘    └───────┘          │
   │                                                            │
   └────────────────────────────────────────────────────────────┘

Client agents

An agent runs on every machine that hosts a service. It is small — tens of megabytes of memory — and it holds no durable state that matters.

What it does:

• Runs health checks for services registered on this machine.
• Answers DNS and HTTP queries on 127.0.0.1:8600 and 127.0.0.1:8500.
• Gossips with every other agent.
• Forwards writes and consistent reads to a server.
• Caches the results of some queries.

What it does not do: participate in Raft, hold the authoritative catalog, or store anything you cannot lose.

An agent’s disk can be deleted and the machine restarted with no data loss. It re-registers its services from configuration files and rejoins the gossip pool. That is worth knowing during an incident, because it makes “restart the agent” a safe first move.

Why applications talk to localhost

   app ──► 127.0.0.1:8500 ──► agent ──► (cache, or a server)

Three consequences fall out of this and they are the reason the design exists:

No bootstrapping problem. The application does not need to discover the discovery service. The address is always localhost.

No network hop for cached reads. Many queries are answered from the agent’s memory. A DNS lookup for a service is typically sub-millisecond.

Failure is local and obvious. If the agent is down, the application fails immediately and loudly, rather than timing out against a remote service that might be fine.

The trade is that you must run an agent everywhere — which in Kubernetes means a DaemonSet, and in a serverless environment means you cannot, which is one of the honest limitations in chapter 15.

Server agents

Servers do everything a client does, plus:

• Participate in Raft, electing a leader and replicating a log.
• Hold the authoritative state: the catalog, the KV store, sessions, ACL tokens, configuration entries, mesh certificates.
• Answer forwarded requests from client agents.
• Federate with servers in other datacentres over the WAN gossip pool.

server = true
bootstrap_expect = 3

bootstrap_expect tells a server how many peers to wait for before forming a cluster. Set it on every server, to the same number, and never to 1 in production — a single-server cluster has no fault tolerance and, worse, will happily form even when the other two were meant to join it.

How many servers

The Raft arithmetic, identical to etcd’s:

servers	quorum	tolerates	verdict
1	1	0	development only
2	2	0	strictly worse than 1
3	2	1	the standard
4	3	1	no better than 3, slower
5	3	2	for larger or more critical clusters
7	4	3	rarely justified

Never an even number. Four servers tolerate exactly as many failures as three and require one more machine to agree on every write, so every write is slower. The rule is the same everywhere Raft appears, for the same reason.

Three for most; five when you want to survive losing two, or when you have enough load that read scaling helps. Beyond five, write latency grows because every commit waits for a majority of a larger set, and you gain fault tolerance you will not use.

The catalog

The catalog is the authoritative answer to “what exists”.

   NODES        machine name, address, node-level metadata
   SERVICES     name, id, address, port, tags, meta, the node it is on
   CHECKS       id, name, status, output, what it is attached to

It is stored in Raft, so it is strongly consistent and survives server failure. And it is derived — the catalog is a reflection of what agents report, not something you edit directly.

Anti-entropy, and the surprise it causes

Each agent continuously reconciles its local view against the catalog.

   agent's local state              catalog on the servers
   ─────────────────                ──────────────────────
   orders  :8080  passing    ──►    orders on node web-1, passing
   payments:9090  critical   ──►    payments on node web-1, critical

   every ~60s, and immediately on change:
     things the agent has that the catalog lacks → ADD
     things the catalog has for this node that
       the agent does not have                   → REMOVE

That last line is the surprise. If you deregister a service through the HTTP API but leave its definition in the agent’s configuration file, the agent puts it back at the next sync. People spend an afternoon on this.

# this works, until the next anti-entropy sync
curl -X PUT http://localhost:8500/v1/agent/service/deregister/orders

# the durable version
rm /etc/consul.d/orders.hcl && consul reload

The agent is the source of truth for its own node. The catalog is a cache of every agent’s opinion. Register and deregister at the agent, and the catalog follows.

The gossip pools

Two separate pools, and mixing them up is a common configuration error.

   LAN pool   every agent in ONE datacentre, clients and servers.
              Fast failure detection, ~200ms probes.
              Port 8301 (TCP and UDP).

   WAN pool   SERVERS ONLY, across datacentres.
              Slower, tuned for higher latency.
              Port 8302 (TCP and UDP).

Gossip is how node failure is detected — not by the servers polling anybody, but by nodes probing each other. Chapter 3 is entirely about the mechanism.

Node failure and service failure are different things. Gossip detects a machine that stopped responding. Health checks detect a service that is failing while its machine is fine. You need both, and they arrive through different paths.

Datacentres and WAN federation

A datacentre in Consul means a group of agents with low latency between them — a real datacentre, an AWS region, a Kubernetes cluster. It is the unit of Raft: each datacentre has its own servers, its own leader, and its own catalog.

   ┌─── dc1 ────────┐        ┌─── dc2 ────────┐
   │ servers (Raft) │        │ servers (Raft) │
   │ catalog, KV    │        │ catalog, KV    │
   └───────┬────────┘        └────────┬───────┘
           │      WAN gossip pool     │
           └──────────────────────────┘
             servers only, port 8302

consul join -wan 10.0.2.5

What federation gives you is the ability to query another datacentre:

dig @127.0.0.1 -p 8600 orders.service.dc2.consul
curl "http://localhost:8500/v1/health/service/orders?dc=dc2&passing"

What it does not give you is a shared catalog or a shared KV store. Each datacentre is independent. A cross-datacentre query is forwarded to that datacentre’s servers and answered there.

This is the correct design and people expect otherwise. A single Raft group spanning continents would mean every write waiting for a transatlantic round trip, and a network partition between regions would take down the majority side too. Independent datacentres mean dc1 keeps working perfectly when dc2 is unreachable — you simply cannot ask about dc2.

If you need replicated KV across datacentres, that is a separate mechanism (consul-replicate, or the Enterprise replication features), and it is asynchronous by necessity.

Ports

   8300  TCP        server RPC — servers only
   8301  TCP+UDP    LAN gossip — every agent
   8302  TCP+UDP    WAN gossip — servers only
   8500  TCP        HTTP API and UI
   8501  TCP        HTTPS API
   8502  TCP        gRPC — for Envoy sidecars
   8503  TCP        gRPC over TLS
   8600  TCP+UDP    DNS

Both TCP and UDP for the gossip ports. A firewall that allows only TCP produces a cluster where gossip mostly works and failure detection is strange and slow, because the fast UDP probe path silently fails and everything falls back to TCP. It is one of the more confusing misconfigurations to diagnose, and chapter 14 covers what it looks like.

Consul on Kubernetes

The shape is the same, expressed in Kubernetes objects:

   servers      StatefulSet, 3 or 5 replicas, persistent volumes
   agents       DaemonSet, one per node
   sidecars     Envoy, injected into pods by a mutating webhook
   sync         a controller mirroring Kubernetes Services into
                the Consul catalog, and optionally the reverse

global:
  name: consul
  datacenter: dc1
server:
  replicas: 3
client:
  enabled: true
syncCatalog:
  enabled: true
  toConsul: true
  toK8S: false
connectInject:
  enabled: true

syncCatalog is the interesting part when Consul and Kubernetes coexist. toConsul publishes Kubernetes Services into Consul so that non-Kubernetes workloads can find them; toK8S does the reverse, creating ExternalName Services for things outside the cluster. That bidirectional bridge is the strongest reason to run Consul in a cluster that already has Kubernetes’ own Service discovery — you are joining two worlds, not replacing one.

Newer versions can run without client agents (dataplane mode), which removes the DaemonSet and talks to servers directly. It is simpler to operate and gives up the local-cache and local-health-check properties this chapter has been arguing for; the trade is reasonable in Kubernetes, where the kubelet is already doing liveness checking.

What to take from this chapter

• Clients hold nothing durable and can be wiped and restarted safely. Servers hold everything in Raft.
• Three or five servers, never an even number. Four tolerates the same failures as three and is slower.
• Applications talk to localhost — no bootstrapping, no network hop, and failure is local and loud.
• The agent is the source of truth for its own node, and anti-entropy re-adds anything you deregistered through the API while leaving it in a config file.
• Two gossip pools: LAN for everything in a datacentre, WAN for servers across them. Both need TCP and UDP.
• Each datacentre has its own Raft, leader and catalog. Federation lets you query across, not share across — and that is the right design.
• In Kubernetes, syncCatalog bridges two worlds, which is the best reason to run Consul alongside a cluster that already has Services.

Next: gossip — the SWIM protocol, how failure detection actually works, and why it scales where heartbeats do not.