13. Tuning and operations

Consul needs less tuning than most of the systems in this collection. What it needs is a handful of settings that are wrong by default for production, and a small set of operational habits.

The settings that matter

performance {
  raft_multiplier = 1
}

This is the single most impactful line in a production Consul configuration and it is missed constantly. The default is 5, chosen so a developer’s laptop does not trigger constant elections. It multiplies every Raft and gossip timing constant, so a leader failure that should be detected in under a second takes five.

   raft_multiplier = 5 (default)      = 1 (production)
   leader lease            2.5s              500ms
   heartbeat timeout       5s                1s
   election timeout        5s                1s
   → failure detected in ~5s         → ~1s

Set it to 1 on real hardware with local SSDs. Leave it at 5 only if your servers are on shared or slow infrastructure, where aggressive timings would cause the elections they are meant to detect.

# server sizing
limits {
  http_max_conns_per_client = 200
  rpc_max_conns_per_client  = 100
}

# do not let one runaway client exhaust the cluster

Sizing

   SERVERS

   small   (< 1,000 services)      2 vCPU,  8 GB, 50 GB SSD
   medium  (< 10,000 services)     4 vCPU, 16 GB, 100 GB SSD
   large   (> 10,000 services)     8 vCPU, 32 GB, 200 GB SSD

   memory is the constraint: the entire state machine — catalog,
   KV store, ACL tokens, sessions — lives in RAM on every server

   CLIENT AGENTS

   ~50 MB baseline, plus a little per registered service and check
   negligible CPU unless you have many script checks

Three things decide server memory:

Number of service instances. Each is a few KB of catalog. Ten thousand instances is tens of megabytes — rarely the problem.

KV store size. Every byte is in memory on every server. This is usually the largest contributor and the easiest to control.

Watchers. Each blocking query holds a connection and some state. A thousand agents each watching ten services is ten thousand watchers, and that is real.

Disk matters for one reason: fsync. Every Raft commit is an fsync on a majority of servers. Network storage with 20 ms fsync latency caps your write throughput at roughly 50 writes per second and makes elections likely under load.

# measure it before blaming Consul
fio --name=fsync --rw=write --sync=1 --bs=4k --size=64m --directory=/opt/consul

Local NVMe or SSD. Never network storage for a Consul server.

Upgrades

Consul supports rolling upgrades and the order is not negotiable.

   1. Snapshot.                     consul snapshot save pre-upgrade.snap
   2. Upgrade the FOLLOWERS, one at a time.
      After each: wait for it to rejoin and become a healthy voter.
   3. Upgrade the LEADER last.
      Optionally step down first, so the switch is deliberate:
        consul operator raft transfer-leader
   4. Upgrade the CLIENT agents. Any order, any pace.

# after each server
consul operator raft list-peers
consul operator autopilot state       # Healthy: true, Failure Tolerance: 1
consul members

Wait for Failure Tolerance to return to its normal value before touching the next server. Upgrading a second server while the first is still catching up is how a rolling upgrade becomes a quorum loss.

Read the upgrade notes. Consul has had version-specific requirements — intermediate versions you must pass through, and settings that must be changed before or after. Skipping several minor versions at once is where the surprises are.

Clients can be older than servers within a version or two; servers must not be older than clients. Upgrade servers first, always.

Backup and restore

consul snapshot save backup-$(date +%Y%m%d-%H%M).snap
consul snapshot inspect backup.snap
consul snapshot restore backup.snap

A snapshot contains the catalog, KV store, sessions, ACL tokens, configuration entries and mesh CA state. It does not contain agent configuration — and it does not need to, because agents re-register their own services through anti-entropy within a minute of the restore.

# a reasonable cron, with a token that can read everything
0 * * * * consul snapshot save /backups/consul-$(date +\%Y\%m\%d\%H).snap \
          && aws s3 cp /backups/consul-*.snap s3://backups/consul/ --sse

Two things people get wrong.

Snapshots contain every ACL token. Encrypt them at rest and restrict access as tightly as you would restrict the tokens themselves.

An untested restore is a hypothesis. Restore into a scratch cluster on a schedule and check that the catalog and KV contents are what you expect. The first time you try a restore should not be during an outage.

Recovering from lost quorum

The worst case: two of three servers are gone permanently.

   1. Do the remaining servers have the data?
      → try to bring the failed ones back first. Always try this.

   2. If not, force a new peer set:
      write raft/peers.json on the SURVIVING server,
      then restart it.

[
  {
    "id": "e4d3c2b1-...",
    "address": "10.0.1.5:8300",
    "non_voter": false
  }
]

Place it at <data_dir>/raft/peers.json and restart. The server reads it, forms a single-node cluster from its existing data, and deletes the file. Then add new servers normally.

This is a data-loss-risking operation. Any write committed on a majority that did not include the surviving server is gone. Take a snapshot of the surviving server’s data directory first, and do it only when you are certain the others are not coming back.

Monitoring

telemetry {
  prometheus_retention_time = "60s"
  disable_hostname          = true
}

curl http://localhost:8500/v1/agent/metrics?format=prometheus

The metrics that earn an alert:

# THE cluster health alerts
consul_autopilot_healthy == 0
consul_autopilot_failure_tolerance == 0        # one failure from an outage
sum(consul_raft_state_leader) != 1             # zero or two leaders

# performance, in order of leading-indicator value
consul_raft_commitTime                          # includes fsync — rises first
consul_raft_leader_lastContact                  # follower lag, ms
consul_raft_apply                               # write throughput
consul_kvs_apply

# gossip
consul_serf_member_failed
consul_serf_member_flap                         # a flapping node

# saturation
consul_runtime_alloc_bytes
consul_runtime_heap_objects
consul_rpc_request
consul_rpc_query
consul_client_rpc_exceeded                      # a client hitting rate limits

# the catalog
consul_catalog_service_query
consul_health_service_query

consul_autopilot_failure_tolerance == 0 is the alert to have if you only have one. It means the next server failure is an outage, and it fires whether the cause is a dead server, a server that never became a voter, or a stale peer nobody cleaned up.

consul_raft_commitTime rising is the earliest warning of a disk problem, and it precedes the leader elections that follow from it by minutes. Graph it.

consul_serf_member_flap catches a node that keeps joining and leaving — usually a network problem or a machine under memory pressure, and always worth investigating because a flapping node generates gossip churn for everyone.

Operational runbook

Draining a node for maintenance:

consul maint -enable -reason="kernel patch"
# wait for traffic to drain — watch your load balancer, not Consul
systemctl stop myapp
# ... do the work ...
systemctl start myapp
consul maint -disable

Permanently removing a client node:

consul leave                     # graceful: marks it LEFT, not FAILED

consul leave versus a crash matters. A graceful leave marks the node left and removes it cleanly. A killed agent is marked failed and lingers in consul members for 72 hours by default (reconnect_timeout), which is deliberate — a node that comes back should rejoin rather than be treated as new — and confusing when you have decommissioned it.

consul force-leave <node>        # for a node that is never coming back
consul force-leave -prune <node> # remove it from the member list entirely

Removing a server:

consul operator raft list-peers
consul operator raft remove-peer -id=<node-id>
# THEN start the replacement

Remove before add. Always.

Running Consul in Kubernetes

server:
  replicas: 3
  resources:
    requests: {memory: "4Gi", cpu: "1000m"}
    limits:   {memory: "8Gi"}
  storage: 50Gi
  storageClass: fast-ssd
  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app: consul
              component: server
          topologyKey: kubernetes.io/hostname
  disruptionBudget:
    enabled: true
    maxUnavailable: 0

Four things in that block that matter:

podAntiAffinity with requiredDuring..., not preferred. Two Consul servers on one node means one node failure costs you two servers, which for a three-server cluster is quorum.

maxUnavailable: 0 in the PodDisruptionBudget. A node drain that evicts a Consul server should be blocked until you do it deliberately.

A real storageClass. The default is often network storage, and the fsync argument above applies with full force.

No CPU limit, only a memory limit. A throttled Consul server misses heartbeats and triggers elections — the same argument as chapter 13 of the OpenTelemetry guide makes about the Collector.

Performance expectations

   DNS query, agent cache hit         ~0.2 ms
   DNS query, cache miss              ~1-2 ms
   HTTP health query, stale           ~1-2 ms
   HTTP health query, consistent      ~2-5 ms
   KV write                           ~5-20 ms   (an fsync on a majority)
   Blocking query notification        ~10-50 ms after the change
   Leader election                    ~1s   with raft_multiplier = 1
                                      ~5s   with the default of 5

   THROUGHPUT, 3 servers on decent hardware
   reads (stale)                      tens of thousands/sec
   writes                             several hundred/sec
   gossip                             tens of thousands of nodes

Writes are the bottleneck and they always will be — every one is a Raft commit with an fsync. If you are trying to write to Consul thousands of times per second, you are using it as a database and it is not one.

What to take from this chapter

• raft_multiplier = 1 in production. The default of 5 makes every failure take five times as long to detect.
• Memory is the server constraint — the whole state machine is in RAM on every server. The KV store is usually the largest part.
• Local SSD, never network storage. Every Raft commit is an fsync, and slow disks cause the elections you were trying to avoid.
• Upgrade followers first, leader last, clients whenever. Wait for Failure Tolerance to recover between each.
• Snapshots contain every ACL token. Encrypt them, and test a restore on a schedule.
• peers.json recovery risks data loss. Try to recover the failed servers first.
• Alert on autopilot_failure_tolerance == 0 if you alert on nothing else, and graph raft_commitTime.
• A killed agent lingers as failed for 72 hours by design; use consul leave or force-leave -prune.
• In Kubernetes: required anti-affinity, maxUnavailable: 0, fast storage, no CPU limit.

Next: the failures — the split cluster, the flapping check, the DNS that kept returning a dead address, and eleven more.