14. Failure modes and war stories

Every failure here is one that has cost somebody an evening. Symptom, mechanism, diagnosis, fix.

1. The gossip key rotation split the cluster

Half the nodes see one membership list, half see another. consul members disagrees depending on which node you run it from.

consul keyring -use was run before every node had -installed the new key. Nodes with the new primary key encrypt with it; nodes without it cannot decrypt, so from each side the other half appears to have vanished.

consul keyring -list      # what every node actually has

Fix by reinstalling the old key everywhere and going back to primary on it, then redoing the rotation properly:

consul keyring -install=<old>
consul keyring -use=<old>
consul keyring -list                     # VERIFY, on a node from each half
consul keyring -install=<new>
consul keyring -list                     # VERIFY AGAIN
consul keyring -use=<new>
consul keyring -remove=<old>

-list between every step. It is the whole prevention.

2. The service you deleted came back

curl -X PUT http://localhost:8500/v1/agent/service/deregister/orders-1
# 200 OK. Gone from the catalog.
# ...60 seconds later it is back.

Anti-entropy. The definition is still in /etc/consul.d/orders.hcl, and the agent re-asserts its local state into the catalog on every sync.

rm /etc/consul.d/orders.hcl
consul reload

The agent is authoritative for its own node. Deregister where the agent learns about the service, not in the catalog.

3. DNS still returns a dead instance

The instance is critical in the catalog. dig against the agent returns the correct list. The application keeps connecting to the dead address.

Something between the application and Consul is caching.

# what Consul says
dig @127.0.0.1 -p 8600 orders.service.consul

# what the system resolver says
dig orders.service.consul

# what the application sees — the only one that matters

The JVM is the classic case: networkaddress.cache.ttl defaults to caching forever when a security manager is installed.

java.security.Security.setProperty("networkaddress.cache.ttl", "0");

Also check nscd, systemd-resolved, a corporate forwarder, and — most often overlooked — the HTTP client’s own connection pool, which holds an open connection to an address it resolved once and never resolves again. A connection pool with an idle timeout is part of your discovery story whether you meant it to be or not.

4. Everything is unhealthy after a network blip

One node had a brief network problem. Now every service on it is critical, and it stayed that way after the network recovered.

serfHealth. Gossip marked the node failed, which marks every service on it unhealthy in one stroke — no service-level check involved.

consul members                                     # is the node alive?
curl "http://localhost:8500/v1/health/state/critical" | \
  jq '.[] | select(.CheckID == "serfHealth")'

If it recovered but is still marked failed, gossip is not reaching it. Check UDP on port 8301 specifically — see failure 5.

5. Failure detection is strangely slow

Nodes eventually get marked failed, but it takes tens of seconds rather than a few.

A firewall allowing TCP but not UDP on 8301. Gossip’s fast path is UDP; without it every probe falls back to TCP, which works and is much slower.

nc -zvu 10.0.1.11 8301        # UDP
nc -zv  10.0.1.11 8301        # TCP

Both protocols on 8301 and 8302. This is a genuinely confusing failure because nothing is broken — everything works, just badly, and the logs say nothing useful.

The other cause is raft_multiplier left at 5. Check that first, since it is one line.

6. Constant leader elections

consul_raft_state_leader flips between servers. Writes intermittently fail.

Three causes, in order of frequency:

Slow disk. Every Raft commit is an fsync. Graph consul_raft_commitTime; anything above a few milliseconds is a problem, and above 20 ms is why you are here.

fio --name=fsync --rw=write --sync=1 --bs=4k --size=64m --directory=/opt/consul

raft_multiplier at 5 on servers that are otherwise fine — the timings are so slack that a brief pause looks like a leader failure. Set it to 1.

CPU starvation. A Consul server sharing a machine with a busy application, or a CPU limit in Kubernetes. Give servers their own resources and set no CPU limit.

7. Quorum lost, and the panic was unnecessary

Two of three servers died. Nobody can write. The instinct is that everything is down.

Discovery keeps working. DNS is a stale read by default, agents keep answering from cache, health checks keep running, and traffic keeps flowing. What stops is change — no registration, no deregistration, no KV writes.

consul operator raft list-peers
consul operator autopilot state
consul info | grep -A5 raft

Try to bring the failed servers back first. peers.json recovery risks losing any write committed on a majority that excluded the survivor, so it is the second option, not the first.

8. A four-server cluster has less fault tolerance than three

Somebody replaced a dead server by adding a new one without removing the old.

consul operator raft list-peers
# srv-1 alive, srv-2 alive, srv-3 DEAD, srv-4 alive
# 4 peers → quorum is 3 → 3 alive → tolerance 0

Adding a machine reduced fault tolerance. consul operator autopilot state says Failure Tolerance: 0 and it is the number to believe.

consul operator raft remove-peer -id=<srv-3-id>

Autopilot’s cleanup_dead_servers normally handles this; it does not if the dead server never became unhealthy in autopilot’s view, which happens when it was removed abruptly.

9. Health checks flap

A service alternates between passing and critical every few probes. Clients see intermittent failures.

No hysteresis. One dropped packet or one slow response marks it critical, the next probe passes, and it returns.

check {
  interval                 = "10s"
  timeout                  = "5s"      # was 1s — too tight
  success_before_passing   = 2
  failures_before_critical = 3
}

Check the timeout against your real p99 latency, not against a round number. A 1-second timeout on an endpoint whose p99 is 800 ms will fail regularly for reasons that have nothing to do with health.

10. The health-check cascade

The database had a slow minute. Everything went down.

   db slow
     → orders /ready calls the db, times out → all orders critical
       → web /ready calls orders, times out  → all web critical
         → nothing in DNS. Total outage.

A health endpoint must check your own ability to serve, not your dependencies’ health. If the database is slow, return 503s to requests and stay in the catalog, so that when it recovers you are already there.

alias_service checks are this cascade written as configuration. Use them only where the dependency is definitional.

11. Deny-by-default broke DNS

ACLs were switched to default_policy = "deny". HTTP works with tokens. DNS returns nothing.

DNS carries no token, so it uses the anonymous token, which now grants nothing.

consul acl policy create -name dns-read -rules '
service_prefix "" { policy = "read" }
node_prefix ""    { policy = "read" }
query_prefix ""   { policy = "read" }'

# either give it to anonymous...
consul acl token update -id anonymous -policy-name dns-read

# ...or better, a dedicated DNS token per agent

acl { tokens { dns = "..." } }

The dedicated token is better: it does not grant anything to unauthenticated HTTP callers.

12. Locked out of ACLs

The bootstrap token is lost and no other management token exists.

Bootstrap reset. Attempt a bootstrap to make a server log the current reset index:

consul acl bootstrap
# Failed ACL bootstrapping: ... reset index: 13

echo 13 > /opt/consul/acl-bootstrap-reset
consul acl bootstrap

Do it on the leader, and note the index comes from the log, not from a command. Recoverable, and an unpleasant hour — put the bootstrap token in a password manager or Vault the moment it is issued.

13. An external service is permanently healthy

A managed database was registered directly into the catalog. It has been down for a day and Consul still says it is healthy.

No agent owns it, so nothing checks it. /v1/catalog/register writes a fact into the catalog and nothing maintains it.

Run consul-esm (External Service Monitor), which watches for nodes tagged external-node: true and health-checks them from wherever it runs:

{
  "Node": "external-rds",
  "NodeMeta": {"external-node": "true", "external-probe": "true"},
  "Address": "orders-db.abc123.rds.amazonaws.com",
  "Service": {"Name": "orders-db", "Port": 5432}
}

Without ESM, do not register external services. A permanently-healthy entry is worse than an absent one, because something will route to it.

14. Two service instances, one disappeared

Two instances were registered on one machine. Only one is in the catalog.

service { name = "orders" port = 8080 }     # id defaults to "orders"
service { name = "orders" port = 8081 }     # id ALSO defaults to "orders"

id must be unique per node, and it defaults to name. The second registration silently replaced the first, and you have half the capacity you believe you have.

service { id = "orders-8080" name = "orders" port = 8080 }
service { id = "orders-8081" name = "orders" port = 8081 }

15. Mesh traffic gets 503s that nothing explains

Services are healthy. Intentions look right. Requests fail.

The sidecar is the authority, not Consul’s configuration:

curl localhost:19000/config_dump | jq
curl localhost:19000/clusters
curl localhost:19000/stats | grep -E "upstream_rq_(5xx|pending_failure_eject)"
curl localhost:19000/certs

Four common causes:

No intention allows it — consul intention check web orders gives a straight answer.

Protocol not declared, so an L7 router silently does nothing and the request goes somewhere unexpected.

A subset filter matches no instances, so a splitter routes weight to an empty cluster. /clusters shows member counts; zero is the tell.

Certificate problems after a CA rotation. /certs shows what the sidecar actually holds and when it expires.

16. The KV store is enormous and the servers are struggling

consul_runtime_alloc_bytes climbing, servers slow, snapshots taking minutes.

Something is using the KV store as a database. Every byte is in memory on every server, and every write is a Raft commit.

consul kv export / | jq -r 'to_entries | .[] | "\(.value.key) \(.value.value | length)"' \
  | sort -k2 -rn | head -20

Move large values to object storage and keep a pointer. Move high-frequency writes anywhere else. The KV store is for coordination, not data, and the limits in chapter 9 are not suggestions.

A triage order

   1. Is the cluster healthy?     consul operator autopilot state
   2. Is there a leader?          consul operator raft list-peers
   3. Do all nodes agree?         consul members  (from several nodes)
   4. What is failing?            /v1/health/state/critical
   5. Read the check Output.      It usually names the problem.
   6. Is it the catalog or DNS?   dig @127.0.0.1 -p 8600 vs
                                  dig  — if they differ, it is caching
   7. Mesh problem?               the sidecar's /config_dump, not Consul's
   8. ACL problem?                the servers' "Permission denied" logs

Step 6 splits the most confusing class of problem in half. “Consul is returning a dead instance” is almost never Consul; it is a cache between Consul and the application.

What to take from this chapter

• consul keyring -list between every rotation step, or you split the cluster into halves that cannot hear each other.
• Anti-entropy re-adds what you deregistered while its definition is still in a config file.
• A dead address that persists is caching, not Consul. Compare dig @127.0.0.1 -p 8600 with dig, and remember connection pools.
• UDP on 8301 is not optional. TCP-only gossip works and is much slower, with no error anywhere.
• raft_commitTime rising precedes the elections it causes. Graph it.
• Losing quorum stops change, not discovery. Traffic keeps flowing.
• Read Failure Tolerance, not the server count. A stale peer makes four servers worse than three.
• Add hysteresis and a realistic timeout, or checks flap on ordinary jitter.
• Never let one health endpoint call another service’s.
• Deny-by-default breaks DNS until the anonymous or DNS token is handled.
• An externally registered service without consul-esm is permanently healthy, which is worse than absent.
• id defaults to name, so two instances on one node silently become one.

Next: the honest comparison — against etcd, Kubernetes, Istio and Nomad, and what the BUSL licence means for the decision.