8. A lookup, traced end to end

An application calls getaddrinfo("orders.service.consul"). This chapter follows that request all the way down, then does the same for the HTTP API, then kills an instance and watches the system react.

1. The application resolves

requests.get("http://orders.service.consul:8080/orders/42")

The HTTP library calls the system resolver. Everything that happens next depends on configuration outside your program, which is why chapter 7 spends its length on caching.

   application
        │
   language runtime cache      ← the JVM caches FOREVER by default
        │
   /etc/nsswitch.conf
        │
   systemd-resolved / nscd     ← its own cache
        │
   /etc/resolv.conf → 127.0.0.53
        │
   forwarded because the domain matches ~consul
        │
        ▼
   127.0.0.1:8600   the local Consul agent

If any layer above returned a cached answer, the story stops here and you got whatever was true when it was cached.

2. The agent parses the name

   orders.service.consul
   │      │       └── the configured domain
   │      └────────── lookup type: service
   └───────────────── the service name

   variants the parser also accepts:
     primary.orders.service.consul       tag filter
     orders.service.dc2.consul           another datacentre
     orders.query.consul                 a prepared query
     web-1.node.consul                   a node address

3. Cache, or a server

dns_config {
  use_cache     = true
  cache_max_age = "10s"
  allow_stale   = true
}

   agent's cache holds a fresh answer?
        │
        ├── YES → answer immediately, ~0.2 ms, no network at all
        │
        └── NO  → RPC to a server

With allow_stale = true — the default — the RPC goes to any server, not the leader. That is what makes DNS throughput scale with server count, and it means DNS keeps working when the cluster has no leader at all.

The agent’s cache is also refreshed in the background by a blocking query, so a popular service’s entry is kept warm and updated within milliseconds of a change rather than being re-fetched on expiry.

4. The server answers from its state machine

   server receives: "healthy instances of `orders` in dc1"
        │
   read the catalog from the Raft FSM (in memory)
        │
   join services to their check states
        │
   filter: any check critical → EXCLUDE
           warning            → INCLUDE (default)
           passing            → include
        │
   return the surviving instances

The state machine is in memory. A read is a map lookup, not a disk access, which is why Consul reads are fast and why memory is the resource that matters on a server.

The response carries the staleness metadata:

X-Consul-Index: 128401
X-Consul-KnownLeader: true
X-Consul-LastContact: 12

LastContact of 12 ms means this server heard from the leader 12 milliseconds ago, so the answer is at most that stale. If it exceeded max_stale, the agent would retry against the leader instead.

5. The agent builds the DNS response

   3 healthy instances
        │
   shuffle                     ← randomised on EVERY response
        │
   truncate to `a_record_limit` if set
        │
   build A records, TTL from dns_config

The shuffle is the load balancing. There is no algorithm, no state and no coordination — just a fresh random order per response, which distributes clients evenly enough for most purposes.

;; ANSWER SECTION:
orders.service.consul.  0  IN  A  10.0.1.12
orders.service.consul.  0  IN  A  10.0.1.11
orders.service.consul.  0  IN  A  10.0.1.13

UDP responses are capped at 512 bytes unless the client offers EDNS0. That is roughly 25 A records. Beyond it, Consul truncates and sets the TC bit, which should make the client retry over TCP — and some clients silently accept the truncated answer instead.

dns_config {
  a_record_limit = 3      # cap explicitly rather than being truncated
  enable_truncate = true
}

For a service with a hundred instances, DNS is the wrong interface. Three addresses chosen at random is fine; a truncated list that some clients handle badly is not.

6. Back up the stack

The answer travels back through every cache that asked. Each may store it for its own idea of how long, regardless of the TTL 0 that Consul set. This is where discovery is most often lost.

The whole path

   getaddrinfo()
        │  0.01 ms   language cache (may return here — often does)
        │  0.05 ms   OS resolver cache
        │  0.10 ms   → 127.0.0.1:8600
        │
   agent parse
        │  0.20 ms   → agent cache HIT → done
        │
        │  1.5 ms    → cache miss → RPC to any server
        │              catalog read from memory
        │              filter by health
        │              back to the agent
        │
   shuffle, build records
        │
   ~0.2 ms cached · ~1.5 ms uncached

The HTTP version

curl "http://localhost:8500/v1/health/service/orders?passing"

Same path, more information:

[
  {
    "Node": {"Node": "web-1", "Address": "10.0.1.11"},
    "Service": {
      "ID": "orders-1", "Service": "orders",
      "Address": "", "Port": 8080,
      "Tags": ["v2", "primary"],
      "Meta": {"version": "1.4.2", "team": "commerce"}
    },
    "Checks": [
      {"CheckID": "serfHealth", "Status": "passing", "Name": "Serf Health Status"},
      {"CheckID": "orders-ready", "Status": "passing", "Output": "HTTP GET ...: 200 OK"}
    ]
  }
]

Note serfHealth. Every node automatically gets this check, and it is the gossip layer’s verdict on whether the node is alive at all. A node that has stopped gossiping has serfHealth critical, which marks every service on it unhealthy — no service-level check involved. Gossip and health checks reaching the same conclusion through different paths is exactly the redundancy you want.

An empty Service.Address means “use the node’s address”, which is the common case.

The blocking-query version

curl -i "http://localhost:8500/v1/health/service/orders?passing"
# X-Consul-Index: 128401

curl "http://localhost:8500/v1/health/service/orders?passing&index=128401&wait=5m"
# ... hangs ...

   t=0        request arrives, index 128401
              server: current index is 128401, nothing to say
              → register a watcher, hold the connection

   t=0..300s  nothing happens. The connection is idle.

   t=47s      orders-2's check goes critical
              → catalog write, index becomes 128402
              → every watcher on this key is woken
              → response sent with the new list

   OR

   t=300s     wait expires → return the same data,
              same index. The client asks again.

Milliseconds from failure to notification, with one idle connection instead of a poll every five seconds. This is the mechanism behind consul-template, Envoy’s configuration updates, and every Consul-aware library.

An instance dies

The end-to-end story, with times.

A clean shutdown

   t=0.0s   the process handles SIGTERM
            → PUT /v1/agent/service/deregister/orders-2
   t=0.01s  the agent removes it locally
   t=0.02s  anti-entropy pushes the change; catalog index bumps
   t=0.03s  every blocking query watcher is woken
   t=0.05s  consul-template renders; HAProxy reloads
   t=0.1s   agent DNS caches updated

   → about 100 ms, and NOT ONE request is sent to the dead instance

Deregistering on SIGTERM is the highest-value thing you can add to a service, and it costs about five lines. Everything below is the fallback for when a process does not get the chance.

A crash

   t=0s     the process dies. No deregistration.
   t=0-10s  the next health check probe fires
            → connection refused
   t=10s    failure 1 of 3
   t=20s    failure 2
   t=30s    failure 3 → CRITICAL
            → catalog write, watchers woken
   t=30.05s consul-template renders; HAProxy reloads
   t=30.1s  DNS stops returning it

   → 30 seconds, during which clients get connection-refused
     and must retry

interval × failures_before_critical. To halve it, halve the interval or reduce the hysteresis — and accept more flapping. This is why clients must retry regardless of how good your discovery is.

The machine dies

   t=0s     power off. No process, no agent, no packets.
   t=0-1s   some node probes it → no ack
   t=1s     indirect probes through 3 peers → no ack
   t=1s     marked SUSPECT, gossiped
   t=~5s    no refutation → marked FAILED
            → serfHealth goes critical for that node
            → EVERY service on it becomes unhealthy at once
   t=~5s    catalog write, watchers woken

   → about 5 seconds, and FASTER than the health-check path

Gossip beats health checks for whole-machine failure, which is the division of labour working as intended: gossip is optimised for “is this node there”, health checks for “is this service working”. A dead machine is caught by the fast path.

What to take from this chapter

A cached answer higher up the stack ends the story before Consul is consulted. The JVM’s forever-cache is the classic case.
The agent answers most DNS from its own cache in ~0.2 ms, kept warm by a background blocking query.
allow_stale sends the RPC to any server, which is what makes DNS scale and keeps it working with no leader.
DNS load balancing is a shuffle — no algorithm, no state, fresh order per response.
UDP caps the answer at about 25 records. For a service with a hundred instances, DNS is the wrong interface.
serfHealth is the gossip verdict on every node, and a failing one marks every service on that node unhealthy.
Blocking queries notify in milliseconds and are how every Consul-aware tool works.
Deregister on SIGTERM. Clean shutdown is ~100 ms and loses no requests; a crash is interval × failures; a dead machine is ~5 s through gossip.
Clients must retry anyway. No discovery system removes the window.

Next: the key-value store — sessions, locks, leader election, and the fencing token that a lock is not a lock without.

Written by Sepuri Sai Krishna