6. Health checks

Registration puts a service in the catalog. Health checks are what keep the catalog honest, and they are the reason Consul is more than a directory.

The defining property: checks run on the local agent. The agent on the machine hosting orders-2 checks orders-2. Nothing polls ten thousand endpoints from one place.

   CENTRAL POLLER                    LOCAL AGENT

   one machine checks everything     each machine checks itself
   scales badly                      scales with the fleet
   sees the network path,            sees the service the way a
     not the service                   neighbour on the box does
   a partition means everything      a partition means one node
     looks down                        cannot report

The seven kinds

HTTP

check {
  id       = "orders-http"
  name     = "HTTP /ready"
  http     = "http://localhost:8080/ready"
  method   = "GET"
  interval = "10s"
  timeout  = "2s"
  header   = { Authorization = ["Bearer check-token"] }
}

2xx passes. 429 is warning. Everything else is critical.

429 mapping to warning rather than critical is deliberate and useful: a service that is rate-limiting the health check is overloaded, not broken, and a warning state keeps it in DNS while making the condition visible.

For HTTPS with a private CA: tls_skip_verify = true works and tls_server_name plus a proper CA is better.

TCP

check {
  tcp      = "localhost:5432"
  interval = "10s"
  timeout  = "1s"
}

Opens a connection and closes it. Better than nothing and much worse than it sounds — a process can accept connections while its thread pool is exhausted, its database is unreachable, or it is thirty seconds into a five-minute startup. Use it only for protocols you have nothing better for.

UDP

check {
  udp      = "localhost:53"
  interval = "10s"
}

Sends a packet; an ICMP port-unreachable fails it. Weak evidence, since silence is indistinguishable from success — for DNS specifically, a script check running dig is far better.

gRPC

check {
  grpc            = "localhost:9090/orders.v1.OrderService"
  grpc_use_tls    = true
  interval        = "10s"
}

Uses the standard gRPC health-checking protocol. Omit the service name after the slash to check overall server health.

Script

check {
  args     = ["/usr/local/bin/check-disk.sh", "/data", "85"]
  interval = "60s"
  timeout  = "5s"
}

Exit code decides: 0 passing, 1 warning, anything else critical. Standard output becomes the check’s Output field, visible in the UI and API — write something useful there, because during an incident it is the difference between “critical” and “critical: disk 91% full on /data”.

Script checks are disabled by default for good reason: they let anyone who can register a service run commands as the agent’s user.

enable_local_script_checks = true    # only from local config files
enable_script_checks       = true    # also from the API — DANGEROUS

Use enable_local_script_checks, never enable_script_checks. The second allows remote code execution through the agent’s HTTP API to anyone who can reach it, and it has been the root of real compromises.

Docker

check {
  docker_container_id = "abc123"
  shell               = "/bin/sh"
  args                = ["/health.sh"]
  interval            = "10s"
}

Runs a command inside a container. Useful when the agent is on the host and the service is containerised.

TTL — the inverted one

check {
  id  = "orders-ttl"
  ttl = "30s"
  notes = "the application must report in every 30s"
}

Nothing probes. The application must actively report. If it does not within the TTL, the check goes critical.

curl -X PUT http://localhost:8500/v1/agent/check/pass/orders-ttl \
  -d 'processed 4,120 messages since last report'
curl -X PUT http://localhost:8500/v1/agent/check/warn/orders-ttl
curl -X PUT http://localhost:8500/v1/agent/check/fail/orders-ttl -d 'queue backed up'

This is the right check for anything with no endpoint to probe: a background worker, a batch consumer, a cron job. A Kafka consumer that has stopped consuming still accepts TCP and still returns 200 from any HTTP endpoint it happens to have. A TTL check driven from inside the consume loop is the only kind that notices.

The failure mode to design around: a TTL check reports on the health of the reporting loop, not the work. If your report is a separate thread, it will keep reporting while the worker thread is deadlocked. Report from inside the loop that does the work.

Alias

check {
  id       = "depends-on-db"
  alias_service = "orders-db"
}

Inherits another service’s health. Useful for a proxy or a facade whose health genuinely equals its backend’s — and dangerous everywhere else, because it is exactly the cascade described at the end of this chapter.

Hysteresis: not flapping

A single failed probe should not remove an instance from rotation. A single successful probe should not put it back.

check {
  http                     = "http://localhost:8080/ready"
  interval                 = "10s"
  timeout                  = "2s"
  success_before_passing   = 2
  failures_before_warning  = 2
  failures_before_critical = 3
}

   without hysteresis:
     one dropped packet → critical → removed from DNS
     next probe passes  → passing  → back in DNS
     the instance flaps, clients see intermittent failures

   with failures_before_critical = 3:
     3 consecutive failures over 30s → critical
     one blip changes nothing

Detection time is interval × failures_before_critical. The default (interval 10s, no hysteresis) detects in 10 seconds and flaps on a single packet loss. 10s × 3 detects in 30 seconds and does not. Choose deliberately; the right answer depends on how quickly you need failover versus how much flapping costs you.

failures_before_warning gives you an intermediate state — visible in the UI, still in DNS — which is a genuinely useful early signal.

Automatic deregistration

check {
  http                              = "http://localhost:8080/ready"
  interval                          = "10s"
  deregister_critical_service_after = "10m"
}

An instance critical for ten minutes is removed from the catalog entirely.

Set this, and do not set it too short. Without it, the catalog fills with dead instances that stay forever — every terminated container, every scaled-down VM. Too short, and an instance having a ten-minute bad time is deregistered and must fully re-register, losing its history.

Ten to thirty minutes is a reasonable range. Never set it shorter than your longest legitimate outage, such as a slow restart or a lengthy migration on startup.

Node checks versus service checks

# a NODE check — no service_id, so it applies to the whole machine
check {
  id       = "disk-space"
  args     = ["/usr/local/bin/check-disk.sh"]
  interval = "60s"
}

A failing node check marks every service on that node unhealthy. That is a powerful and blunt instrument. Disk full, out of memory, clock skew — genuine node-level problems belong here. Anything specific to one service does not, or one service’s problem removes every service on the box.

The three states

   passing    included in DNS and in ?passing queries
   warning    INCLUDED in DNS by default. Visible in the UI.
   critical   excluded from DNS and from ?passing

Warning being included in DNS surprises people. It is deliberate — a warning means “degraded, still usable” — and it is configurable:

dns_config {
  only_passing = true      # warning is now excluded too
}

Leave it as the default unless you have a reason. The value of a warning state is that it is visible without being disruptive; making it disruptive removes the distinction.

Writing a health endpoint that tells the truth

The same three rules as chapter 7 of the HAProxy guide, because they are properties of health checking rather than of any one tool.

Separate liveness from readiness. /healthz means “the process works” and should almost never be false. /ready means “send me traffic” and may legitimately be false during startup, during a drain, or while a cache warms. Consul should check readiness.

Check dependencies shallowly. Verify the database pool can hand out a connection. Do not run a query that touches user data, and do not measure anything expensive — a check running every ten seconds on fifty instances is 300 executions a minute.

Never cascade. This is the one that causes outages:

   web /ready  ──calls──►  orders /ready  ──calls──►  db check

   the database has a slow minute
        ↓
   orders /ready times out    → every orders instance critical
        ↓
   web /ready times out       → every web instance critical
        ↓
   NOTHING is in DNS. A slow minute became a total outage.

Your health endpoint must check your own ability to serve, not your dependencies’ health. If the database is slow, you should return 503s to requests — and stay in the catalog, so that when the database recovers you are already there.

alias_service checks are this cascade with configuration syntax. Use them only where the dependency really is definitional.

Inspecting checks

consul monitor -log-level=info

curl http://localhost:8500/v1/agent/checks | jq
curl "http://localhost:8500/v1/health/state/critical" | jq \
  '.[] | {Node, ServiceName, CheckID, Output}'

curl "http://localhost:8500/v1/health/service/orders" | jq \
  '.[] | {id: .Service.ID, checks: [.Checks[] | {CheckID, Status, Output}]}'

/v1/health/state/critical is the first command in an incident. Everything currently failing, across the whole datacentre, in one request.

The Output field is where the answer usually is — the HTTP body, the script’s stdout, the connection error. It is why writing a useful message in your health endpoint pays for itself.

Maintenance mode

consul maint -enable -service=orders -reason="deploying v1.5"
consul maint -disable -service=orders

consul maint -enable -reason="kernel patch"     # the whole node
consul maint -disable

Maintenance mode registers a critical check, so the service leaves DNS immediately while keeping its registration and its real check results. This is the correct way to take something out of rotation for a deploy — better than deregistering, because the registration and history survive, and better than stopping the process, because it is deliberate and has a reason attached that shows in the UI.

Put it in your deploy script, on both sides of the restart.

What to take from this chapter

Checks run on the local agent, which scales with the fleet and sees the service the way a neighbour does.
Seven kinds. TCP is weak; TTL is the only one that catches a background worker that has stopped working.
A TTL check reports on the reporting loop. Report from inside the loop that does the work, not from a separate thread.
Use enable_local_script_checks, never enable_script_checks — the latter is remote code execution through the agent API.
Add hysteresis. Detection time is interval × failures_before_critical; without it a single dropped packet removes an instance.
Set deregister_critical_service_after, and never shorter than your longest legitimate outage.
A node check marks every service on the node unhealthy. Use it only for genuine node problems.
Warning is included in DNS by default. That is deliberate.
Never let one health endpoint call another service’s. That cascade turns a slow dependency into a total outage.
consul maint is the right way to drain for a deploy.

Next: discovery — the DNS interface, the HTTP API, prepared queries, and where the convenience of DNS quietly betrays you.

Written by Sepuri Sai Krishna