16. Cheat sheet

A production server configuration

datacenter       = "dc1"
data_dir         = "/opt/consul"
node_name        = "srv-1"
server           = true
bootstrap_expect = 3
bind_addr        = "10.0.1.5"
client_addr      = "0.0.0.0"
retry_join       = ["10.0.1.5", "10.0.1.6", "10.0.1.7"]

ui_config { enabled = true }

performance {
  raft_multiplier = 1          # THE production setting. Default 5.
}

encrypt = "3Qc0dLxJ1LEnaTBRnrPVL/BsFBs4gDLXCaLDF9nq6/Y="
encrypt_verify_incoming = true
encrypt_verify_outgoing = true

tls {
  defaults {
    ca_file   = "/etc/consul.d/tls/ca.pem"
    cert_file = "/etc/consul.d/tls/server.pem"
    key_file  = "/etc/consul.d/tls/server-key.pem"
    verify_incoming = true
    verify_outgoing = true
  }
  internal_rpc { verify_server_hostname = true }    # impersonation
}

ports {
  http  = -1                   # no plaintext API
  https = 8501
  grpc  = 8502
}

acl {
  enabled        = true
  default_policy = "deny"
  down_policy    = "extend-cache"
  enable_token_persistence = true
  tokens { agent = "..." dns = "..." }
}

connect { enabled = true }

autopilot {
  cleanup_dead_servers      = true
  server_stabilization_time = "10s"
}

telemetry {
  prometheus_retention_time = "60s"
  disable_hostname          = true
}

enable_local_script_checks = true     # NOT enable_script_checks
enable_script_checks       = false    # RCE via the agent API

dns_config {
  allow_stale   = true
  max_stale     = "10s"
  use_cache     = true
  cache_max_age = "10s"
  service_ttl   = { "*" = "5s" }
}

A client agent

datacenter = "dc1"
data_dir   = "/opt/consul"
retry_join = ["10.0.1.5", "10.0.1.6", "10.0.1.7"]
encrypt    = "..."
performance { raft_multiplier = 1 }
acl { enabled = true, default_policy = "deny", down_policy = "extend-cache",
      tokens { agent = "..." dns = "..." } }

A service definition

service {
  id      = "orders-8080"          # UNIQUE PER NODE. Defaults to name!
  name    = "orders"
  port    = 8080
  tags    = ["v2", "primary"]
  meta    = { version = "1.4.2", team = "commerce", protocol = "http" }

  check {
    id                                = "orders-ready"
    http                              = "http://localhost:8080/ready"
    interval                          = "10s"
    timeout                           = "5s"
    success_before_passing            = 2
    failures_before_critical          = 3
    deregister_critical_service_after = "10m"
  }

  connect {
    sidecar_service {
      proxy { mode = "transparent" }
    }
  }
}

Health check types

check { http = "http://localhost:8080/ready" interval = "10s" }   # 2xx pass,
                                                                  # 429 warn
check { tcp  = "localhost:5432" interval = "10s" }                # weak
check { udp  = "localhost:53"   interval = "10s" }
check { grpc = "localhost:9090/svc.Health" grpc_use_tls = true }
check { args = ["/check.sh","/data"] interval = "60s" }           # 0/1/other
check { docker_container_id = "abc" shell="/bin/sh" args=["/h.sh"] }
check { ttl  = "30s" }                                            # app reports
check { alias_service = "orders-db" }                             # CASCADE RISK

curl -X PUT localhost:8500/v1/agent/check/pass/<id> -d 'note'
curl -X PUT localhost:8500/v1/agent/check/warn/<id>
curl -X PUT localhost:8500/v1/agent/check/fail/<id>

Detection time = interval × failures_before_critical.

DNS

   <service>.service.consul                healthy instances, shuffled
   <tag>.<service>.service.consul          filtered by tag
   <service>.service.<dc>.consul           another datacentre
   <node>.node.consul
   <query>.query.consul                    prepared query
   <service>.connect.consul                mesh instances
   <service>.virtual.consul                mesh virtual IP

dig @127.0.0.1 -p 8600 orders.service.consul
dig @127.0.0.1 -p 8600 orders.service.consul SRV      # gives the PORT
dig @127.0.0.1 -p 8600 primary.postgres.service.consul

# systemd-resolved
[Resolve]
DNS=127.0.0.1:8600
Domains=~consul

# dnsmasq
server=/consul/127.0.0.1#8600

UDP caps the answer at ~25 records. The JVM caches DNS forever — set networkaddress.cache.ttl=0.

HTTP API

# discovery — USE THE HEALTH ENDPOINT, not the catalog
curl "localhost:8500/v1/health/service/orders?passing"
curl "localhost:8500/v1/health/service/orders?passing&tag=primary&dc=dc2"
curl "localhost:8500/v1/health/service/orders?filter=Service.Meta.version==\"1.4.2\""
curl "localhost:8500/v1/health/state/critical"          # FIRST in an incident

# catalog — includes UNHEALTHY instances
curl localhost:8500/v1/catalog/services
curl localhost:8500/v1/catalog/service/orders
curl localhost:8500/v1/catalog/nodes
curl localhost:8500/v1/catalog/datacenters

# agent
curl localhost:8500/v1/agent/services
curl localhost:8500/v1/agent/checks
curl localhost:8500/v1/agent/members
curl localhost:8500/v1/agent/metrics?format=prometheus
curl -X PUT localhost:8500/v1/agent/service/register -d @svc.json
curl -X PUT localhost:8500/v1/agent/service/deregister/orders-1
curl -X PUT localhost:8500/v1/agent/maintenance?enable=true\&reason=deploy

# consistency
?stale        any server, scales, works with NO leader     ← discovery
              (the default for DNS)
(none)        the leader, without confirming leadership    ← API default
?consistent   the leader, confirming leadership            ← locks, CAS

X-Consul-Index:       128401     ← feed back for blocking queries
X-Consul-KnownLeader: true
X-Consul-LastContact: 12         ← ms of staleness

Blocking queries

curl -i "localhost:8500/v1/health/service/orders?passing"
# X-Consul-Index: 128401
curl "localhost:8500/v1/health/service/orders?passing&index=128401&wait=5m"

Always send back the index. An index that goes backwards is a reset — start from 0.

KV store

consul kv put   config/orders/timeout 30
consul kv get   config/orders/timeout
consul kv get -detailed config/orders/timeout      # ModifyIndex
consul kv get -recurse config/orders/
consul kv delete -recurse config/orders/
consul kv export config/ > backup.json
consul kv import @backup.json

# check-and-set
consul kv put -cas -modify-index=128401 config/orders/timeout 60
consul kv put -cas -modify-index=0      config/new "v"      # create only

# HTTP — values are base64
curl localhost:8500/v1/kv/config/orders/timeout | jq -r '.[0].Value' | base64 -d
curl -X PUT localhost:8500/v1/txn -d '[{"KV":{"Verb":"cas", ...}}]'   # max 64

512 KB per value. In memory on every server. Every write is a Raft commit. Secrets → Vault. Data → a database. Blobs → object storage.

Sessions and locks

curl -X PUT localhost:8500/v1/session/create -d '{
  "Name": "orders-leader",
  "Checks": ["serfHealth", "orders-ready"],
  "LockDelay": "15s", "TTL": "30s", "Behavior": "release"}'

curl -X PUT "localhost:8500/v1/kv/service/orders/leader?acquire=<session>"
curl -X PUT "localhost:8500/v1/kv/service/orders/leader?release=<session>"
curl -X PUT localhost:8500/v1/session/renew/<id>

consul lock service/orders/leader ./my-process
consul lock -n 3 service/workers ./worker           # semaphore

A lock without a fencing token the resource checks is a strong suggestion. Use ModifyIndex — it only increases: UPDATE r SET data=?, fence=? WHERE fence < ?

Mesh

Kind = "service-defaults"
Name = "orders"
Protocol = "http"                     # REQUIRED for L7 features

Kind = "service-intentions"
Name = "orders"
Sources = [
  { Name = "web", Action = "allow" },
  { Name = "*",   Action = "deny"  },
]

Kind = "service-resolver"
Name = "orders"
DefaultSubset = "stable"
Subsets = {
  stable = { Filter = "Service.Meta.version == \"1.4.2\"" }
  canary = { Filter = "Service.Meta.version == \"1.5.0\"" }
}
Failover = { "canary" = { Targets = [{ ServiceSubset = "stable" }] } }
LoadBalancer = { Policy = "maglev"
                 HashPolicies = [{ Field="header" FieldValue="x-session-id" }] }

Kind = "service-splitter"            # weights MUST sum to 100
Name = "orders"
Splits = [
  { Weight = 95, ServiceSubset = "stable" },
  { Weight = 5,  ServiceSubset = "canary",
    RequestHeaders = { Set = { "x-canary" = "true" } } },
]

Kind = "service-router"              # HTTP only
Name = "orders"
Routes = [
  { Match = { HTTP = { Header = [{ Name="x-beta", Exact="true" }] } }
    Destination = { Service = "orders", ServiceSubset = "canary" } },
  { Match = { HTTP = { PathPrefix = "/" } }
    Destination = { Service = "orders", NumRetries = 3,
                    RetryOnConnectFailure = true } },
]

Order: ROUTER → SPLITTER → RESOLVER.

consul config write orders.hcl
consul config read  -kind service-resolver -name orders
consul config list  -kind service-splitter
consul config delete -kind service-router -name orders

consul intention create -allow web orders
consul intention check web orders
consul connect envoy -sidecar-for orders

# the sidecar is the AUTHORITY
curl localhost:19000/config_dump | jq
curl localhost:19000/clusters
curl localhost:19000/certs
curl localhost:19000/stats | grep upstream_rq_5xx

ACLs

consul acl bootstrap                        # ONCE. Save the SecretID.
consul acl policy create -name p -rules @p.hcl
consul acl role   create -name orders -policy-name p
consul acl token  create -role-name orders
consul acl token  create -service-identity "orders"
consul acl token  create -node-identity "web-1:dc1"
consul acl token  list
consul acl token  update -id anonymous -policy-name dns-read

export CONSUL_HTTP_TOKEN=...
curl -H "X-Consul-Token: ..." https://localhost:8501/v1/kv/x

service "orders"    { policy = "write" }
service_prefix ""   { policy = "read"  }
node_prefix ""      { policy = "read"  }
key_prefix "config/orders/" { policy = "read" }
session_prefix ""   { policy = "write" }
# deny beats write. Resources: agent node service key session event
#   query acl operator mesh peering keyring

Migrate: default_policy = "allow" → collect “Permission denied” logs → fix → "deny". DNS uses the anonymous token — deny-by-default breaks it.

Operations

consul members            consul members -detailed
consul info
consul monitor -log-level=debug
consul rtt web-1 web-2                       # measured latency

consul operator raft list-peers
consul operator raft remove-peer -id=<id>    # REMOVE BEFORE ADD
consul operator raft transfer-leader
consul operator autopilot state              # read Failure Tolerance
consul operator autopilot get-config

consul snapshot save backup.snap             # contains EVERY ACL TOKEN
consul snapshot inspect backup.snap
consul snapshot restore backup.snap

consul maint -enable -service=orders -reason="deploy"
consul maint -disable -service=orders
consul leave                                 # graceful
consul force-leave -prune <node>             # gone for good

consul keyring -install=<new>                # 1
consul keyring -list                         #   VERIFY
consul keyring -use=<new>                    # 2
consul keyring -remove=<old>                 # 3

consul reload
consul validate /etc/consul.d/

Ports

   8300  TCP      server RPC        servers only
   8301  TCP+UDP  LAN gossip        every agent — BOTH PROTOCOLS
   8302  TCP+UDP  WAN gossip        servers only
   8500  TCP      HTTP API + UI
   8501  TCP      HTTPS API
   8502  TCP      gRPC (Envoy xDS)
   8600  TCP+UDP  DNS
  19000  TCP      Envoy sidecar admin

Metrics

consul_autopilot_healthy == 0
consul_autopilot_failure_tolerance == 0        # THE alert
sum(consul_raft_state_leader) != 1             # zero or two leaders
consul_raft_commitTime                          # includes fsync — rises FIRST
consul_raft_leader_lastContact
consul_raft_apply  consul_kvs_apply
consul_serf_member_failed  consul_serf_member_flap
consul_runtime_alloc_bytes
consul_client_rpc_exceeded

Sizing

   servers   small  2 vCPU  8 GB · medium  4/16 GB · large  8/32 GB
             LOCAL SSD — every Raft commit is an fsync
   agents    ~50 MB, negligible CPU

   3 servers → tolerate 1 · 5 → tolerate 2 · NEVER an even number

   DNS cached ~0.2 ms · uncached ~1-2 ms · KV write ~5-20 ms
   reads tens of thousands/s · writes several hundred/s

Failure detection times

   clean shutdown (deregister on SIGTERM)      ~100 ms, no lost requests
   process crash (interval × failures)         ~30 s
   machine death (gossip)                      ~5 s
   leader failure (raft_multiplier = 1)        ~1 s
                  (raft_multiplier = 5)        ~5 s

Fifteen rules

1. raft_multiplier = 1 in production. The default of 5 is for laptops.
2. Three or five servers. Never an even number.
3. Local SSD for servers. Every commit is an fsync.
4. id must be unique per node — it defaults to name.
5. Use /v1/health/service/X?passing, not the catalog endpoint.
6. Add hysteresis to checks, and set a timeout above your real p99.
7. Set deregister_critical_service_after, longer than your worst legitimate outage.
8. Never let one health endpoint call another service’s.
9. Deregister on SIGTERM. ~100 ms and no lost requests.
10. consul keyring -list between every rotation step.
11. verify_server_hostname = true, and enable_script_checks = false.
12. Store the bootstrap token before doing anything else.
13. Remove a dead server before adding its replacement.
14. A lock needs a fencing token the resource checks, or it is a suggestion.
15. A dead address that persists is caching, not Consul. Compare dig @127.0.0.1 -p 8600 with dig.

---

That is Consul, and the twenty-second guide on this site. The next one is MinIO — object storage, and where all the data these systems point at actually lives.