Skip to content
Toolforge

7. Service discovery

Two interfaces, and the choice is not obvious.

   DNS       works in every language with zero code
             loses ports (unless SRV), caches badly,
             no way to learn about a change


   HTTP      full information, blocking queries for push-style
             updates, filtering
             needs a client, or a template, or a sidecar

DNS

Section titled “DNS”
Terminal window
dig @127.0.0.1 -p 8600 orders.service.consul
;; ANSWER SECTION:
orders.service.consul.  0  IN  A  10.0.1.11
orders.service.consul.  0  IN  A  10.0.1.12
orders.service.consul.  0  IN  A  10.0.1.13

Only healthy instances are returned. Order is randomised on each response. So a client that resolves and connects to the first answer gets crude round-robin load balancing for free, which is most of what many applications need.

The name forms:

   <service>.service.consul                 healthy instances
   <tag>.<service>.service.consul           filtered by tag
   <service>.service.<dc>.consul            in another datacentre
   <node>.node.consul                       one node's address
   <query>.query.consul                     a prepared query
   <service>.connect.consul                 mesh-enabled instances
   <service>.virtual.consul                 the mesh virtual IP

SRV records, and the port problem

Section titled “SRV records, and the port problem”

An A record gives you an address and no port. If your service is not on a well-known port, you need SRV:

Terminal window
dig @127.0.0.1 -p 8600 orders.service.consul SRV
;; ANSWER SECTION:
orders.service.consul. 0 IN SRV 1 1 8080 web-1.node.dc1.consul.
orders.service.consul. 0 IN SRV 1 1 8081 web-1.node.dc1.consul.
orders.service.consul. 0 IN SRV 1 1 8080 web-2.node.dc1.consul.


;; ADDITIONAL SECTION:
web-1.node.dc1.consul. 0 IN A 10.0.1.11
web-2.node.dc1.consul. 0 IN A 10.0.1.12

Very little uses SRV records. Browsers do not. Most HTTP client libraries do not. Kubernetes does for some things. In practice you either fix your ports by convention, or you use the HTTP API.

The DNS trap: caching

Section titled “The DNS trap: caching”

This is the thing that quietly betrays people, and it is worth being blunt about.

   Consul returns TTL 0 — "do not cache this"


   But between Consul and your application there may be:
     the OS resolver cache (systemd-resolved, nscd)
     the JVM's own DNS cache
     a corporate DNS forwarder
     a container runtime's resolver
     the application's HTTP client, holding a connection


   Any of them may ignore the TTL. Then a dead instance
   is still being connected to, minutes later.

The JVM is the notorious case. Java’s default networkaddress.cache.ttl with a security manager installed is -1 — cache forever. An application resolves orders.service.consul at startup and never resolves again for the life of the process.

java.security.Security.setProperty("networkaddress.cache.ttl", "0");
java.security.Security.setProperty("networkaddress.cache.negative.ttl", "0");

Or -Dsun.net.inetaddr.ttl=0. If you use DNS discovery from a JVM and have not set this, your discovery is not working, and it will look like it works because the addresses were right when the process started.

You can also ask Consul to hand out a non-zero TTL, which at least makes caching bounded and predictable:

dns_config {
  service_ttl = {
    "*"      = "5s"
    "orders" = "0s"
  }
  node_ttl  = "30s"
  use_cache = true
  cache_max_age = "10s"
}

A small non-zero TTL is often better than zero, because zero invites resolvers to do whatever they like, while a 5-second TTL is a number they mostly honour.

Forwarding a real domain to Consul

Section titled “Forwarding a real domain to Consul”

Nobody wants to configure applications to use port 8600.

# systemd-resolved: /etc/systemd/resolved.conf.d/consul.conf
[Resolve]
DNS=127.0.0.1:8600
Domains=~consul
# dnsmasq: /etc/dnsmasq.d/10-consul
server=/consul/127.0.0.1#8600
# CoreDNS in Kubernetes
consul {
    errors
    cache 30
    forward . 10.0.1.5:8600
}

Consul also listens on port 53 directly if you let it bind a privileged port, but forwarding from the system resolver is the normal arrangement.

The .addr. form for containers

Section titled “The .addr. form for containers”
Terminal window
dig @127.0.0.1 -p 8600 orders.service.consul ANY

Where a service’s address is an IP the node does not own — a container with its own IP — Consul returns a CNAME to a synthetic <hex>.addr.dc1.consul name that resolves to it. Resolvers that do not follow CNAMEs in the same response will fail here, which is a rare but genuinely confusing failure.

The HTTP API

Section titled “The HTTP API”
Terminal window
curl "http://localhost:8500/v1/health/service/orders?passing" | jq '.[] | {
  node: .Node.Node,
  address: .Service.Address // .Node.Address,
  port: .Service.Port,
  tags: .Service.Tags,
  meta: .Service.Meta
}'

Everything DNS gives you, plus ports, tags, metadata, per-check output, and filtering. This is the correct interface for anything you are writing yourself.

Blocking queries — the good part

Section titled “Blocking queries — the good part”

The HTTP API can hold a request open until something changes.

Terminal window
# first request: note the index
curl -i "http://localhost:8500/v1/health/service/orders?passing"
# X-Consul-Index: 128401


# second request: return only when the index changes, or after 5 minutes
curl "http://localhost:8500/v1/health/service/orders?passing&index=128401&wait=5m"
   POLLING                          BLOCKING QUERY


   ask every 5 seconds              ask once, wait
   5 seconds of staleness           notified in milliseconds
   constant load                    one idle connection
   most requests return             returns only on change
     the same answer

This is how every Consul-aware tool works — consul-template, Envoy’s xDS integration, the Nomad and Kubernetes syncs. A change to a service propagates to every watcher in milliseconds rather than at the next poll.

Two rules for using them correctly:

Always send back the index you were given, and treat an index that goes backwards as a reset — start over from index 0. This happens after a leader change or a restore.

Add jitter. Consul already jitters the return of blocking queries by up to 1/16 of wait, but if you have a thousand watchers on one key, add your own on top or you get a thundering herd on every change.

Watches

Section titled “Watches”

Consul can run a command or POST to an endpoint when something changes, so you do not have to write the blocking-query loop:

watch {
  type    = "service"
  service = "orders"
  handler_type = "http"
  http {
    path   = "http://localhost:9000/reload"
    method = "POST"
  }
}

Types: key, keyprefix, services, nodes, service, checks, event.

consul-template

Section titled “consul-template”

The tool that connects Consul to everything that has a configuration file.

/etc/consul-template/haproxy.ctmpl
backend orders
    balance leastconn
{{ range service "orders|passing" }}
    server {{ .Node }}-{{ .Port }} {{ .Address }}:{{ .Port }} check
{{ end }}
template {
  source      = "/etc/consul-template/haproxy.ctmpl"
  destination = "/etc/haproxy/haproxy.cfg"
  command     = "haproxy -c -f /etc/haproxy/haproxy.cfg && systemctl reload haproxy"
}

This is the single most useful integration pattern in Consul, and it is how you get dynamic backends into software that was never designed for them. A blocking query notices the change, the template is rendered, the configuration is validated, and the process reloads — in seconds, with no human involved. Chapter 4 of the HAProxy guide covers what a server line can carry; consul-template writes them for you.

Note the && in the command. Rendering an invalid configuration and reloading into it is how a template turns a routine scale-up into an outage. Always validate before reloading.

The same pattern drives nginx upstreams, Prometheus scrape configs, and application configuration files.

Prepared queries

Section titled “Prepared queries”

A named, server-side query with logic in it — the main thing DNS alone cannot express.

Terminal window
curl -X POST http://localhost:8500/v1/query -d '{
  "Name": "orders-with-failover",
  "Service": {
    "Service": "orders",
    "OnlyPassing": true,
    "Failover": {
      "NearestN": 2,
      "Datacenters": ["dc2", "dc3"]
    }
  },
  "DNS": {"TTL": "10s"}
}'
Terminal window
dig @127.0.0.1 -p 8600 orders-with-failover.query.consul
   healthy instances in this datacentre?   → return them
   none?                                    → try the 2 NEAREST
                                              datacentres, by the
                                              network coordinates
                                              from chapter 3
   still none?                              → try dc2, then dc3

Automatic cross-datacentre failover, through a DNS name. An application that knows nothing about datacentres gets regional failover for free.

NearestN uses Consul’s network tomography, so “nearest” means measured round-trip time rather than a list somebody wrote down and stopped updating.

Prepared queries can also template, so one definition covers every service:

Terminal window
curl -X POST http://localhost:8500/v1/query -d '{
  "Name": "",
  "Template": {"Type": "name_prefix_match", "Regexp": "^geo-(.*)$"},
  "Service": {
    "Service": "${match(1)}",
    "Failover": {"NearestN": 3}
  }
}'

Now geo-orders.query.consul and geo-payments.query.consul both work, with failover, from one definition.

Choosing an interface

Section titled “Choosing an interface”
   Legacy application, cannot change code          → DNS
   Need the port and it is not conventional        → SRV, or HTTP
   Writing the client yourself                     → HTTP + blocking
   Configuring a proxy, nginx, HAProxy             → consul-template
   Need cross-datacentre failover                  → prepared query
   Need to react within milliseconds               → blocking query
   Running the mesh                                → the sidecar does it

The default recommendation: DNS to get started, consul-template for anything with a configuration file, blocking queries for anything you write.

And the caveat that runs through this chapter: if you use DNS, verify your resolver actually honours the TTL. Test it — resolve, kill an instance, resolve again, and watch how long the dead address persists. If it persists, discovery is not working no matter how healthy the catalog is.

What to take from this chapter

Section titled “What to take from this chapter”
  • DNS returns only healthy instances, in randomised order — free round-robin, in any language, with no code.
  • DNS loses the port unless you use SRV, and very little speaks SRV.
  • Caching between Consul and your application defeats discovery. The JVM caches forever by default; test your resolver rather than assuming.
  • A small non-zero TTL is often better than zero, because resolvers honour it more predictably.
  • Blocking queries turn polling into push and are how every Consul-aware tool works. Send back the index, and treat a backwards index as a reset.
  • consul-template is the highest-value integration — dynamic backends for software that never expected them. Always validate before reloading.
  • Prepared queries give cross-datacentre failover through a DNS name, using measured latency rather than a static list.

Next: a lookup traced end to end — from the DNS query to the answer, and the HTTP version with a blocking query.

Written by Sepuri Sai Krishna

© 2026CC BY 4.0Source