11. DNS as service discovery
DNS · 11. DNS as service discoveryDNS is the oldest service discovery mechanism in existence and the most widely deployed, and it is used for two quite different jobs that are worth separating: choosing between healthy endpoints, and finding out what the endpoints are at all.
WHAT DNS IS GOOD AT a stable name for a changing set of addresses coarse geographic steering surviving the loss of a whole region working for every client ever written
WHAT DNS IS BAD AT fast failover — TTLs and caches (chapter 6) per-request decisions weighted distribution with any accuracy knowing anything about a client's actual connectionRound-robin, and its honest limits
Section titled “Round-robin, and its honest limits” example.com. 60 IN A 192.0.2.1 example.com. 60 IN A 192.0.2.2 example.com. 60 IN A 192.0.2.3
THE RESOLVER usually rotates the order per response. THE CLIENT usually takes the first. so load spreads — roughly. WHY THE DISTRIBUTION IS NEVER EVEN
− one resolver serves thousands of users, and it rotates per QUERY, not per user. a cached answer is handed out in the SAME order until it expires. − glibc's getaddrinfo re-sorts by RFC 6724, which can undo the rotation entirely. − browsers cache their own choice. − connection pools resolve once and keep the connection forever. − a dead address stays in the set until somebody edits the zone.The last line is the decisive one: round-robin DNS has no health checking. One of three servers failing means a third of new connections fail, for as long as it takes a human to notice. That is the gap every other technique in this chapter exists to close.
dig +short example.com Afor i in 1 2 3; do dig +short @1.1.1.1 example.com A | tr '\n' ' '; echo; donegetent ahostsv4 example.comHealth-checked DNS failover
Section titled “Health-checked DNS failover” THE MANAGED-PROVIDER FEATURE
1. the provider probes your endpoints from multiple locations 2. an endpoint that fails N checks is REMOVED from the answer set 3. the TTL is kept very low — 30 or 60 s
TIME TO RECOVERY detection 10–60 s + answer change ~0 s + TTL expiry 30–60 s + client caches 0 s to forever ───────────────────────────────── REALISTIC 1–3 minutes, with a long tail SO DNS FAILOVER IS A REGIONAL-DISASTER TOOL, NOT AN INSTANCE-FAILURE TOOL.
FOR INSTANCE FAILURE, USE A LOAD BALANCER: one stable address, health checks in milliseconds, and no cache anywhere in the path.This is the division of labour worth internalising. A load balancer handles “this backend is down” in under a second behind an address that never changes. DNS handles “this entire site is gone”, which is exactly the case a load balancer cannot help with because the load balancer is in it.
Geographic and latency steering
Section titled “Geographic and latency steering” GEODNS answer differs by the resolver's location (or by ECS, if sent). eu users → 192.0.2.1 us users → 198.51.100.1
LATENCY-BASED the provider measures real RTT from networks to regions and answers with the fastest. Route 53's latency-based routing.
WEIGHTED answer A 80% of the time, B 20%. used for canaries and gradual migrations.
MULTI-VALUE + HEALTH CHECKS return up to 8 healthy addresses and let the client choose.Anycast: the better answer, usually
Section titled “Anycast: the better answer, usually” INSTEAD OF DIFFERENT ANSWERS PER REGION, GIVE EVERYONE THE SAME ADDRESS AND LET ROUTING DECIDE.
example.com. A 192.0.2.10 announced from Frankfurt, Ashburn, Singapore, São Paulo
+ no TTL involved. no cache to wait for. + failover is a BGP withdrawal: seconds, globally, with no client cooperation + one record to manage − needs your own AS and address space, or a provider who has them − TCP sessions can break if routes flap mid-connectionEvery CDN and every large DNS operator is anycast underneath. The reason it beats GeoDNS is that it removes caching from the failover path entirely: withdrawing a route takes effect in seconds for everybody, while changing a DNS answer takes effect whenever each cache feels like it.
SRV, and discovery proper
Section titled “SRV, and discovery proper” ROUND-ROBIN TELLS YOU WHERE. SRV TELLS YOU WHERE AND ON WHAT PORT, WITH WEIGHTS.
_etcd-server._tcp.example.com. SRV 0 10 2380 etcd1.example.com. 0 10 2380 etcd2.example.com. 0 10 2380 etcd3.example.com.dig +short _etcd-server._tcp.example.com SRVdig +short _ldap._tcp.dc._msdcs.example.com SRVdig +short _postgres._tcp.service.consul SRV @127.0.0.1 -p 8600 WHO ACTUALLY USES IT Active Directory — entirely dependent Kerberos, LDAP, XMPP, SIP etcd's discovery bootstrap Consul Kubernetes headless services Minecraft, of all things
WHO DOES NOT HTTP. no browser implements SRV. HTTPS/SVCB is the belated fix.Kubernetes
Section titled “Kubernetes” EVERY SERVICE GETS A NAME, SERVED BY CoreDNS.
<svc>.<ns>.svc.cluster.local
ClusterIP SERVICE A record → the virtual IP. one address, load-balanced by kube-proxy or eBPF, NOT by DNS.
HEADLESS SERVICE (clusterIP: None) A records → EVERY pod IP. this is the DNS-as-discovery case.
STATEFULSET POD <pod>.<svc>.<ns>.svc.cluster.local → a stable per-pod name, which is the whole reason StatefulSets exist.
SRV RECORDS _<port-name>._<proto>.<svc>.<ns>.svc… → port numbers, for clients that ask.kubectl run -it --rm dnsutils --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3 -- sh# then:# dig +short my-svc.default.svc.cluster.local# dig +short -t srv _http._tcp.my-svc.default.svc.cluster.localkubectl -n kube-system get configmap coredns -o yamlkubectl -n kube-system logs -l k8s-app=kube-dns --tail=20 THE THREE KUBERNETES DNS PROBLEMS
1. ndots:5 — chapter 3. five wasted NXDOMAINs per external lookup.
2. CoreDNS AS A SINGLE POINT OF LOAD. every pod's every lookup. the fix is NodeLocal DNSCache: a per-node caching resolver so most queries never leave the node.
3. CONNTRACK AND UDP. a famous race in the Linux conntrack table caused ~1-in-10,000 DNS timeouts of exactly 5 seconds under load. mitigations: `single-request-reopen`, NodeLocal DNSCache, or TCP.The five-second timeout is worth recognising on sight. A latency histogram with a spike at exactly 5,000 ms and nothing between 50 ms and 5 s is a DNS retry after a dropped UDP packet, every time, and it is almost always this.
Consul, and the service-registry model
Section titled “Consul, and the service-registry model” CONSUL REGISTERS SERVICES, HEALTH-CHECKS THEM, AND EXPOSES THE HEALTHY SET OVER DNS.
<service>.service.<dc>.consul A/SRV <tag>.<service>.service.consul <service>.query.consul prepared queries
THE DIFFERENCE FROM PLAIN DNS: UNHEALTHY INSTANCES ARE NOT RETURNED. the registry knows, because it is checking.dig @127.0.0.1 -p 8600 +short web.service.consuldig @127.0.0.1 -p 8600 +short -t srv web.service.consulconsul catalog servicesConsul’s DNS interface is the clearest illustration of the pattern: DNS as a read interface onto a registry that has real health information, rather than as the registry itself. The same shape appears in CoreDNS’s Kubernetes plugin and in external-dns writing cloud DNS records from Kubernetes objects.
external-dns, and closing the loop
Section titled “external-dns, and closing the loop”The pattern that ties this chapter to chapter 12 is a controller that writes public DNS records from cluster state.
AN INGRESS OR SERVICE OBJECT annotation: external-dns.alpha. kubernetes.io/hostname=api.example.com │ ▼ external-dns CONTROLLER watches the API, computes the desired record set │ ▼ YOUR DNS PROVIDER'S API Route 53, Cloudflare, Google, RFC 2136kubectl annotate ingress api external-dns.alpha.kubernetes.io/hostname=api.example.com external-dns.alpha.kubernetes.io/ttl=60kubectl -n external-dns logs deploy/external-dns --tail=20dig +short api.example.com WHAT TO GET RIGHT − a TXT OWNERSHIP RECORD, so the controller only touches records it created. without it, a misconfiguration deletes records somebody else owns. − RESTRICT THE API CREDENTIAL to one zone. − `--policy=upsert-only` while you are learning, so nothing is deleted. − DO NOT let it manage the apex if you depend on a provider ALIAS record.The ownership TXT record is the detail worth insisting on. It is how the controller distinguishes “a record I made and should reconcile” from “a record a human made and must not touch”, and it is the difference between automation and an outage.
Weighted and canary routing, honestly
Section titled “Weighted and canary routing, honestly” example.com. A 192.0.2.1 weight 90 example.com. A 192.0.2.2 weight 10
WHAT YOU ASKED FOR: 10% of traffic to the canary. WHAT YOU GET: ~10% of RESOLUTIONS, each of which serves an unknown and unequal number of requests for the length of a TTL.
A resolver serving 50,000 users that happens to cache the canary answer sends all 50,000 to it.For a canary that must be precise, the decision belongs at a layer that sees requests — an Envoy route with weighted clusters or an Istio VirtualService — and DNS should point at one stable entry point. Weighted DNS is for coarse, long-running splits: a migration between two providers, or a region drain.
What to take from this chapter
Section titled “What to take from this chapter”- DNS does two different jobs: choosing between healthy endpoints and finding out what the endpoints are. It is much better at the second.
- Round-robin has no health checking. One of three addresses failing breaks a third of new connections until a human edits the zone. Distribution is also never even, because resolvers rotate per query while serving many users, and
getaddrinfore-sorts anyway. - Health-checked DNS failover realistically takes one to three minutes, with a long tail from client caches. It is a regional-disaster tool, not an instance-failure tool — that is a load balancer’s job, behind an address that never changes.
- Geographic steering resolves the resolver’s location, not the user’s, unless ECS is sent — and Cloudflare deliberately does not send it.
- Anycast beats GeoDNS for failover because it removes caching from the path: a BGP withdrawal is global in seconds with no client cooperation.
SRVadds a port and weights and is used by Active Directory, Kerberos, LDAP, etcd, Consul and Kubernetes — and by no browser, which is whatHTTPS/SVCBfinally addresses.- In Kubernetes, a ClusterIP service’s A record is a virtual IP load-balanced by kube-proxy, not by DNS; a headless service returns every pod IP; and a StatefulSet’s stable per-pod names are the reason it exists.
- The three Kubernetes DNS problems are
ndots:5, CoreDNS as a central load (fixed with NodeLocal DNSCache), and the conntrack race producing exactly-5-second timeouts — recognisable on sight in a latency histogram. - Consul and CoreDNS show the right pattern: DNS as a read interface onto a registry with real health information, not as the registry itself.
Next: operating a zone — registrars, the apex problem in practice, zone-as-code, and the changes that take a company off the internet.