13. Tuning and operations
DNS · 13. Tuning and operationsDNS tuning splits cleanly in two, and the halves have almost nothing in common: the resolver side, where caching and latency live, and the authoritative side, where capacity and abuse live.
"DNS IS SLOW" │ ├── a few queries, occasionally │ → stub timeouts, ndots, a dead │ nameserver in resolv.conf │ ├── all queries, uniformly │ → the resolver is far away, or │ the cache is cold │ ├── the first query for each new name │ → normal. that is a tree walk. │ └── exactly 5000 ms, sometimes → a dropped UDP packet and a retry. conntrack, or loss.The stub, first
Section titled “The stub, first”The cheapest wins in DNS are here, and they are configuration rather than capacity.
/etc/resolv.conf — WHAT TO SET
options timeout:1 attempts:2 rotate single-request-reopen edns0
timeout:1 not 5. a dead server should cost 1 s, not 5. attempts:2 2 × number of nameservers, total. rotate spread across the listed servers instead of always using the first. single-request-reopen A and AAAA on separate sockets. works around a conntrack/NAT race. ndots:1 unless you genuinely need search domains.cat /etc/resolv.confresolvectl statisticssystemd-resolve --statistics 2>/dev/null AND THE BIGGEST WIN: A LOCAL CACHE.
run a caching resolver on every host or every node. systemd-resolved, dnsmasq, Unbound in forwarding mode, NodeLocal DNSCache in Kubernetes.
EFFECT a cache hit is ~0.05 ms instead of ~1 ms over the network the network resolver's load drops by 70–95% a brief resolver outage becomes invisibleResolver tuning
Section titled “Resolver tuning” UNBOUND, THE SETTINGS THAT MATTER
num-threads: <cores> msg-cache-size: 128m rrset-cache-size: 256m ← twice the msg cache, conventionally cache-min-ttl: 0 ← raising this lies to clients. 60 is a defensible compromise. cache-max-ttl: 86400 cache-max-negative-ttl: 3600 prefetch: yes ← refresh popular entries BEFORE they expire prefetch-key: yes ← and their DNSKEYs serve-expired: yes serve-expired-ttl: 86400 edns-buffer-size: 1232 so-rcvbuf: 8m outgoing-range: 8192 harden-dnssec-stripped: yes qname-minimisation: yesunbound-control stats_noreset | grep -E 'num.query|cachemiss|recursion.time'unbound-control dump_cache | wc -lunbound-checkconf THE NUMBER TO WATCH: CACHE HIT RATE.
hit_rate = (total.num.queries - total.num.cachemiss) / total.num.queries
HEALTHY: 80–95% on a busy resolver. BELOW 60%: either the cache is too small, TTLs upstream are very short, or you are seeing random-subdomain attack traffic.Prefetching
Section titled “Prefetching” WITHOUT PREFETCH a popular record expires → the next user to ask PAYS THE FULL RECURSION. one user in every TTL period eats 80 ms.
WITH PREFETCH when a cached entry is used and is in the last 10% of its TTL, the resolver refreshes it in the background. NOBODY pays.It costs a small amount of extra upstream traffic for popular names and removes the periodic latency spike entirely. For a resolver serving interactive users it is close to a free win.
QNAME minimisation
Section titled “QNAME minimisation” THE OLD BEHAVIOUR ask the ROOT for the full name: "www.secret-project.example.com A" the root now knows the whole thing.
RFC 9156: ASK ONLY WHAT IS NEEDED to the root: "com NS" to .com: "example.com NS" to example: "www.secret-project… A"
DEFAULT ON in Unbound, Knot Resolver, BIND 9.14+, and the major public resolvers.It occasionally exposes broken authoritative servers that mishandle NS queries for intermediate names, which is why qname-minimisation-strict exists and why the default is the relaxed variant that falls back.
Authoritative tuning
Section titled “Authoritative tuning” THE SHAPE OF THE PROBLEM IS DIFFERENT: no cache, stateless, and the traffic is adversarial as often as not.
CAPACITY a modern NSD or Knot instance serves 100k–1M queries/second per host for a small zone. the bottleneck is PACKETS, not CPU — so NIC queues, RSS and kernel bypass matter more than zone size.
MEMORY roughly proportional to zone size. a 1M-record zone: 1–3 GB, more if signed.nsd-control stats | grep -E 'num.queries|num.type'knotc stats | grep -E 'request|response'rndc stats && grep -A20 'Incoming Requests' /var/named/data/named_stats.txt RESPONSE RATE LIMITING — ALWAYS ON
# BIND rate-limit { responses-per-second 15; nxdomains-per-second 5; slip 2; ← every 2nd dropped response becomes a truncated one, so legitimate clients retry over TCP window 5; };slip is the detail that makes RRL safe: rather than silently dropping everything above the limit, it periodically returns a truncated response, which a real client follows to TCP and a spoofed reflection target ignores. Without it, RRL penalises legitimate shared resolvers.
The random-subdomain attack
Section titled “The random-subdomain attack” aaaa1.example.com, bbbb2.example.com, cccc3.example.com, …
EVERY QUERY IS A CACHE MISS AT EVERY RESOLVER AND A REAL QUERY TO YOUR AUTHORITATIVE SERVERS.
IT ATTACKS BOTH ENDS AT ONCE: the resolver's cache is useless, and your authority takes the full load.
DEFENCES NSEC/NSEC3 AGGRESSIVE USE (RFC 8198) a validating resolver can prove a whole RANGE of names does not exist, from one cached NSEC record, and answer NXDOMAIN without asking you. RRL with nxdomains-per-second a wildcard, if it suits the zone anycast, to spread the loadRFC 8198 is the elegant one: it is the only mechanism that turns DNSSEC into a performance feature rather than a cost, because a signed zone lets resolvers answer non-existence locally for an entire gap in the namespace.
What to measure
Section titled “What to measure” ON THE RESOLVER cache hit rate 80–95% recursion time, p50/p99 <50 ms / <500 ms SERVFAIL rate <0.1% queries/second outbound query rate (a proxy for cache effectiveness) DNSSEC validation failures
ON THE AUTHORITATIVE SIDE queries/second, by type NXDOMAIN ratio ← a spike means an attack or a broken client response size distribution TCP vs UDP ratio ← rising TCP means truncation RRL drops zone serial per server AXFR success/failure
FROM OUTSIDE resolution time from several regions agreement between NS servers DNSSEC chain validity days until domain expiry# a quick external latency checkfor r in 1.1.1.1 8.8.8.8 9.9.9.9; do echo -n "$r "; dig +noall +stats @$r example.com | grep 'Query time'done# packet-level, on a resolversudo dnstop -l 3 eth0sudo tcpdump -ni any port 53 -c 20Capacity planning, with numbers
Section titled “Capacity planning, with numbers” QUERY VOLUME, ROUGH GUIDES
per interactive user 1–5 q/s while browsing per web page load 10–80 lookups per container, steady state 0.1–2 q/s a Kubernetes node, 50 pods 20–200 q/s a 1,000-person office 500–3,000 q/s peak an authoritative zone for a busy site, per million daily users ~300–2,000 q/s, dominated by TTL choice WHAT TO SIZE FOR
RESOLVER one modest instance handles 10k–50k q/s with a warm cache. run TWO, in different failure domains, and let the stub rotate.
AUTHORITATIVE packets, not queries. a 10 Gb NIC and NSD will saturate long before the zone size matters.
AND SIZE FOR THE ATTACK, NOT THE TRAFFIC: a reflection attack is 10–100× your normal volume, which is why the answer is almost always anycast rather than bigger boxes.Load testing a resolver
Section titled “Load testing a resolver”# the standard tool, replaying a query filednsperf -s 10.0.0.53 -d queries.txt -c 50 -l 60resperf -s 10.0.0.53 -d queries.txt -m 50000# generate a realistic query file from real traffictcpdump -ni any -w dns.pcap port 53tshark -r dns.pcap -T fields -e dns.qry.name -e dns.qry.type | awk 'NF==2 {print $1, $2}' > queries.txtThe detail that makes a load test meaningful is the query mix. A test that asks for the same name a million times measures the cache’s hash table; a test built from captured production queries measures the thing you care about, including the long tail of unique names that forces real recursion.
Logging, without drowning
Section titled “Logging, without drowning” QUERY LOGGING IS THE MOST VALUABLE SECURITY TELEMETRY AVAILABLE AND THE EASIEST TO MAKE UNUSABLE.
dnstap a structured binary log format, supported by Unbound, Knot, BIND, CoreDNS. the right choice. query log plain text, enormous, expensive to write sampling 1-in-N, for volume aggregate only unique names per client per hour — often enough for detection, and far cheaperunbound-control set_option log-queries: yesdnstap -u /var/run/unbound/dnstap.sock -y # live, YAMLkubectl -n kube-system logs -l k8s-app=kube-dns | grep -v NOERRORLogging everything at a resolver serving 20,000 queries per second produces a terabyte a day and nobody reads it. The pattern that works is dnstap into a stream processor, aggregate counts retained for months, and full records retained for days.
Where the latency actually is
Section titled “Where the latency actually is” A REALISTIC BREAKDOWN FOR A WEB PAGE
DNS, warm cache 1 ms × 10 lookups DNS, cold for one domain 80 ms, once TCP handshake 30 ms TLS handshake 60 ms first byte 100 ms+
SO DNS IS ALMOST NEVER THE PROBLEM — UNLESS
− a nameserver is dead and the stub waits 5 s − ndots:5 is adding 5 NXDOMAINs per lookup − a CNAME chain crosses three zones cold − UDP loss is causing 5-second retries − the resolver is on another continentThe practical conclusion is that DNS latency work is removing outliers, not shaving averages. A p50 of 1 ms with a p99.9 of 5,000 ms is the normal pathology, and every item in that list produces exactly that shape.
What to take from this chapter
Section titled “What to take from this chapter”- The two halves — resolver and authoritative — share almost nothing. Classify the symptom before tuning.
- The cheapest wins are in
/etc/resolv.conf:timeout:1,attempts:2,rotate,single-request-reopen, andndots:1unless you need search domains. - A local caching resolver on every host is the highest-leverage change available — 0.05 ms hits, a 70–95% drop in upstream load, and immunity to brief resolver outages.
- Watch the cache hit rate: 80–95% is healthy, below 60% means a small cache, very short upstream TTLs, or random-subdomain attack traffic.
- Prefetch refreshes popular entries in the last 10% of their TTL, removing the periodic spike where one unlucky user pays the full recursion.
- QNAME minimisation stops the root and TLDs seeing your full names, and is now on by default nearly everywhere.
- Authoritative servers are packet-bound, not CPU-bound. RRL should always be on, and
slipis what keeps it safe for legitimate shared resolvers. - Random-subdomain attacks hit both ends at once. RFC 8198 aggressive NSEC use is the elegant defence and the one case where DNSSEC becomes a performance feature.
- Key authoritative metrics: NXDOMAIN ratio (a spike means an attack or a broken client), TCP/UDP ratio (rising means truncation), and serial agreement across every
NS. - DNS latency work is about removing outliers, not averages. The normal pathology is a 1 ms p50 with a 5,000 ms p99.9, and every cause in this chapter produces that exact shape.
Next: failure modes — the incidents DNS actually produces, what each looks like from the outside, and how to confirm it.