5. A lookup traced

Everything in the previous four chapters happens in one call. This chapter follows it, cold cache first, with real numbers.

getaddrinfo("www.example.com", "443", &hints, &res);

   ①  the application's own cache
   ②  nsswitch, /etc/hosts
   ③  the stub resolver builds a query
   ④  UDP to the configured resolver
   ⑤  resolver: cache miss
   ⑥  ask a ROOT server       → referral
   ⑦  ask a .COM server       → referral
   ⑧  ask example.com's server → ANSWER
   ⑨  cache it, return it
   ⑩  the stub sorts and returns

① and ② Before any packet

   APPLICATION-LEVEL CACHES
     a browser keeps its own DNS cache
       (~60 s in Chrome, independent of
        TTL)
     a JVM caches FOREVER by default
       (networkaddress.cache.ttl=-1 when a
        security manager is set; 30 s
        otherwise) — the classic cause of
        a Java service pinned to a dead IP
     Go, Python, Rust: no cache by default
     glibc: no cache, unless nscd is
       running

nscd -g | head -20                      # if installed
resolvectl statistics                   # systemd-resolved's cache
sudo resolvectl flush-caches

   THEN /etc/nsswitch.conf DECIDES THE
   ORDER:

     hosts: files mdns4_minimal
            [NOTFOUND=return] dns myhostname

   `files`  → /etc/hosts. checked FIRST.
   `mdns4`  → .local names, via Avahi
   `dns`    → the stub resolver

A name in /etc/hosts never reaches DNS at all, which is both the oldest debugging trick in existence and the cause of the most frustrating “it works on that machine” reports.

③ The stub builds a query

   www.example.com has 2 dots ≥ ndots:1,
   so it is tried as absolute first.

   QUERY
     ID:      0x8f3a   (random)
     FLAGS:   QR=0 OPCODE=QUERY RD=1
              ← RD: "recurse for me"
     QDCOUNT: 1
     QUESTION: www.example.com. IN A
     OPT RR:   EDNS0, UDP size 1232,
               DO=0 (or 1 if validating)

   AND, SEPARATELY, ONE FOR AAAA.

   getaddrinfo with AF_UNSPEC sends BOTH
   queries. glibc sends them in parallel
   unless `single-request` is set, in which
   case it serialises them — a workaround
   for broken middleboxes that doubles
   latency.

dig +norecurse +noall +question www.example.com
strace -f -e trace=sendto,recvfrom getent hosts www.example.com 2>&1 | head

④ UDP, port 53

   SRC  10.0.0.5:51423  (random source
                         port — entropy
                         against spoofing)
   DST  10.0.0.1:53
   ONE PACKET, ~60 bytes.

   NO HANDSHAKE. NO CONNECTION. the reply
   is matched on {ID, source port,
   question, server address}.

That matching tuple is the whole of classic DNS security, and its weakness is the subject of chapter 9. Sixteen bits of ID plus sixteen of source port is 2^32 guesses, which sounds adequate and was shown in 2008 not to be.

⑤ The resolver: cache miss

   LOOK UP (www.example.com, A, IN)
     → miss
   LOOK UP THE CLOSEST ENCLOSING ZONE IT
   DOES KNOW
     → nothing for example.com
     → nothing for com
     → it has the ROOT. always.

   SO IT STARTS AT THE ROOT. not because
   every lookup does, but because this
   resolver knows nothing else yet.

⑥ The root server

dig +norecurse @a.root-servers.net www.example.com A

   ;; flags: qr  ← NO `aa`. this is a
                   REFERRAL.
   ;; ANSWER: 0, AUTHORITY: 13,
      ADDITIONAL: 27

   ;; AUTHORITY SECTION:
   com.  172800 IN NS a.gtld-servers.net.
   com.  172800 IN NS b.gtld-servers.net.
   …

   ;; ADDITIONAL SECTION:
   a.gtld-servers.net. 172800 IN A
        192.5.6.30
   a.gtld-servers.net. 172800 IN AAAA
        2001:503:a83e::2:30
   …

   NOTE: the root did not look at
   `www.example.com`. it matched the
   LONGEST SUFFIX IT HAS A DELEGATION FOR
   — `com` — and handed over.

   TTL 172800 = 2 DAYS. the resolver will
   not ask a root about `com` again for two
   days, which is why root traffic is a
   rounding error of real query volume.

⑦ The TLD server

dig +norecurse @a.gtld-servers.net www.example.com A

   ;; flags: qr   ← still a referral

   ;; AUTHORITY SECTION:
   example.com. 172800 IN NS ns1.example.com.
   example.com. 172800 IN NS ns2.example.com.

   ;; ADDITIONAL SECTION:
   ns1.example.com. 172800 IN A 192.0.2.1
   ns2.example.com. 172800 IN A 198.51.100.1
        ↑ GLUE. without this the resolver
          would have to resolve
          ns1.example.com, which requires
          asking example.com. a cycle.

⑧ The authoritative server

dig +norecurse @ns1.example.com www.example.com A

   ;; flags: qr aa    ← AUTHORITATIVE
   ;; ANSWER: 1

   ;; ANSWER SECTION:
   www.example.com. 300 IN A 203.0.113.42

   THREE OTHER THINGS IT MIGHT HAVE SAID

   a CNAME chain
     www.example.com. CNAME cdn.example.net.
     → the resolver must now resolve
       cdn.example.net, from the root
       again unless cached. a CNAME to
       another zone is a WHOLE SECOND
       RESOLUTION.

   NXDOMAIN
     ;; flags: qr aa; status: NXDOMAIN
     with an SOA in AUTHORITY, whose
     `minimum` field sets the NEGATIVE
     cache TTL.

   NODATA
     status: NOERROR, ANSWER: 0, plus an
     SOA. "the name exists; this type does
     not."

⑨ Caching, and what gets stored

   FROM ONE LOOKUP THE RESOLVER NOW HOLDS

   com.             NS set    2 days
   a.gtld-servers.net A       2 days
   example.com.     NS set    2 days
   ns1.example.com. A         2 days
   www.example.com. A         300 s

   SO THE NEXT LOOKUP OF
   anything.example.com SKIPS STEPS ⑥ AND
   ⑦ ENTIRELY. the tree walk happens once
   per branch, not once per name.

Note

This is why dig +trace is misleading as a performance model. It deliberately re-walks from the root every time with no cache, which is the rarest case in production. A real resolver’s first query for a new domain costs three round trips; its second costs zero.

⑩ Back to the application

   THE STUB NOW HAS
     A:    203.0.113.42
     AAAA: 2001:db8::2a

   getaddrinfo SORTS THEM BY RFC 6724
     prefer matching scope
     prefer native over tunnelled
     prefer the longest matching prefix
     → and, in practice, prefer IPv6 if
       the host has a global IPv6 address

   RETURNS A LINKED LIST. the application
   tries them in order — or, if it is a
   browser or a modern library, in
   parallel with a head start, which is
   Happy Eyeballs (RFC 8305).

getent ahostsv4 www.example.com
getent ahosts www.example.com
curl -v --resolve www.example.com:443:203.0.113.42 https://www.example.com/

Happy Eyeballs matters because it hides a class of failure: a host with broken IPv6 connectivity still works, because the browser gives IPv6 a 250 ms head start and then races IPv4. The same application in a library without Happy Eyeballs hangs for the full TCP timeout, which is why “the browser works but our service does not” is a recognisable IPv6 symptom.

The same lookup, after a change

Running the trace again immediately after editing a record shows exactly where the staleness lives, which is the most useful version of this exercise.

   YOU CHANGE www.example.com FROM
   203.0.113.42 TO 203.0.113.99.

   ① your authority           NEW, instantly
   ② a resolver that never
     asked                    NEW, on first
                              query
   ③ a resolver holding the
     old record               OLD, until its
                              TTL expires
   ④ a browser that resolved
     a minute ago             OLD, ~60 s
   ⑤ a JVM process            OLD, possibly
                              forever
   ⑥ an open connection pool  OLD, until a
                              connection
                              breaks

dig +short @ns1.example.com www.example.com
dig +short @1.1.1.1 www.example.com
dig @1.1.1.1 www.example.com | awk '/^www/ {print "expires in", $2, "s"}'

Why the AAAA query matters even without IPv6

   getaddrinfo ASKS FOR BOTH. ALWAYS.

   SO A ZONE WITH NO AAAA RECORD STILL
   RECEIVES AN AAAA QUERY FOR EVERY LOOKUP,
   AND MUST ANSWER IT PROPERLY — NODATA,
   NOT NXDOMAIN, NOT A TIMEOUT.

   A SERVER OR MIDDLEBOX THAT DROPS AAAA
   QUERIES INSTEAD OF ANSWERING THEM
   PRODUCES A 5-SECOND STALL ON EVERY
   LOOKUP, VISIBLE ONLY AS LATENCY.

That specific misbehaviour — a firewall or load balancer silently dropping AAAA — was common enough that options single-request and no-aaaa exist as workarounds, and it remains worth testing explicitly whenever name resolution is mysteriously slow but reliable.

The budget, cold and warm

   COLD CACHE, EVERYTHING REMOTE

   stub → resolver              0.5 ms
   resolver → root              15 ms
   resolver → .com              25 ms
   resolver → example.com       40 ms
   resolver → stub              0.5 ms
   ─────────────────────────────────────
   TOTAL                        ~80 ms
   (and the AAAA query runs in parallel)

   WARM, www.example.com CACHED
   stub → resolver → stub        ~1 ms

   WARM BRANCH, NEW HOST IN THE SAME ZONE
   one round trip to example.com  ~40 ms

   LOCAL CACHE ON THE HOST
   ~0.05 ms. no packet leaves.

   THE OPERATIONAL CONCLUSION

   a 300-second TTL on a hot name means
   one in ~300 requests pays 1 ms instead
   of 0.05 ms. the TTL is almost never a
   latency problem.

   WHAT IS a latency problem: a cold
   resolver, a dead nameserver with a
   5-second stub timeout, a CNAME chain
   crossing three zones, or ndots:5.

Watching it happen

Every stage above is observable, and running these side by side once is worth more than re-reading the diagram.

# 1. what the application asks the kernel
strace -f -e trace=network getent hosts www.example.com 2>&1 | grep -E 'sendto|recvfrom'

# 2. the packets, decoded
sudo tcpdump -ni any -s0 port 53 -vv

# 3. the recursion, from the outside
dig +trace +nodnssec www.example.com

# 4. the recursion, from inside the resolver
unbound-control set_option verbosity: 3
journalctl -fu unbound | grep -E 'resolving|query response'

# 5. what the resolver has cached right now
dig +norecurse @1.1.1.1 www.example.com
unbound-control dump_cache | grep example.com

The pairing of (3) and (5) is the instructive one: +trace shows you the work a cold resolver would do, and +norecurse shows you that a warm one does none of it. Almost every wrong intuition about DNS performance comes from having seen only the first.

What to take from this chapter

• Before any packet there are application caches (/etc/hosts, a browser’s own ~60 s cache, and the JVM’s, which caches forever by default) and nsswitch.conf’s ordering.
• getaddrinfo with AF_UNSPEC sends two queries, A and AAAA, normally in parallel. single-request serialises them and doubles latency.
• A query is one UDP packet with no handshake, matched on {ID, source port, question, server} — 32 bits of entropy, which 2008 showed to be insufficient.
• A root server does not look at the full name. It matches the longest suffix it has a delegation for and hands over, with a two-day TTL — which is why root traffic is a rounding error.
• Glue in the ADDITIONAL section is what breaks the cycle of needing example.com to resolve its own nameserver.
• An authoritative answer carries aa. The alternatives are a CNAME chain (which may require a whole second resolution in another zone), NXDOMAIN with an SOA, or NODATA — NOERROR with zero answers.
• One lookup caches the whole branch: the com and example.com NS sets and their glue, for days. The tree walk happens once per branch, not once per name.
• dig +trace is a misleading performance model because it deliberately discards exactly the caching that makes DNS fast.
• getaddrinfo sorts results by RFC 6724, and Happy Eyeballs races IPv6 against IPv4 — which is why a broken-IPv6 host works in a browser and hangs in a library that lacks it.
• Cold lookups cost ~80 ms over three round trips; warm ones cost ~1 ms, and a local cache ~0.05 ms. The TTL is almost never the latency problem — a dead nameserver with a 5-second stub timeout is.

Next: caching and TTL — the bargain in detail, negative caching, and why nothing ever propagates.