14. Failure modes

Every failure below has taken a real site off the internet. They are grouped by what you see first.

“It works for me”

   SYMPTOM   some people can reach the site,
             some cannot, and both are
             certain.

   CAUSE     a cache. almost always.
             different resolvers hold
             different answers with
             different remaining TTLs.

# the truth, from your own authority
dig +short @ns1.example.com www.example.com
# what the world sees
for r in 1.1.1.1 8.8.8.8 9.9.9.9 208.67.222.222 64.6.64.6; do
  printf '%-16s %s\n' "$r" "$(dig +short @$r www.example.com | tr '\n' ' ')"
done
# how old is the cached answer?
dig @1.1.1.1 www.example.com | awk '/^www/ {print "TTL remaining:", $2}'

The remaining-TTL trick settles the argument in one command: if a record configured at 300 shows 47, the resolver cached it 253 seconds ago and will not ask again for 47 more. Nothing is broken; the change is simply not yet visible there.

SERVFAIL

   SERVFAIL MEANS TWO COMPLETELY DIFFERENT
   THINGS (chapter 7).

   THE ONE-COMMAND TEST

dig example.com              # SERVFAIL
dig +cd example.com          # works? → DNSSEC is the problem
dig +dnssec example.com | grep -i 'EDE'   # and here is why
delv example.com             # a full validation trace

   IF +cd FIXES IT: DNSSEC.
     − a DS at the parent with no matching
       DNSKEY
     − expired RRSIGs
     − a key rolled without a DS update
     − an algorithm the resolver rejects
     → chapter 9

   IF +cd DOES NOT FIX IT: REACHABILITY.
     − every nameserver is down or
       unreachable
     − the delegation points at servers
       that do not serve the zone
     − a firewall is blocking TCP/53 and
       the answer needs it
     − the resolver cannot reach the
       parent either

Danger

The DNSSEC case is the most severe failure in this guide because it is total and self-inflicted. Every validating resolver — including 8.8.8.8 and 1.1.1.1, so 30–40% of global queries — withholds the answer entirely. The domain does not resolve. And the fix has to propagate through the parent’s TTL, which for .com is two days on the DS record.

“The record is there and still says NXDOMAIN”

   SYMPTOM   you created it, dig on the
             authority shows it, and your
             own machine insists it does
             not exist.

   CAUSE     NEGATIVE CACHING. somebody —
             probably you, testing — looked
             it up before it existed, and
             the SOA's `minimum` is 86400.

dig +short example.com SOA | awk '{print $7}'   # the minimum field
dig @ns1.example.com +short new.example.com     # authority: present
dig @1.1.1.1 +short new.example.com             # resolver: nothing
sudo resolvectl flush-caches                    # your machine only

The only real fix is to wait, and the prevention is a sane minimum of 300–900 seconds. This is the strongest argument for setting that field correctly, because the person it inconveniences most is always the engineer making the change.

Exactly 5,000 ms

   SYMPTOM   a latency histogram with a
             normal p50 and a spike at
             precisely 5 seconds, with
             nothing in between.

   CAUSE     a lost UDP query or reply, and
             the stub's 5-second default
             timeout before it retries.

   WHY IT IS LOST
     − Linux conntrack race on parallel
       A/AAAA queries from the same socket
       (the classic Kubernetes bug)
     − UDP buffer overflow on a busy
       resolver
     − a middlebox dropping the second of
       two in-flight queries
     − genuine packet loss

grep -E 'timeout|single-request' /etc/resolv.conf
conntrack -S | grep -E 'insert_failed|drop'
ss -ulnp | grep ':53'
netstat -su | grep -iE 'receive errors|buffer'

   FIXES, IN ORDER
     1. a LOCAL CACHING RESOLVER. the
        query never leaves the host.
     2. options single-request-reopen
     3. options timeout:1 attempts:2
     4. NodeLocal DNSCache, in Kubernetes
     5. use TCP (use-vc), as a last resort

“The domain vanished entirely”

   SYMPTOM   total, global, and it got worse
             over the first hour rather than
             starting bad.

   THE GRADUAL ONSET IS THE TELL: caches
   expiring one by one.

   CAUSES
     1. NS records at the registrar point
        somewhere wrong
     2. all authoritative servers down
     3. DNSSEC broken (above)
     4. the domain expired
     5. the zone's SOA EXPIRE elapsed on
        every secondary

whois example.com | grep -iE 'name server|status|expir'
dig +norecurse @a.gtld-servers.net example.com NS
for ns in $(dig +short @a.gtld-servers.net example.com NS); do
  echo -n "$ns: "; dig +norecurse +short @"$ns" example.com SOA || echo FAIL
done

The registry’s view is the one that matters, and dig +norecurse at the TLD servers is how you get it without a cache in the way. If the parent lists nameservers that answer REFUSED, nothing you do in your own zone can help.

“Mail stopped being delivered”

   MAIL HAS MORE DNS DEPENDENCIES THAN
   ANYTHING ELSE, AND EACH FAILS QUIETLY.

   MX missing / CNAME'd
     → inbound bounces
   SPF with two TXT records
     → permerror → rejected as spam
   SPF over 10 lookups
     → permerror → same
   DKIM selector record deleted
     → signatures fail
   DMARC p=reject with broken SPF/DKIM
     → YOUR OWN MAIL IS REJECTED
   PTR missing for the sending IP
     → greylisted or refused

dig +short example.com MX
dig +short example.com TXT | grep spf1 | wc -l    # must be 1
dig +short _dmarc.example.com TXT
dig -x 203.0.113.9 +short
# count SPF lookups
dig +short example.com TXT | grep -o 'include:[^ ]*' | wc -l

The DMARC case deserves emphasis: publishing p=reject before SPF and DKIM are both verified-working means you have instructed the world to discard your own email. Always deploy p=none with rua= reporting first, read the reports for a few weeks, then tighten.

“Certificate renewal failed”

   ACME DNS-01 PUTS YOUR DNS PROVIDER IN
   THE CERTIFICATE PATH.

   FAILURES
     − the API token expired or lost
       permissions
     − the TXT record's TTL is so high that
       a previous challenge value is still
       cached
     − CAA records do not list the CA you
       are using
     − the provider rate-limited the
       automation
     − a wildcard cert, which can ONLY use
       DNS-01

dig +short _acme-challenge.example.com TXT
dig +short example.com CAA
certbot renew --dry-run

A CAA record that lists the wrong issuer produces a refusal at the CA with a message about policy, not about DNS, which is why this one is reliably misdiagnosed.

“An internal name resolves externally, or not at all”

   SYMPTOM   a laptop works on the office
             network and not on a VPN, or
             the reverse.

   CAUSES
     − split-horizon drift (chapter 12)
     − the client is using DoH to a public
       resolver, which has no idea your
       internal zone exists
     − VPN split-tunnelling chose the wrong
       resolver for this route
     − a search-domain change

resolvectl status                  # per-link resolvers and domains
resolvectl query db.internal
grep -E 'DNSOverTLS|DNS=' /etc/systemd/resolved.conf
dig use-application-dns.net        # is the Firefox canary blocked?

“A subdomain is serving somebody else’s content”

   SUBDOMAIN TAKEOVER (chapter 12).

   CONFIRM
     the CNAME target does not resolve, or
     resolves to a provider's shared
     infrastructure that anybody can claim.

dig +short blog.example.com CNAME
dig +short "$(dig +short blog.example.com CNAME)" A || echo "DANGLING"
curl -sI https://blog.example.com | head -5

Fix by deleting the record, not by re-creating the service. The record is the vulnerability.

“Everything resolves, but slowly, for one application”

   SYMPTOM   one service's latency is high;
             other things on the same host
             are fine.

   CAUSES
     − the application resolves on every
       request instead of caching or
       pooling
     − it uses a library that ignores
       /etc/resolv.conf (Go's pure-Go
       resolver behaves differently from
       cgo's)
     − it resolves a name with a CNAME chain
       crossing several zones
     − it is asking for a type nobody
       answers quickly (SRV, or AAAA into a
       black hole)

sudo tcpdump -ni any port 53 | grep -c example.com   # how often?
GODEBUG=netdns=2 ./app 2>&1 | head                   # Go: which resolver?
dig +short www.example.com CNAME                     # chain depth
dig +trace +nodnssec www.example.com | grep -c CNAME

The Go case is worth singling out because it is invisible in code: Go chooses between its pure-Go resolver and cgo’s getaddrinfo at runtime based on the contents of /etc/resolv.conf and nsswitch.conf, and the two have different timeout, search and caching behaviour. GODEBUG=netdns=2 prints which one is in use, and netdns=cgo or netdns=go forces it.

“It broke right after a provider change”

   THE THREE THINGS THAT SURVIVE A
   MIGRATION BADLY

   1. PROVIDER-SPECIFIC RECORDS. an ALIAS
      or flattened apex does not export.
      the new zone has no apex record and
      nobody noticed because the old one
      was still being served.

   2. DNSSEC. the DS at the parent points
      at the old provider's key. total
      outage for validating resolvers.

   3. THE OLD PROVIDER DELETED TOO SOON.
      resolvers holding the old delegation
      get REFUSED.

dig +short example.com                        # is there an apex record?
dig +norecurse @a.gtld-servers.net example.com DS
dig +short @old-ns1.example.net example.com SOA || echo "old provider gone"

A diagnostic order

   1. IS IT A CACHE?
      query the authoritative server
      directly by name. if that is right
      and resolvers are wrong, you are
      waiting on a TTL. stop.

   2. IS IT DNSSEC?
      dig +cd. one command.

   3. IS THE DELEGATION RIGHT?
      dig +norecurse at the TLD servers.
      compare with your own zone's NS set.

   4. DO ALL NAMESERVERS AGREE?
      the SOA serial loop. a mismatch is a
      failing transfer.

   5. IS IT THE CLIENT?
      /etc/resolv.conf, ndots, nsswitch,
      /etc/hosts, resolvectl status, the
      application's own cache (the JVM).

   6. IS IT THE NETWORK?
      TCP/53 open? UDP replies arriving?
      EDNS size causing fragmentation?
      dig +tcp, dig +bufsize=512.

   7. IS IT THE REGISTRAR?
      whois. expiry, status, nameservers,
      DS.

# the one-screen triage
D=example.com
dig +short @ns1.$D $D SOA
dig +cd +short $D
dig +norecurse @a.gtld-servers.net $D NS
for ns in $(dig +short $D NS); do printf '%-26s %s\n' "$ns" \
  "$(dig +short @"$ns" $D SOA | awk '{print $3}')"; done
whois $D | grep -iE 'status|expir|name server'

Note

Steps 1 and 2 resolve the large majority of real incidents and cost two commands between them. The discipline worth building is to run both before forming a theory, because a cache and a DNSSEC failure look identical from a user’s description and nothing alike from dig.

What to take from this chapter

• “It works for me” is a cache, nearly always. Query your authority directly, then several public resolvers, and read the remaining TTL to see how old a cached answer is.
• SERVFAIL splits with one command: dig +cd. If that works, it is DNSSEC; if not, it is reachability or delegation. Check for Extended DNS Errors before guessing further.
• A broken DNSSEC chain is total and self-inflicted, affects 30–40% of global queries, and the DS fix must wait out the parent’s TTL — two days for .com.
• “The record exists but says NXDOMAIN” is negative caching governed by the SOA’s minimum. Set it to 300–900; the person it hurts most is the engineer making the change.
• Exactly 5,000 ms in a latency histogram is a lost UDP packet and a stub retry — the conntrack race, a full buffer, or real loss. A local caching resolver is the fix.
• A domain that vanished gradually over an hour is caches expiring: check the registry’s NS view with dig +norecurse at the TLD servers, not your own zone’s.
• Mail has the most DNS dependencies and they fail quietly: a CNAME’d MX, two SPF records, the ten-lookup limit, a deleted DKIM selector, a missing PTR, and p=reject deployed before SPF and DKIM work — which instructs the world to discard your own mail.
• ACME DNS-01 puts your DNS provider’s API in the certificate path, and a wrong CAA record fails at the CA with a message about policy rather than DNS.
• Internal names failing on a VPN or off-network is split-horizon drift or a client using DoH to a public resolver.
• A subdomain serving somebody else’s content is a dangling CNAME. Delete the record; it is the vulnerability.
• Run step 1 and step 2 before forming a theory. Two commands, and they resolve most incidents.

Next: choosing a setup — managed versus self-hosted, one provider or two, and how much of this to adopt.