6. Caching and TTL
DNS · 6. Caching and TTLThis is the chapter that prevents outages. Everything in it follows from one fact: DNS has no invalidation.
THERE IS NO "FLUSH THE WORLD". NO CALLBACK. NO INVALIDATION MESSAGE. NO WAY TO REACH A CACHE YOU DO NOT OWN.
AN OLD ANSWER LEAVES A CACHE WHEN ITS TTL EXPIRES, AND NOT BEFORE.“Propagation” is not a thing
Section titled ““Propagation” is not a thing” WHAT PEOPLE PICTURE a change spreading outward from your zone to resolvers worldwide, taking "24 to 48 hours".
WHAT ACTUALLY HAPPENS your authoritative servers answer the new value IMMEDIATELY — to anyone who asks them. resolvers holding the OLD value do not ask. they have an answer. they use it until it expires. SO THE TIME TO FULL VISIBILITY IS
max(remaining TTL across every resolver holding the old value) ≤ the TTL that was in effect WHEN THEY CACHED IT
NOT the TTL you just set. the OLD one.The migration recipe
Section titled “The migration recipe” DAY −3 lower the TTL on the records you will change, from 3600 to 60. WAIT at least the OLD TTL (3600 s), ideally 2× it. DAY 0 change the records. visibility is now ~60 s. VERIFY query several public resolvers directly, and your own authoritative set. DAY +1 once stable, raise the TTL back. NEVER change the record and the TTL in the same operation and expect speed.dig +short @ns1.example.com www.example.com # the truthfor r in 1.1.1.1 8.8.8.8 9.9.9.9 208.67.222.222; do echo -n "$r "; dig +short @$r www.example.comdonedig www.example.com | grep -E '^www' # remaining TTLThe last command is the useful habit: the TTL in a resolver’s reply counts down, so a value of 43 on a record configured at 300 means it was cached 257 seconds ago. That is how you measure a cache’s state from outside.
Who caches, and for how long
Section titled “Who caches, and for how long” LAYER OBEYS TTL? ───── ────────── authoritative server does not cache recursive resolver YES, usually, with a floor and a ceiling corporate forwarder usually systemd-resolved yes dnsmasq yes, with its own --max-cache-ttl the stub / glibc no cache at all nscd its OWN ttl, 3600 by default, IGNORING the record a browser its own, ~60 s, ignoring the record the JVM FOREVER by default Go / Rust / Python no cache a connection pool effectively forever — it never re-resolves RESOLVER CLAMPS (and they differ)
Unbound cache-min-ttl: 0 cache-max-ttl: 86400 BIND max-cache-ttl 604800 min-cache-ttl 0 1.1.1.1 floors very low TTLs 8.8.8.8 caps at ~6 hours in practice many ISPs floor at 30–300 s to cut their own query volumeA TTL of 1 is therefore a request, not an instruction. Several large resolvers will floor it to 30 seconds or more, and some corporate forwarders ignore it entirely.
The two caches that ruin migrations
Section titled “The two caches that ruin migrations” 1. A CONNECTION POOL. it resolved once, at startup, and holds TCP connections forever. a DNS change is invisible to it until a connection breaks. a JDBC pool, an HTTP keep-alive pool, a gRPC channel.
2. THE JVM. networkaddress.cache.ttl defaults to caching successful lookups indefinitely under a security manager, and 30 s otherwise — and plenty of deployed code sets it to -1 on purpose.// in code, or as -Dsun.net.inetaddr.ttl=30java.security.Security.setProperty( "networkaddress.cache.ttl", "30");java.security.Security.setProperty( "networkaddress.cache.negative.ttl", "0");The operational conclusion is uncomfortable and worth stating: a DNS change does not reliably move traffic. It moves new resolutions. For a reliable cutover you need either a load balancer in front (so the address never changes), or an application that re-resolves, or a restart.
Negative caching
Section titled “Negative caching” NXDOMAIN AND NODATA ARE CACHED TOO, AND THE TTL COMES FROM A DIFFERENT PLACE:
THE SOA'S `minimum` FIELD, in the AUTHORITY section of the negative answer.
example.com. IN SOA ns1.example.com. admin.example.com. ( 2024061501 ; serial 7200 ; refresh 3600 ; retry 1209600 ; expire 300 ) ; MINIMUM ← negative ; cache TTLdig nope.example.com | grep -A2 'AUTHORITY'dig +short example.com SOA WHY THIS MATTERS
create a new subdomain, and anyone who looked it up BEFORE it existed caches the NXDOMAIN for `minimum` seconds.
A LARGE `minimum` (86400 IS STILL COMMON) MEANS A NEW RECORD IS INVISIBLE FOR A DAY TO ANYONE WHO CHECKED EARLY — INCLUDING THE IMPATIENT ENGINEER WHO TESTED IT FIVE MINUTES BEFORE CREATING IT.Set the SOA minimum to 300–900. It is the single most commonly misconfigured number in a zone file, and its effect is the “I created the record, it still says NXDOMAIN” experience.
Serve-stale
Section titled “Serve-stale” RFC 8767: SERVE-STALE
if the authoritative servers are unreachable, a resolver MAY keep serving an expired answer rather than fail.
Unbound serve-expired: yes serve-expired-ttl: 86400 BIND stale-answer-enable yes; max-stale-ttl 86400; 1.1.1.1 and 8.8.8.8 both do it.This turned a class of total outage into a degradation. When a provider’s authoritative servers went down historically, every domain they served vanished as caches expired; with serve-stale, resolvers keep answering with the last known value. It is also why “our nameservers are down but the site still works for some people” is now a normal observation rather than a contradiction.
Choosing TTLs
Section titled “Choosing TTLs” RECORD TTL WHY ────── ─── ─── apex A / AAAA 300 you will want to move it www A / AAAA 300 CDN / load balancer target 60 the provider needs agility MX 3600 mail retries; agility is cheap TXT (SPF/DKIM/DMARC) 3600 NS (at the apex) 86400 changes are rare and coordinated delegation NS (parent) 86400+ set by the registry SOA minimum 300 negative caching _acme-challenge TXT 60 written and read within seconds a record during a migration 60 THE COST OF A LOW TTL
more queries at your authoritative servers — which for a managed provider is their problem and is priced per million, and for a self-hosted setup is a real capacity question.
AND: essentially no latency cost to users, per chapter 5. a cache miss is one round trip to a resolver that already knows the branch.The default advice that survives scrutiny is 300 seconds for anything that points at infrastructure, 3600 for policy records, 86400 for NS — and 60 whenever you are actively changing something.
Flushing what you can actually reach
Section titled “Flushing what you can actually reach”You cannot flush the internet. You can flush the three or four caches between you and it, and knowing which is which saves a lot of confusion during a change.
# your own machinesudo resolvectl flush-caches # systemd-resolvedsudo killall -HUP dnsmasq # dnsmasqsudo unbound-control flush www.example.com # one namesudo unbound-control flush_zone example.com # a whole zonesudo rndc flushname www.example.com # BINDsudo dscacheutil -flushcache # macOS, plus:sudo killall -HUP mDNSResponderipconfig /flushdns # Windows
# the browserchrome://net-internals/#dnsabout:networking#dns AND THE ONES YOU CANNOT FLUSH every other resolver on earth your users' browsers their operating systems a JVM in somebody else's datacentre a connection pool that resolved last TuesdaySome public resolvers offer a purge interface — Google’s dns.google/cache and Cloudflare’s 1.1.1.1/purge-cache — and they are worth knowing about for an emergency. They cover two resolvers out of thousands, which is useful for testing and close to irrelevant for a migration.
The TTL you serve is not the TTL they keep
Section titled “The TTL you serve is not the TTL they keep” A RESOLVER MAY SHORTEN IT (a cache-max-ttl setting, or memory pressure evicting the entry early).
A RESOLVER MAY LENGTHEN IT (a cache-min-ttl floor, or serve-stale after expiry).
SO THE TTL IS A HINT WITH TWO BOUNDS YOU DO NOT CONTROL. plan for "roughly", never for "exactly".Cache poisoning, and why TTLs are a security parameter
Section titled “Cache poisoning, and why TTLs are a security parameter” THE KAMINSKY ATTACK (2008)
1. make the resolver query a name you control the parent of: random1.example.com 2. flood it with forged replies that contain not just an answer but an AUTHORITY section claiming new NS records for example.com 3. guess the 16-bit ID and the source port 4. if one lands before the real reply, the resolver caches YOUR nameservers for the whole zone, for the TTL YOU chose 5. failed? try again with random2. no cache blocks you.
THE FIX WAS SOURCE PORT RANDOMISATION, 0x20 ENCODING, AND — PROPERLY — DNSSEC.The reason it belongs in a caching chapter: the attack’s whole payoff is controlling what sits in somebody else’s cache, and for how long. A cache is a trust store with no authentication, and that is exactly the gap chapter 9 closes.
What to take from this chapter
Section titled “What to take from this chapter”- DNS has no invalidation. There is no flush, no callback, no way to reach a cache you do not own.
- Nothing “propagates”. Your servers answer the new value immediately; resolvers holding the old one do not ask again until it expires.
- Time to full visibility is bounded by the old TTL, not the new one. Lower the TTL, wait out the old TTL, then change the record.
- A resolver’s reply counts the TTL down, which is how you measure a cache’s age from outside.
- Many layers ignore the TTL entirely:
nscduses its own, browsers use ~60 s, the JVM caches forever by default, and a connection pool never re-resolves at all. - Therefore a DNS change does not reliably move traffic — it moves new resolutions. Reliable cutovers need a stable address in front, an application that re-resolves, or a restart.
- Very low TTLs are a request: large resolvers floor them, and corporate forwarders sometimes ignore them.
- Negative answers are cached using the SOA’s
minimumfield. Aminimumof 86400 makes a newly created record invisible for a day to anyone who checked early. Set it to 300–900. - Serve-stale (RFC 8767) turned authoritative outages from total failures into degradations — and prolongs bad answers too, so always fix forward rather than removing authority.
- Sensible defaults: 300 s for infrastructure records, 3600 for policy records, 86400 for
NS, 60 while actively changing. Low TTLs cost queries, not user latency. - The Kaminsky attack was entirely about controlling what sits in somebody else’s cache and for how long — a cache is a trust store with no authentication, which is what DNSSEC addresses.
Next: the wire protocol — the message format, the 512-byte legacy, EDNS0, TCP fallback, and the amplification problem.