The three-second DNS stall
For weeks my laptop felt "a bit sluggish" on the first page load of any site, then perfectly fast afterwards. I blamed Wi-Fi, then the browser, then the weather. It turned out to be a number I typed into a config file two years ago.
Measuring instead of guessing
curl can split a request into phases, which is the fastest way to stop arguing with yourself:
curl -so /dev/null -w 'dns=%{time_namelookup} tcp=%{time_connect} tls=%{time_appconnect} total=%{time_total}\n' https://example.org
The first run printed dns=3.004. The second run, a few seconds later, printed dns=0.002. Three seconds, almost exactly, every time the name was not already cached.
Round numbers are a clue
Nothing on the internet is reliably slow by exactly three seconds. Timeouts are. So the question became: which component is waiting three seconds for an answer that never arrives, and then falling back to one that works?
My local resolver was configured with two upstreams. The first one had quietly stopped answering over the transport I had chosen. Every cold lookup went to it first, waited for the full timeout, then tried the second upstream, which answered in 20 ms.
The fix
- Remove the dead upstream instead of relying on fallback.
- Query upstreams in parallel where the resolver supports it, and take the first answer.
- Lower the per-upstream timeout to something that matches reality (a few hundred milliseconds, not seconds).
After the change, cold lookups dropped to ~30 ms and the "sluggish laptop" disappeared.
Lesson: if a delay is a suspiciously round number, go looking for a timeout before you go looking for a slow network.