Tony Finch – blog
tl;dr - you can use my program adns-masterfile to pre-heat a DNS cache as part of a planned failover process.
adns
One of my favourite DNS tools is GNU adns,
an easy-to-use asynchronous stub resolver library.
I started using adns in 1999 when it was still very new. At that time
I was working on web servers at Demon Internet. We had a daily log
processing job which translated IP addresses in web access logs to
host names, and this was getting close to taking more than a day to
process a day of logs. I fixed this problem very nicely with adns, and
my code is included as an example adns client program.
Usually when I want to do some bulk DNS queries, I end up doing a
clone-and-hack of adnslogres. Although adns comes with a very
capable general-purpose program called adnshost, it doesn't have any
backpressure to control the number of outstanding queries. So if you
try to use it for bulk queries, you will overwhelm the server with
traffic and most of the queries will time out and fail. So, although
my normal approach would be to massage the data with an ad-hoc Perl
script so I can pipe it into a general-purpose program, with adns I
usually end up writing special-purpose C programs which have their own
concurrency control.
cache preheating
In our DNS server setup we use
keepalived to do automatic failover of
the recursive server IP addresses. I wrote about our keepalived
configuration last year,
explaining how we can control which servers are live, which are
standby, and which are not part of the failover pool.
The basic process for config changes, patches, upgrades, etc., is to take a stanby machine out of the pool, apply the changes, test them, then shuffle the pool so the next machine can be patched. Patching all of the servers requires one blip of a few seconds for each live service address.
However, after moving a service address, the new server has a cold cache, so (from the point of view of the user) a planned failover looks like a sudden drop in DNS server performance.
It would be nice if we could pre-heat a server's cache before making it live, to minimize the impact of planned maintenance.
rndc dumpdb
BIND can be told to dump its cache, which it does in almost-standard textual DNS master file format. This is ideal source data for cache preheating: we could get one of the live servers to dump its cache, then use the dump to make lots of DNS queries against a standby server to warm it up.
The named_dump.db file uses a number of slightly awkward master file
features: it omits owner names and other repeated fields when it can,
and it uses () to split records across multiple lines. So it needs a
bit more intelligence to parse than just extracting a field from a web
log line!
lex
I hadn't used lex for years and years before last month. I had
almost forgotten how handy it is, although it does have some very
weird features. (Its treatment of leading whitespace is almost as
special as tab characters in make files.)
But, if you have a problem whose solution involves regular expressions
and a C library, it's hard to do better than lex. It makes C feel
almost like a scripting language!
(According to the Jargon File, Perl was described as a "Swiss-Army chainsaw" because it is so much more powerful than Lex, which had been described as the Swiss-Army knife of Unix. I can see why!)
adns-masterfile
So last month, I used adns and lex to write a program called
adns-masterfile
which parses a DNS master file (in BIND named_dump.db flavour) and
makes queries for all the records it can. It's a reasonably handy way
for pre-heating a cache as part of a planned server swap.
I used it yesterday when I was patching and rebooting everything to
deal with the glibc CVE-2015-7547 vulnerability.
performance
Our servers produce a named_dump.db file of about 5.3 million lines
(175 MB). From this adns-masterfile makes about 2.6 million queries,
which takes about 10 minutes with a concurrency limit of 5000.