Tony Finch – blog

Back 5 years ago when I was working with adns, one of the things I played with a bit was a perl wrapper. adns is absolutely fantastic for bulk log processing - being able to do more than 10,000 concurrent queries so that your’re using all your CPU and not blocking on the network is a god-send. However C makes this more painful than it ought to be.

I never finished the perl wrapper because other things became more important, and when I next had the time and the inclination to look at it Net::DNS existed, so I thought there would be little point.

I’ve been paying gradually more and more attention to SpamAssassin recently, and it uses Net::DNS’s background query feature to run all its DNS queries concurrently with its pattern matching. As a result of this I’ve found out that Net::DNS’s background query handling is utterly stupid: it uses a separate socket for each query, rather than stuffing them all down the same socket and using the DNS protocol’s query ID field to tie responses to queries.

This causes excessive resource usage which greatly restricts the number of concurrent queries it can handle, even on a sensible OS. On Windows it dies if the concurrency goes above about 350, which occasionally happens with SpamAssassin. http://bugzilla.spamassassin.org/show_bug.cgi?id=3924

So now I have the bit between my teeth. Must f1xx0r!