Tony Finch – blog

The 101st IETF meeting is in London this coming week, and it starts with the IETF 101 Hackathon.

I thought I could do some useful work on DNS privacy. There is lots of work going on in this area, part of which is adding lots of new transport options for DNS - as well as the traditional DNS-over-UDP and DNS-over-TCP, there is now DNS-over-TLS and (soon) DNS-over-HTTPS, and maybe DNS-over-DTLS and DNS-over-QUIC.

My idea was to set up a proxy that could provide DNS-over-TLS and DNS-over-HTTPS in front of a trad DNS server (for my purposes, specifically BIND, but the proxy does not need to care).

Choice of proxy

There are a number of proxies that make sense as a base for this work: I need TLS support and HTTP support and a reasonably light-weight implementation - HAProxy is one, but I chose to use NGINX. There’s a variant of NGINX called OpenResty which includes LuaJIT and a bunch of other libraries and plugins.

Lua is a really nice language for scripting an event-driven server like NGINX, since Lua has coroutines. This means you can write straight-line Lua, and whenever it calls into a potentially-blocking OpenResty API, it actually suspends the coroutine and drops into the NGINX event loop.

Existing infrastructure

I have a little virtual machine cluster on my workstation for development and prototyping - I use it for my work porting the IP Register database to PostgreSQL. The relatively unusual part of the setup is that I have a special DNS zone (signed with DNSSEC) for these VMs, and the VMs and their provisioning scripts have DNS UPDATE privileges on this DNS zone. When a VM boots and gets IP addresses form DHCP and SLAAC, it UPDATEs its entry in my dev zone. When it generates its ssh host keys, it puts the corresponding SSHFP records in the zone.

Plan of attack

My prototypes are configured using Ansible, so that there’s relatively little required to bring them up to production quality. So the basic plan was to write a playbook to install OpenResty, configure NGINX, and add a bit of Lua to massage DNS-over-HTTPS requests into DNS-over-TLS.

I have a bit of previous experience with Lua, but I have never used NGINX before, so there will be a certain amount of learning…

Installation

There are pre-built OpenResty packages which are quite convenient - very similar to the existing setup I have for PostgreSQL. Happily the OpenResty packages include a SysVinit rc script so they are compatible with my non-systemd VMs.

TLS certificates

To get a certificate for my service I needed to use Let’s Encrypt, which I have not done before (since we have an easy enough TLS certificate service at work). I chose to use the dehydrated ACME client since I have friends who report that it is very satisfactory. Since I’m a DNS geek, I thought it would be fun to use the ACME DNS-01 challenge. (I realised later that this was a lucky choice, since my VMs have RFC 1918 IPv4 addresses.)

One simple script (basically copied off the dehydrated wiki) and two lines of configuration, and I could run dehydrated -c and get a TLS certificate within a few seconds. Absolutely brilliant. My first experience of really using Let’s Encrypt was really pleasing.

DNS over TLS

This is fairly easy to set up with a bit of Googling StackOverflow to work out how to configure NGINX. The trick is to use a stream {} section instead of the usual http {} section.

    stream {
        upstream dns {
            server 131.111.57.57:53;
        }

        server {
            listen [::]:853 ssl;
            proxy_pass dns;

            ssl_certificate      cert.pem;
            ssl_certificate_key  cert.key;
        }
    }

Getting to grips with OpenResty

I spent a while reading around the OpenResty web site looking for documentation, but I wasn’t having much luck beyond the “hello world” example. What I needed was some example code that showed how to pick apart an HTTP request and put together a response.

I found OpenResty’s Lua DNS library which I thought might be useful for cribbing from, but it did not help at all with HTTP.

Introspecting Lua

One of the coolest talks at the LuaConf I went to several years ago was about how Olivetti use (used?) Lua to write test code for their scanner/printer devices. The bulk of the firmware was written in C++, but it included a Lua interpreter which was able to dig through the RTTI, allowing a test engineer to tab-complete through the device’s internal data structures and APIs.

So, being unable to find anything else, I thought I could get some idea of what the OpenResty API looked like by doing a bit of reflection.

I found a nice script called inspect.lua which pretty-prints a Lua data structure. You can dump all the global variables recursively with inspect(_G), which gave me a nice listing of the Lua standard library and some OpenResty bits.

I added the following variant of “hello world” to nginx.conf so I could conveniently curl a list of the OpenResty ngx module.

    location /ngx {
        default_type text/plain;
        content_by_lua_block {
            ngx.say(require("inspect")(ngx))
        }
    }

Writing an HTTP request handler

There were some fairly promising functions called things like ngx.req.get_method() and ngx.req.get_headers() which turned out to do reasonably obvious things. I also dug through the OpenResty Lua module sources that were installed on my dev VM to get a better idea of how they worked.

This was just about enough that I could write a handler to implement most of the DNS-over-HTTPS requirements.

The main stumbling point came when I needed to do base64url decoding of DNS query packets embedded in HTTP GET requests.

Whither base64url

After a bit of grepping it became evident that OpenResty has support for normal base64 - I found its resty.core.base64 module, but disappointingly it does not include base64url support. However, running strings on nginx revealed that nginx does have base64url support, though OpenResty does not expose it to Lua.

The base64.lua module uses the LuaJIT FFI to call some OpenResty C wrapper functions that convert vanilla C types to NGINX’s internal types, then call NGINX’s base64 functions.

The LuaJIT FFI is a glorious thing: as well as calling C functions, you can directly access C structures from Lua. So OpenResty’s C wrapper layer is not in fact needed.

So, after a bit of clone-and-hack (and an embarrassing diversion spending ages working out that I had mistyped a variable name) I was able to write a 50 line base64url.lua module which called ngx_decode_base64url() directly.

DNS time

At this point my HTTP handler was able to get a DNS wire format query from an HTTP GET or POST request, so I needed to forward it to a DNS server, and return the response to the client.

During my base64url adventures I had worked out that the documentation I had been looking for belongs to the OpenResty Lua Nginx Module, so it was straightforward to write the back half of the proxy.

This back end is the bare minimum: just 20 lines of code including error checking and tracing. The whole DNS-over-HTTPS handler is less than 100 lines of Lua.

This is where it becomes obvious that OpenResty shines, because all the front-end POST reading and back-end socket calls are potentially blocking, but I can write straight-line sequential code without any inversion of control.

DoH clients

At this point I had a server, but no client for testing it!

I was sitting next to the author of doh-proxy who told me that he has one. I installed it, but found that it would hang after sending a request. (I did not debug this problem.)

So I decided to go low-tech.

The doh-client printed its base64url-encoded query, so I could curl it myself. I just needed something that could pretty-print the response.

First attempt was with drill which can dump DNS packets and print dumped packets. However its dump format is an ASCII hex dump, not binary, so I found myself writing a complicated curl | perl | drill pipeline and it was getting silly.

So fairly rapidly I moved on to the second attempt in pure perl, using Net::DNS for query packet construction and response packet pretty-printing, MIME::Base64 for that troublesome base64url encoding, LWP::UserAgent for performaing the HTTPS request. 20 lines of code, and I have a client that works with my server!

Tomorrow

There are a couple of obvious next steps.

Firstly, I should extract this work from my prototyping setup, so that it can be published in a form that’s plausibly useful for other people. Edited to add: Done!

Secondly, my trivial DoH back end needs some work. It uses one brief DNS-over-TCP connection per HTTP request, which is rather wasteful. It would be a lot cooler to keep one or more persistent TCP connections open to the DNS server, and multiplex DNS-over-HTTP requests onto these DNS-over-TCP connections. It looks like NGINX and OpenResty have lots of support for connection pooling, so I should work out how I can make good use of it.

Today has been pleasingly successful, so I hope tomorrow will have more of the same!