Metadata for login credentials with regpg

Originally posted at https://www.dns.cam.ac.uk/news/2019-09-28-regpg-login.html

This month I have been ambushed by domain registration faff of multiple kinds, so I have picked up a few tasks that have been sitting on the back burner for several months. This includes finishing the server renaming that I started last year, solidifying support for updating DS records to support automated DNSSEC key rollovers, and generally making sure our domain registration contact information is correct and consistent.

I have a collection of domain registration management scripts called superglue, which have always been an appalling barely-working mess that I fettle enough to get some task done then put aside in a slightly different barely-working mess.

I have reduced the mess a lot by coming up with a very simple convention for storing login credentials. It is much more consistent and safe than what I had before.

The login problem

One of the things superglue always lacked is a coherent way to handle login credentials for registr* APIs. It predates regpg by a few years, but regpg only deals with how to store the secret parts of the credentials. The part that was awkward was how to store the non-secret parts: the username, the login URL, commentary about what the credentials are for, and so on. The IP Register system also has this problem, for things like secondary DNS configuration APIs and database access credentials.

There were actually two aspects to this problem.

Ad-hoc data formats

My typical thoughtless design process for the superglue code that loaded credentials was like, we need a username and a password, so we’ll bung them in a file separated by a colon. Oh, this service needs more than that, so we’ll have a multi-line file with fieldname colon value on each line. Just terrible.

I decided that the best way to correct the sins of the past would be to use an off-the-shelf format, so I can delete half a dozen ad-hoc parsers from my codebase. I chose YAML not because it is good (it’s not) but because it is well-known, and I’m already using it for Ansible playbooks and page metadata for this web server’s static site generator.

Secret hygiene

When designing regpg I formulated some guidelines for looking after secrets safely.

From our high-level perspective, secrets are basically blobs of random data: we can’t usefully look at them or edit them by hand. So there is very little reason to expose them, provided we have tools (such as regpg) that make it easy to avoid doing so.

Although regpg isn’t very dogmatic, it works best when we put each secret in its own file. This allows us to use the filename as the name of the secret, which is available without decrypting anything, and often all the metadata we need.

That weasel word “often” tries to hide the issue that when I wrote it two years ago I did not have an answer to the question, what if the filename is not all the metadata we need?

I have found that my ad-hoc credential storage formats are very bad for secret hygiene. They encourage me to use the sinful regpg edit command, and decrypt secrets just to look at the non-secret parts, and generally expose secrets more than I should.

If the metadata is kept in a separate cleartext YAML file, then the comments in the YAML can explain what is going on. If we strictly follow the rule that there’s exactly one secret in an encrypted file and nothing else, then there’s no reason to decrypt secrets unnecessarily everything we need to know is in the cleartext YAML file.

Implementation

I have released regpg-1.10 which includes ReGPG::Login a Perl library for loading credentials stored in my new layout convention. It’s about 20 simple lines of code.

Each YAML file example-login.yml typically looks like:

# commentary explaining the purpose of this login
---
url: https://example.com/login
username: alice
gpg_d:
  password: example-login.asc

The secret is in the file example-login.asc alongside. The library loads the YAML and inserts into the top-level object the decrypted contents of the secrets listed in the gpg_d sub-object.

For cases where the credentials need to be available without someone present to decrypt them, the library looks for a decrypted secret file example-login (without the .asc extension) and loads that instead.

The code loading the file can also list the fields that it needs, to provide some protection against cockups. The result looks something like,

my $login = read_login $login_file, qw(username password url);
my $auth = $login->{username}.':'.$login->{password};
my $authorization = 'Basic ' . encode_base64 $auth, '';
my $r = LWP::UserAgent->new->post($login->{url},
          Authorization => $authorization,
          Content_Type => 'form-data',
          Content => [ hello => 'world' ]
      );

Deployment

Secret storage in the IP Register system is now a lot more coherent, consistent, better documented, safer, … so much nicer than it was. And I got to delete some bad code.

I only wish I had thought of this sooner!