How not to design an MTA - part 3 - local delivery

How depressing. These essays were supposed to turn up reasonably frequently, but I got hung up on the original plan for essay 3 and it still isn’t quite writable. In essay 2 I argued that it’s very hard indeed to implement strong partitions within a program running on Unix, even when it runs as multiple processes. Essay 3 was supposed to be a more nuanced argument about security engineering, and therefore more positive about partitioning, but it’s hard to be opinionated when you don’t have a definite answer.

So let’s go back to basics and take a look at another public interface of an MTA. In essay 1 I talked about local message submission, in which I argued that every current MTA does it wrong. This time I’m going to look at the local delivery agent - LDA - and again, I think every current MTA does it wrong.

Background

Local delivery on Unix is complicated. The set of valid local parts of email addresses (the bit before the @) is basically the same as the list of local users, defined in the passwd file. But extra valid local parts can be defined in /etc/aliases. Delivery is usually just appending the message to /var/mail/user, but if the user has a .forward file, that may specify an alternative mailbox, or redirection to another user or address, or sending the message to a program, or any combination of the above. And aliases have the same delivery options. Furthermore, special entries in the aliases file can cause magic return-path rewriting, which is used to make delivery errors from mailing lists go to the list managers instead of the person contributing the message to the list. And finally there may be a mechanism for accepting email for otherwise-unknown local parts.

Trad MTAs

Sendmail and Exim do (almost) all of their email routing and delivery as root. So essentially all of the complicated programming outlined in the previous paragraph must be trusted by every user of the computer, and it must be robust and correct.

The “almost” arises from the fact that some parts of the process may be run under the identity of the destination user, in particular the process of writing the message to a mailbox or delivering it to a program. Interpreting .forward files may also be done as the user, which is particularly important if they contain complicated filtering scripts as for Exim or Sieve. But still, the end result of routing the address is a filename or program, and you have to trust the MTA to get it right.

The qmail approach

There are a couple of differences in the way that qmail does things.

Most interestingly, it adds a couple of security boundaries. The first one occurs between deciding that an address is local and the whole process of local delivery. In this case the qmail-lspawn daemon has the necessary privilege and is responsible for switching to the appropriate user and running the program that delivers the message. The second one occurs at a redirection (including aliases and forwards), in which case the message is re-injected into the queue.

Less interestingly, it has its own way of describing aliases and forwards. The .qmail file has a very slightly simpler format than a .forward file, but essentially the same features. It also has an alternative aliases file format which is reminiscent of a password file, and symultaneously less flexible and more redundant than a normal aliases file. Finally, it falls back to looking at a special “alias” user’s .qmail files.

To some extent, this reduces and divides up the complexity of local delivery. But not much. It reduces it by using simpler file formats (“don’t parse”), but that doesn’t achieve much, since the .qmail processing is done under the identity of the user who created the file. It divides it by separating alias handling from forward handling, but this means the whole thing requires four programs and extra re-injections.

The postfix approach

Postfix is much more compatible with the traditional way of doing things: it understands the standard /etc/aliases and .forward files without qmail's gratuitous wheel-reinvention. And, just as in essay 2 it has less security partitioning than qmail - where qmail's LDA is three programs (not counting re-injection this time), Postfix's LDA is one.

Essentially all of the traditional local delivery logic is implemented inside Postfix’s “local” daemon. This implies that it must do a lot of work as root, in particular alias processing: it has to work out which local user the alias redirects to before it can drop privileges. The manual’s “security” section is remarkably coy about the details of this, saying only that it is “security sensitive” (duh).

What is worse is that it has to include a lot of support for Postfix’s infrastructure: It has lots of options in the configuration file which it must therefore parse, and it also has to support all of Postfix’s database types. Some of these can be fiendishly complicated: LDAP, SQL, embedded databases, regex matches, sockets, etc. etc. As a result it is one of the largest parts of Postfix. Are you really happy running all this as root, even for a “secure” MTA?

This mention of other database types hints at the complexity quagmire that email can be. Unix MTAs often have to do the interoperability between the Internet and “enterprise messaging systems” - which often means LDAP is necessary. And they have to do virtual hosting of various kinds: a virtual domain may be just an aliases file without local users, or it may be a pile of local mailboxes with a local user per domain, or it may be more complicated.

Postfix’s LDA has the traditional Unix logic hard-coded. It can’t do virtual domains. Postfix implements alias-based virtual domains as part of its main routing logic (but trad aliases are not done this way - huh?), and it has a separate parallel local delivery daemon for domains with mailboxes which duplicates much of the local(8) code (double-huh?).

Step back a bit

So how would we like it to work?

We want all the complicated routing logic to be performed by the main MTA process, running as a non-privileged user (with only access to the email queue and anciliary databases). The LDA does not worry about .forward or /etc/aliases. It does not link to database libraries.

However, we cannot trust the result of this routing: if we did, the MTA could alter any file on the system, so it could gain root privilege by appending an appropriate “message” to the password file. The LDA must have some way of verifying that the MTA’s instructions are legitimate.

For Sendmail and Exim, the LDA necessarily trusts the MTA, because the MTA already is root. For qmail and Postfix, the LDA doesn’t need to verify anything because it does the work itself, either as the destination user or as root.

Another wrong way

In simple cases it's easy for the LDA to verify its instructions: the MTA might say "deliver to this file as this user! and you know you can because the file's name appears in the user's .forward!", and with a bit of filesystem permissions checking the LDA can be satisfied. However it can't do this kind of check against the result of a database query. Even if we allow the LDA to link to the database library, it still can't verify that arbitrary SQL is legitimate.

The fanf way

In most cases, you don't want the LDA to deliver to many places. You might have a standard place for a user's mailboxes, e.g. ~/mail/, which is where the local MUAs and IMAP servers look. Delivery to other places is forbidden. So it should be really easy to tell the LDA where these places are, entirely independently of the MTA configuration: no matter whether the latter uses plain text /etc/aliases or databases or LDAP or whatever. You could just list the allowed paths in a file.

The LDA just gets an instruction from the MTA: deliver this message to this file as this user. The LDA switches to the user, checks that the file is listed as being OK, checks filesystem permissions, and writes the message. It then sends back a success or failure indication to the MTA. Really simple. Works for trad Unix delivery and virtual delivery and weird delivery.

Delivery to programs is similar - and in fact, here there are precedents for an anciliary permitted-destinations list. The Sendmail restricted shell, smrsh (which is unrelated to the James Bond villains), has a list of commands that it is willing to run. This is also a bit like userv.

One problem - easily solved - is that a simple list of destinations is not enough. You need some pattern matching, such as “anything under this directory”, or “this file in my home directory, wherever that is”. You want a system-wide default so that users don’t have to be bothered with this detail, but you still want clever/demanding users to be able to refine it - e.g. “I want my inbox in ~/mailbox so never deliver to the default /var/mail/me”. (This is still much simpler than .foward syntax.) You also want a tool to auto-generate the allowed-paths file from a .forward file or aliases file, so the common case remains easy to use.

Tangential issues

It's very common for redirections discovered in the course of local delivery to cause the message to be re-injected. This is bad for performance (which was going to be the topic of essay 4, but may now turn out to be essay 5). With my design, local delivery happens after all redirections have occurred, so the message doesn't have to make another trip through the MTA.

We’d also like to be able to handle redirections from more complicated filtering scripts in this efficient way, but at the same time we would prefer these scripts (written by untrusted users) to be run with the privileges of their authors, not with the privileges of the MTA. A small extension to the MTA->LDA protocol allows the MTA to ask for a script to be run and to get the results back: redirections, rejections, deliveries, etc. It can even do this when verifying recipient addresses at SMTP time, so that scripted rejections don’t turn into accept-then-bounce.

Exim does the redirection short-cut, but it lost the filter verification short-cut when its security was improved for version 4. It’s nice that my LDA design allows these performance and anti-spam features whilst remaining secure.