How not to design an MTA - part 1 - the sendmail command

(I say part 1, but don’t expect the sequels to arrive quickly.)

The sendmail command is the de facto standard API for submitting email on unix, whether or not it is implemented by Sendmail. All other MTAs have a sendmail command that is compatible with Sendmail for important functions. (Notice how I pedantically use the uc/lc distinction.)

The traditional implementation

Sendmail and its early successors (including Exim) have been setuid root programs that implement all of the MTA functions. They are also decentralized, in that each instance of Sendmail or Exim does (mostly) the whole job of delivering a message without bothering much about what else is going on on the system. The combination of these facts is bad:

(1) A large setuid root program is a serious vulnerability waiting to happen. Sendmail has a long history of problems; Exim is lucky owing to conscientiousness rather than to good architecture.

(2) Particularly subtle problems arise from the effects of what sendmail inherits from its parent process, such as the environment and file descriptors. For example, consider sendmail invoked by a CGI. If the web server is careless and doesn’t mark its listening socket close-on-exec, the socket is inherited by the CGI and thence sendmail, which may then take ages to deliver the message. You can’t restart the web server while this is going on, because a sendmail process is still listening on port 80, which means you can’t restart the web server at all if the CGI is popular.

(3) Independent handling of messages makes load management very difficult. This is not only the load on the local machine, but also the load it imposes on the recipients. Sendmail and Exim lack the IPC necessary to be able to find out if the load is a problem before it is too late.

The qmail approach

Message submission in qmail is done with the qmail-inject program. This performs some header fix-ups, and it can extract the message envelope from the header rather than taking the envelope as separate arguments (like sendmail -t as opposed to plain sendmail). It then calls the setuid qmail-queue to add the message to the queue.

(4) The simple, braindead qmail-queue program does not impose any policy checks on messages it accepts, because that would be too complicated and therefore liable to errors. The fix-ups performed by qmail-inject are within the user’s security boundary, not the MTA’s, so they are a courtesy rather than a requirement.

The Postfix approach

Postfix is very similar to qmail as far as message submission is concerned, except that rather than fixing up a message, its sendmail command transforms the message into Postfix's internal form before handing it to postdrop which drops it in a queue. The fix-ups are performed later by the cleanup program, whcih also operates on messages received over the network. Which brings us to:

(5) Sendmail, qmail, and Postfix do not have an idea of message submission versus message relay. For example they tend to fix up all messages, wherever they come from - or in qmail’s case, not fix them at all.

Step back a bit

So what are the requirements?

(a) A clear security boundary between local users and the MTA. Note that all the MTAs rely on setuid or setgid programs that insert messages directly into the queue. Postfix and qmail ensure they are relatively small and short-lived, but they are still bypassing the most security-conscious part of the MTA, i.e. the smtp server. This opens up an extra avenue for attack - albeit only for local users. But why do they need special privileges?

(b) Policy checks on email from local users along the same lines as those from remote MUAs. Is this message submission or message relay? Does it need to be scanned for viruses? What are the size limits? Does address verification imply this user (e.g. nobody) cannot send email at all?

If you have a sophisticated system for smtp server policy checks, why bypass that for local messages? Exim can sort-of do what I want, but it retro-fits the policy checks onto the wrong architecture.

The fanf approach

The sendmail program is a very simple RFC 2476 message submission client: it talks SMTP to a server and expects the server to do the necessary fix-ups. It doesn't need any special privilege: from the server's point of view it is just another client.

It’s not quite as simple as that. You need to authenticate the local user to the server, because users should not be able to fake Sender: fix-ups, and there are situations when you will want to treat users differently, e.g. email from a mailing list manager. So instead of SMTP over TCP, talk it over a unix domain socket, which allows unforgeable transmission of the client user ID.

Problem (1) solved: no setuid or setgid programs. Problem (2) solved: client process is short-lived and synchronous. Problem (3) solved: messages all go through the same channel. Problem (4) solved: messages all go through the same policy engine. Problem (5) solved: the policy engine is powerful enough to know when submission-mode fix-ups are becessary.

One thing you do lose by this approach is that the sendmail command only works when the SMTP listener is running, which is not a problem with the other designs. But I’m not convinced this is a serious difficulty, and in fact it can be viewed as an advantage - it doesn’t let email silently disappear into an unserviced queue.

A question arises with this architecture - which also arises for remote MUAs - which is, where is the best place to generate the message envelope? i.e. the transport-level sender and recipient addresses. RFC 2476 says that the client does this job, however no-one has written a decent sendmail -t replacement, and even “serious” MUAs get this job wrong. Furthermore, the server still has to parse the header and perform various checks and fix-ups, so why shouldn’t it generate the envelope too? Hence draft-fanf-smtp-rcpthdr, which also has the best description of how to handle submission-time fix-ups of re-sent messages.