Tony Finch – blog

One of the few remnants of the old JANET grey book email system that we still support is abbreviated email domains such as in <fanf2@cam>. Exim has two configuration options which we use to implement this: widen_domains = cam.ac.uk : ac.uk and rewrite_headers.

When someone sends email to an abbreviated address, Exim does not recognize the domain as being one of the ones it is explicitly configured to handle, so looks it up in the DNS. This DNS lookup fails, so Exim tries appending each of the widen_domains in turn and re-doing the DNS lookup. Thus @hermes becomes @hermes.cam.ac.uk, which works, and @cam becomes @cam.cam.ac.uk, which doesn’t, then @cam.ac.uk, which does.

Abbreviated domains are not valid on the public Internet, so they must be replaced by their un-abbreviated equivalents before the message leaves ppswitch. Email messages have two sets of metadata: the envelope, which determines who receives the message and who gets any delivery failure reports (i.e. the sender); and the header, which is what the user sees. [1] Since Exim is concerned with getting messages from A to B, it generally only looks at the envelope, and leaves the header alone. However when the envelope contains an abbreviated address there is probably a copy of the address in the header, so Exim must fix up both of them. This is specified by the rewrite_headers option.

[1] The two do not necessarily agree: for example, a message sent via a mailing list has the list’s address in its To: header, but the list members’ addresses in the envelope. However, when a message is submitted the initial envelope is created from the addresses that the message’s author specified in the header, so the two sets of addresses start off the same.

That is all well and good, but there are some subtleties. Widening an abbreviated domain is a side-effect of the address being run through Exim’s routers. Routing happens in two situations: when Exim is verifying addresses in order to decide if the message is acceptable; and when Exim is working out where to deliver the message after it has been accepted. The rewrite_headers option only has an effect in the latter case, because in the former case Exim has not received the message header yet. (Even if the header had been received, Exim doesn’t record these header fix-ups persistently but instead works them out again whenever necessary.)

This is not a problem for recipient addresses, because these are all run through the routers at delivery time. However the sender address is not, because it is not necessary to work out the sender’s destination in order to deliver the message to the recipients. The sender address is only run through the routers in order to verify it before the message is accepted. Therefore any fix-ups that may result from expanding the sender address do not happen!

Fortunately no-one configures their software to use abbreviated sender addresses, so we can safely prevent people from making this configuration error and thereby emitting broken messages, without causing disruption. This required a somewhat subtle change to Exim’s source code. The idea is to ignore the widen_domains option when verifying a sender address, so that an abbreviated domain remains unrecognized and the address fails to verify, causing the message to be rejected. However if the postmaster has turned off rewrite_headers (for example, because they are working in a walled garden where it is OK for abbreviated domains to propagate) then the original problem doesn’t obtain, so widen_domains should not be ignored.

Even more subtly, abbreviated addresses can appear in aliases files, for example <fanf2@ucs.cam.ac.uk> -> <fanf2@cam> which means that if I send a message using the alias, Exim will generate the abbreviated address as a result of sender verification. In this situation, the abbreviated address does not appear in the message header, so it does not need to be fixed up, so it should be permitted. Thus, we ignore widen_domains if we are verifying a sender address, but not if rewrite_headers = false, and only if this is the original sender address (not the result of alias expansion).

I didn’t work out the latter “even more subtle” part of this fix until after I had rolled out my new Exim configuration yesterday and after I had received a bug report from a computer officer whose webserver’s email was being bounced. Oops! I’ll have to try the roll-out again on Monday…