This entire discussion has been growing increasingly abstract, and that's
rarely a good thing in standards work.

Rather than continue in that direction, I decided to do something I should
have done a lot sooner: I implemented the draft.

<disclaimer>

The fact that I have implemented this draft in Oracle Messaging Server does not
imply any commitment on the part of Oracle to actually support this extension.
And should Oracle decide to support it, this doesn't mean it will be done
in any particular release or in any particular time frame.

</disclaimer>

(Sorry about having to include that, but rules are rules.)

Implementation work is usually very instructive, and this time was no
exception. I'll list the simple things I learned first:

(1) The draft says that the extension is valid for LMTP, but doesn't give any
    guidance as to what that means. This could be interpreted as saying that
    an implementation that uses LMTP has to support the MT-PRIORITY extension 
    there to be compliant. That would in turn mean that prioritization has
    to be done on the LMTP server side, which may be a bit difficult as there's
    no queue there. Our implementation, and I suspect many others, handles
    prioritization on the LMTP client side.

    That said, it's entirely possible for the MT-PRIORITY to affect the LMTP
    server by changing processing or I/O handling in some way, or affect
    subsequent store access behavior, or may be needed just so it can be
    logged. So the extension should be allowed over LMTP, but some clarifying
    text is needed to say an implementation MAY choose to handle
    prioritization on the client side of the LMTP connection only.

(2) The draft also allows MT-PRIORITY on submission, which of course makes
    complete sense. However, given the overall state of MUAs in the world
    today,  it is next to certain that clients are going to be used that don't
    support it. (And please don't try the line that this can always be dealt
    with by requiring the use of certain clients. I'm talking about the real
    world, not fantasyland.) As such, there are going to be cases where the
    MSA needs to attach an MT-PRIORITY to messages that don't have one. It
    would be good to mention this, but even if it's not discussed there needs
    to be an enhanced status code defined to indicate when it has been done,
    and more generally to indicate when the MT-PRIORITY has been upgraded.

(3) Implementations that support Sieve need to provide a way for Sieves to
    test the current MT-PRIORITY value. I implemented this as an environment
    item because that's a place that allows for implementation-specific values.
    The alternative would be to do it as a full-blown Sieve extension and add
    it to the envelope test. Given that MT-PRIORITY is an SMTP extension the
    envelope test is the natural place to do it, but a problem with that may
    be it restricts the test to implementations and situations where the
    envelope test itself is supported. This is not a problem for our
    implementation, but it may be a problem for others.

    Another issue is the lack of a defined comparator that will work with
    MT-PRIORITY values. i;ascii-numeric doesn't handle signed comparisons.
    An i;ascii-integer comparator is needed, so I added one and I think I'll
    go ahead and add the definition to the comparator registry.

    It may also be a bad idea to add all this in the current draft, especially
    given where this draft is in the process. But it needs to be done somewhere.

(4) The draft doesn't say anything about logging. The current MT-PRIORITY
    value does appear in Received: fields, but that's not the same thing.
    Logging of how MT-PRIORITY is used is going to be a requirement in some
    environments, so a general suggestion along the lines of "MT-PRIORITY
    values SHOULD be logged as part of any logging of message transactions" is
    in order.

(5) The suggested text for the X.3.TBD3 error code doesn't conform to the
    requirement that the text begin with the new priority value. (Note that
    the new error code suggested under (2) should have the same requirement.)

(6) The draft talks about greylisting, but fails to mention that
    connection-level and SMTP HELO/HELO greylisting options exist, and when
    these options are used there's an issue if the trust relationship is
    established through the use of SASL. This needs to be pointed out.

I also spotted a minor typo in the Introduction: sattelite -> satellite.

The draft also spells out numbers in some places and writes them as
actual numbers in others, making it hard to search for things. I don't care
which approach is used as long as it is consistent.

Now on to the more difficult stuff.

I first need to say a few words about how our product is implemented. We have
separate entities we call "channels" that messages can be routed to based on
various criteria. Each channel has it's own queue for messages, and can be
separately configured as to how many delivery threads to use, how frequently to
retry, what IP address and port to use, and so on and so forth. Additionally,
each queue supports three priorities, basically meaning higher priority
messages get tended to first and will be retried more frequently.

I realized right away that not only does a single channel approach not suffice
because a single channel doesn't have enough available priorities (the draft
requires six; see section 5), but also because even if we were to add
additional queue priorities, it's unlikely that that would actually create a
useful distinction between all the different priorities. For a useful
distinction to exist, especially across so many different levels, you need
access to things like different connections that shape traffic differently, and
in our world that means different channels.

So what I did was provide various hooks so that routing can be adjusted based
on priority. The obvious way to configure things would be to have two channels
and route three MT-PRIORITY levels to each one, but of course many other
arrangements are possible, up to and including ones that fullly support 19
distinct levels.

But there's no way I can guarantee that things will be configured this way. I
can't even test for it, since I have no way of testing external conditions. And
this is true no matter what implementation strategy I use, because any
sufficiently flexible configuration mechanism can be used to defeat priority
handling just as easily as it can be used to enforce it.

What this means is that the six level business is really an operational
requirement not an implementation requirement. What the implementation needs to
do is provide the *means* to support at least six priority levels. That's all
that can be reasonably expected, and the draft really needs to make that clear.

This then raises the question of whether or not we should make use of this
extension contingent on operational support of six levels. I think the answer
to that is a rather emphatic "no" - it should be left to the service profile to
decide how many levels they actually need. And that also needs to be made very
clear - in particular, statements as to supporting six levels being a
requirement, which sure sound like an operational requirement to me, need to
either be qualified or removed.

But this in turn raises another question: Should there be some indication of
what service profile is being provided? Proposals have been made previously to
do things like include the available priorities in the EHLO response. There are
two problems with that: (1) It doesn't actually identify the profile, and (2)
There's nothing a client can usefully do with the information.

A better alternative would be to include some sort of profile name in the EHLO
response. This wouldn't be for operational use, but rather for auditing
purposes. Of course an implementation like ours, and I suspect most others,
would not be done in a profile-specific way, so this would have to be a
configuration option. And I'm not sufficiently fond of the idea to say it
should be done. I'm only offering it as a suggestion.

Notifications next. The draft says that notifications SHOULD be processed at
the same priority as the message they are responding to. Having now implemented
this, I'm somewhat tempted to make that a MUST, because if you try and do it
any other way it's like opening a pandora's box of failure cases. Perhaps a
warning to the effect that, "Regardless of the approach chosen, schemes which
cause notifications to be completely lost MUST NOT be used."

But the draft then goes on to say that:

   For delivery reports received by an MTA, processing rules specified
   in Section 4.1 apply.  But note that while it might be tempting to
   handle all delivery reports (a.k.a.  DSNs) at their stated priority,
   under the assumption that failure notices need to get through quickly
   in some situations, but such a policy creates an exposure to fake-DSN
   attacks if sources of DSNs can't be reliably established.

This seemed to make sense until I tried to implement it, at which point I
realized it makes no sense at all. The problem is fundamental: There is in
general no way to tell whether or not you're dealing with a DSN. Yes, we
have defined format for DSNs, but there are still plenty of things out there
that generate DSNs in a different format. 

Even if we were to require support for NOTARY (and thus for our DSN format), we
need to be able to make priority decisions during envelope processing, and the
only indication there is the empty MAIL FROM. But while DSNs are required to
use an empty MAIL FROM, the converse isn't true: Plenty of things other than
DSNs use empty MAIL FROM.

And if that wasn't enough, the implication here seems to be that DSNs are
intrinsicly harder or more difficult to authenticate. Huh? That would seem to
imply that DSNs arrive from some sort of alternate source, or we're supposed to
base authentication decisions on the MAIL FROM. I don't know of any of the
former and the latter is a *really* bad idea. DSNs are just messages that
arrive from some source, and that source is either appropriately authenticated
or it isn't. Yes, you can make make additional choices once you've received the
message based on the  content being a DSN, but that's no different than any
other sort of content-based decisions. This isn't X.400, where DSNs aren't
actually messages.

I therefore strongly recommend dropping the entire paragraph except for the
first sentence.

And that's about it. I will say that having implementing the draft in what
I think is is a useful way, I'm a lot more confident that this extension
makes sense.

I'll try to follow up on some of the other traffic on this topic in a bit.

				Ned