The stages of conversion for Internet Mail

As Naresh pointed out in his history of content conversion blog, Exchange 2000 swapped the roles of our primary transport from X.400 to SMTP. As such we also more closely integrated in native MIME formatted message into Exchange store by introducing the .STM file.

We are unfortunately bound by the fact that our richest client (Outlook) still requires messages to be presented in a MAPI format. Up until Exchange 2000, message conversion took place immediately upon receipt of the Information Store (Exch5.5), or when received by the IMC (Exch4.0 and Exch5.0). With Exchange 2000 we introduced a delayed conversion model whereby full conversion of the message would only take place when it was absolutely needed. This introduced several phases of conversion.

The first phase is when a message is first received in native SMTP/MIME format and routed into the Information Store, then entire message is referenced by pointer to the message in the STM file. At this point, the message is lives in the SMTP temp tables while the delivery of the message is being handled.

In the next phase we parse enough details from the SMTP header message, such as Sender, Date, Receiver, etc. to render the presence of a message in a folder. (think about the information you see when looking at a list of messages in Folder in Outlook) At this stage we also take a look over the MIME data to determine if we have any attachments and if any of the attachments are of type ms-tnef. If a TNEF type is encountered, then all aspects of the message are immediately converted into MAPI format. Such as when a message has arrived with the same organization between two Outlook Clients. Otherwise, we determine if any attachments exists to accurately display the “paper clip” icon beside the message in Outlook. At this point the message is delivered into the correct Inbox folder based upon the recipient.

When the an Outlook client attempts to open the message, we simply convert the appropriate bodypart of the MIME message on the fly from MIME to RTF and present it to the client. In addition we reparse the MIME data to retrieve the details of the attachment from the MIME bodypart header and promote those into the EDB file. When the client closes the message, the message body still remains in the STM file and the attachment details remain in the EDB. If the client attempts to forward the message, all aspects of the message are converted from MIME to MAPI format and the representation of the message in the STM file is removed if no one else is using it.

If the client attempts to open an attachment, all aspects of the attachment are promoted to the EDB file before being presented to the client. Note that during this time the original MIME message still resides in the STM file, including the attachment information to preserve the original MIME until complete conversion has been performed. The promotion is necessary to maintain the integrity of the original MIME data.

Finally if the client modifies an attachment or message body, the entire message is promoted to the EDB and references to the original MIME stream are removed from the STM file. A forward operation is an operation that would also force the complete conversion of the message.

Such is the life of an inbound message in Exchange…