"Failed to decode the S/MIME message. The S/MIME message may not be valid." when using POP adapter to process incoming emails in BizTalk server 2009

"Failed to decode
the S/MIME message. The S/MIME message may not be valid." when using POP
adapter to process incoming emails in BizTalk server 2009

Considering the
following scenario:

You are using the POP3 adapter to process incoming e-mails
in BizTalk server 2009, and there is a sporadic failure with certain e-mail
messages. These failures result in suspended resumable messages with the
following error information:

A message received by adapter
"POP3" on receive location "POP3ReceiveLocation" with URI xxx
is suspended.

Error details: There was an
authentication failure. "Failed to decode the S/MIME message. The S/MIME
message may not be valid.".
 

You have already found the KB article 982634: https://support.microsoft.com/kb/982634/en-us
, but it does not apply to you, as you had BizTalk server 2009 CU2 deployed
already. The symptoms are the same however.

Analysis:

When we took a BizTalk trace to investigate into this issue,
below information could be observed:

=====================================

[CSharp]:MIME_SMIME_Decoder.Execute2
entered

[CSharp]:MIME_SMIME_Decoder.Execute2:
failed could not find mimeversion header.

[CSharp]:MIME_SMIME_Decoder.Execute
caught MIME exception failed Message:Exception of type
'Microsoft.BizTalk.Component.MIMEException' was thrown.

 

==========================

 

We noticed that it was not detecting the version of MIME.
Further investigation revealed that a registry key was being read to determine
the amount of data to be read before the MIME version was determined and that
defaults to 4kb. This was done for performance reasons.

Solution:

To resolve this issue, please do the following:

Add the following registry key with the following value [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc.3.0]

"BytesToScanForMimeVersion"=dword:00019000

Hex19000/Dec102400 is
the maximum value for this registry value.

In BizTalk versions before 2010, only the first 4KB of the
message MIME headers are scanned to get the MIME version. Thus, messages with
excessively long headers will not get decoded.

 

Best regards,

Rachel Huang