Customer was running a WCF application on WinXP. His WCF app messages were ending up in the deadletter queue with error "The Signature is Invalid."
We changed the security Configuration to
<transport msmqAuthenticationMode="None" msmqProtectionLevel="None"/>
Now we get access is denied error
Thanks to James Birdsall, we were able to resolve the problem. The problem was caused due the fact that Win2k8 rejects a MD5 Signed document from XP Sp2. Inorder for the Win2k8 and higher to accept messages From Win2k, Win2k3 and WinXp you will have to add the following Registry Key (NOT VALUE)
NOTE: If you are receiving messages on a cluster then you need to create the key on both the physical nodes and the cluster node.
This registry key allows the Win2k8 Box to Receive MD2, MD4, MD5 Messages. If you want to be selective and say do not want to accept MD5 messages but accept MD2 and MD4 messages then you can add the following registry value
DisableMD5 dword value 0x8003
For more details refer to article http://msdn.microsoft.com/en-us/library/ms706984(VS.85).aspx and James Birdsall's (developer) blog at http://blogs.msdn.com/motleyqueue/archive/2007/09/29/authentication-and-msmq4.aspx