Encryption and MSMQ4

  • Article

Just as with authentication, the security sweep for Vista and Windows Server 2008 had an effect on MSMQ's encryption feature. In previous versions, MSMQ offered two "privacy levels", BASE and ENHANCED. The two levels reflected the length of the keys used: BASE used 40-bit keys and ENHANCED used 128-bit keys. Both levels allowed a choice of two algorithms, RC2 and RC4, with RC2 being the default. Under the new rules, both 40-bit keys and RC2 have been deprecated as weak, and even the combination of ENHANCED and RC4 is regarded as marginal.

In response, MSMQ4 added a new privacy level, MQMSG_PRIV_LEVEL_BODY_AES, which supports three values for the "encryption algorithm": 128-, 192-, and 256-bit keys. This is obviously a bit of a reversal of the previous model: the privacy level is now specifying an algorithm and the encryption algorithm is actually determining the key length.

What are the implications for your existing applications?

For messages being sent from older versions to MSMQ4, it's very smooth sailing. MSMQ4 will accept messages encrypted with any length of key and any algorithm. No code changes to your application or configuration changes on your MSMQ4 server are required.

For messages being sent from MSMQ4 to older versions, it's also smooth sailing. As with authentication, MSMQ4 promotes the privacy level requested by the application and then uses the default algorithm for that level (now RC4 for BASE or ENHANCED, or a 256-bit key for AES). Previous versions did not support AES encryption, but compatibility is automatic: as part of the encryption feature, MSMQ on the sender has always had to retrieve the destination's public keys from Active Directory, and this information allows MSMQ4 to distinguish older servers from MSMQ4 servers so that an appropriate privacy level can be selected. If the destination supports AES, that is used; for older versions, ENHANCED is preferred, but MSMQ4 can still fall back to BASE if the destination supports nothing better. However, MSMQ4 will never use a privacy level lower than that requested by the application.

Issue: I'm writing a new app or modifying an existing one. What privacy level or algorithm should my code request?

Solution: Request ENHANCED with RC4. This combination allows MSMQ4 to promote to AES when the destination is also MSMQ4, while also allowing it to use ENHANCED when the destination is running a previous version but preventing it from falling back to BASE. Furthermore, if your app will also run on previous versions of MSMQ, it will be using the most secure combination that those versions support.

Issue: If BASE is so weak, how can I prevent my existing applications from using it?

Solution 1: If your existing applications request ENHANCED, you're fine, both on MSMQ4 and on earlier versions.

Solution 2: If you have applications which request BASE, or you aren't sure, you can turn off BASE at a lower level. MSMQ4 introduces registry values which are used to control the default algorithms for the three privacy levels. Setting a value to 0 turns off that privacy level entirely. If you don't want applications ever using BASE, you can turn it off, but be sure you don't need it first!

Issue:  My application absolutely depends on using ENHANCED encryption with RC2 (for example). How can I get it working with MSMQ4's promotion?

Solution: For this example, you would turn off AES by setting its registry value to 0 and set ENHANCED's registry value to RC2. Don't do this if you don't have to, because these settings apply to all applications using MSMQ on that machine!

Issue: I have some really old installations (some versions of NT4 or Windows 2000, or Windows 98) which may only support BASE.

Solution: That's fine. Your applications must be requesting BASE in order to be sending to those machines, and MSMQ4 will fall back to BASE as long as it hasn't been turned off in the registry.

Issue: What happens if MSMQ4 can't find a level which is as good as or better than that requested by the application and supported by the destination?

Solution: If you requested acknowledgements and/or deadlettering, then the message will be NACKed and/or deadlettered. The message class of the NACK/deadletter will be 0x8008, MQMSG_CLASS_NACK_COULD_NOT_ENCRYPT, "The message could not be encrypted for the destination."

 

The registry values, if required, are created under the key HKLM\SOFTWARE\Microsoft\MSMQ\Parameters\security. The names of the values are BaseEncryptionAlg, EnhancedEncryptionAlg, and AesEncryptionAlg, respectively. The values, in hexadecimal, are:

  • RC2: 6602
  • RC4: 6801
  • AES 128-bit: 660E
  • AES 192-bit: 660F
  • AES 256-bit: 6610

Remember that the RC2/RC4 values are only meaningful in BaseEncryptionAlg and EnhancedEncryptionAlg, and the AES values should only be used in AesEncryptionAlg.