CryptEncrypt and RSACryptoServiceProvider::Encrypt

The RSACryptoServiceProvider class provides two methods, Encrypt and Decrypt which seem to be the managed counterparts to CAPI's CryptEncrypt and CryptDecrypt functions.  However, if you try to encrypt using CAPI and decrypt using managed code, you'll end up with a CryptographicException saying "bad data".  What's going on here?

Since v1.0 of the runtime, the output from RSACryptoServiceProvider::Encrypt has been reversed in memory.  Input to RSACryptoServiceProvider::Decrypt is also swapped before it's processed.  Since the two reversals undo each other, using the RSA class to round trip data works perfectly.

However, if you use CAPI for one of the steps, the memory reverse won't undo itself, and the output won't contain valid padding -- in this case leading to a bad data exception.  The solution is to introduce the reversal yourself.  To interop, you'd need to either:

  1. CryptEncrypt
  2. Reverse
  3. RSACryptoServiceProvider::Decrypt

Which gets the output from CAPI setup so that Decrypt's own memory reverse puts the bytes in a state that they can be decrypted.  Going the other direction would of course be:

  1. RSACryptoServiceProvider::Encrypt
  2. Reverse
  3. CryptDecrypt

The Reverse in this direction undoes the reversal that Encrypt does.  Reversing the memory can be done with a call to Array.Reverse();

Why not change the behavior of Encrypt and Decrypt now?  Doing that would prevent people using the same code on different versions of the runtime from being able to interop with each other.  Even worse, if you wrote some data out to a file while your app was running on a version of the runtime that had the old behavior, and before your app started up again the computer admin uninstalled the old version of the runtime and installed a new version which didn't have this behavior, when your app started back up it wouldn't be able to read its file anymore.

Instead, when writing code that needs to interact with CryptEncrypt and CryptDecrypt, you need to remember to apply this transform to your data in order for your code to work properly.

Updated 11:19: Added a reference to Array.Reverse

Comments (10)

  1. Garry Trinder says:

    Know your libraries.

    Use System.Array.Reverse

    It’s two times faster and bug-free.

  2. Thanks TAG 🙂 Speed in this case is probably not a huge factor, considering that the maximum size of this array is on the order of 100 or so bytes, but I agree using a built in method is better.


  3. Eric W says:

    Was the byte reversal a design decision or just a bug that slipped through in v1?

  4. I’m not sure — I wasn’t involved in crypto in the v1.0 timeframe, and everyone who was has moved on at this point.


  5. Maxwell Sandford says:

    Lame! Why don’t you just give us DecryptEx() with a boolean parameter to specify interop reversal is needed. Of course the programmer can simply wrap Decrypt() to accomplish the same thing–most will do so–but you can fix this on your end without programmers having to write code that fails and then research the topic on their own.

  6. We certainly could add something along the lines of DecryptNoReverse … however we don’t get much customer feedback on this interop issue, so it’s not high on our list of priorities. If you’d like it fixed, please file a bug (and get others to vote) in the MSDN Product Feedback Center — if it turns out that this issue does affect many people, we can look at it for a future release.


  7. Nick Jones says:

    I think we might be having a problem with this.

    We are finding an XML document signed with an RSA key in .NET 1.1 no longer verifies in .NET 2.0. Further investigation revealed that the .NET 1.1 signing was wrong.

    Is it possible this is being caused by what you are describing is due to RSACryptoProvider reversing some data somewhere? If so, interopobility so we can check signatures on documents signed in .NET 1.1 would be helpful!

  8. Nick,

    I don’t think this is the root of your problem. This issue aaffects the Encrypt method only. In XML DSig we use the RSAPKCS1SignatureFormatter to create the signature, and that relys on the SignHash method.


  9. I spent 3 days trying to figure out whey decryption of CryptEncrypt output is not working

    I founded by chance when compared the data (byte by byte)

    Then I stopped in your blog. I felt upset. Such an important issue should be clearly mentioned in the API/Framework documentaion

    what a mess!!!!

  10. TRVQ2 says:

    Can somebody send to me an compatibility example with C# and win32 source code? We have an issue and are trying to resolve. Thanks!

Skip to main content