Just a quick note that I've updated the examples. I added an example for the CAPI RC4 encryption that does work. Along the way, I got smarter about managed C++ and C# interop, which turned out to be a bit of an adventure. I didn't find the documentation on MSDN exceptionally helpful in this area. Maybe there's a good book on the topic, but I haven't found it yet. We had a huge amount of snow for this area – accumulations of about 2-3 feet at my house, and I couldn't go anywhere, so that's what I ended up doing.
The reason for the interop adventure was that there is no RC4 implementation in .NET. There's a couple of those on the Internet, but I didn't want to burden people with getting and installing some 3rd party library. Along the way, I figured out a mystery of CAPI RC4 encryption. Turns out that if you set a 40-bit RC4 key, there are 3 modes of operation:
- CRYPT_NO_SALT – nothing is added to the key. Office doesn't set this flag.
- Salt added – some bunch of random bits from the hash is added to the key – seems only useful for temporary keys. We don't use this one, either.
- Default – basically, if you import a 5 byte (40-bit) key, it is the exact same thing as importing a 16-byte (128 bit) key where the last 11 bytes are all 0, since CAPI does that for you under the covers unless you set CRYPT_NO_SALT.
The sample code that I have now makes this explicit, and we'll update the document to make this clear as well. This only applies to 40-bit
A last thing that ought to be tidied up is that when you decrypt the two parts of the verifier (encrypted salt and encrypted hash of salt), you do not reset the RC4 stream between encryption operations.
I'm working on doing the legacy RC4 example, and when that's done, it will be posted to www.codeplex/offcrypto. After that, I'll probably move on to the new encryption and some of the signing. If you're interested in this, I'd suggest signing up to the RSS feed on the offcrypto project on CodePlex – I may not remember to post here when I change things.