CryptographicException: The data to be decrypted exceeds the maximum for this modulus of 0 bytes
Hi all,
Some time ago a customer of mine was getting the following CryptographicException when calling EncryptedXml.DecryptDocument method to decrypt an XmlDocument in their .NET app:
[System.Security.Cryptography.CryptographicException] = {"The data to be decrypted exceeds the maximum for this modulus of 0 bytes"}
_HResult = -2146233296
_HResult = 0x80131430
No Inner Exception
This was only happening with a smart card certificate associated to a specific third-party CSP (non-MS).
In order to troubleshoot this, I used my CryptoAPI Tracer script to get some traces while reproducing the issue. Thanks to my script I could see these calls that .NET is making to the third-party CSP behind the scenes (I will omit key container and CSP names as they are not needed to understand the issue):
>>>>>>>>>>>>>>>>>>>>>>
CryptAcquireContextA (0x570)
IN
pszContainer
001bf1a0 "Cert Container Name"
pszProvider
001b4698 "Third-party CSP"
dwProvType
PROV_RSA_FULL
dwFlags
0
OUT
hProv
0x1bea28
RESULT
CryptAcquireContextA (0x570) SUCCEEDED
<<<<<<<<<<<<<<<<<<<<<<
>>>>>>>>>>>>>>>>>>>>>>
CryptGetUserKey (0x570)
IN
hProv
0x1bea28
dwKeySpec
AT_KEYEXCHANGE
OUT
hUserKey
0x1bf1a0
RESULT
CryptGetUserKey (0x570) SUCCEEDED
<<<<<<<<<<<<<<<<<<<<<<
>>>>>>>>>>>>>>>>>>>>>>
CryptGetKeyParam (0x570)
IN
hKey
0x1bf1a0
dwParam
KP_ALGID
pbData
NULL
dwDataLen
0
dwFlags
0
OUT
dwDataLen
4
RESULT
CryptGetKeyParam (0x570) SUCCEEDED
<<<<<<<<<<<<<<<<<<<<<<
>>>>>>>>>>>>>>>>>>>>>>
CryptGetKeyParam (0x570)
IN
hKey
0x1bf1a0
dwParam
KP_ALGID
pbData
0x1bd228
dwDataLen
4
dwFlags
0
OUT
bData
001bd228 0000a400
dwDataLen
4
RESULT
CryptGetKeyParam (0x570) SUCCEEDED
<<<<<<<<<<<<<<<<<<<<<<
>>>>>>>>>>>>>>>>>>>>>>
CryptGetKeyParam (0x570)
IN
hKey
0x1bf1a0
dwParam
KP_KEYLEN
pbData
NULL
dwDataLen<br>0
dwFlags
0
OUT<br>dwDataLen<br>4
RESULT
CryptGetKeyParam (0x570) SUCCEEDED
<<<<<<<<<<<<<<<<<<<<<<
>>>>>>>>>>>>>>>>>>>>>>
CryptGetKeyParam (0x570)
IN<br>hKey<br>0x1bf1a0
dwParam<br>KP_KEYLEN
pbData<br>0x1bd228
dwDataLen<br>4
dwFlags
0
OUT<br>bData<br>001bd228 00000000
dwDataLen<br>4
RESULT
CryptGetKeyParam (0x570) SUCCEEDED
<<<<<<<<<<<<<<<<<<<<<<
These calls mean the following: before .NET tries to decrypt the XML, it asks the CSP for some information on the key it will use to decrypt, like its length (CryptGetKeyParam API with KP_KEYLEN flag). For some unknown reason the third-party CSP is returning a length of 0 for the key (pbData points to 0), which is invalid, thus .NET won't continue with the decryption and will raise the exception we've seen. Why is the CSP returning an invalid length? Only the CSP developers can answer that question.
In my customer's particular case, the CSP developers quickly localized and fixed the issue.
I hope this helps.
Regards,
Alex (Alejandro Campos Magencio)