"ASN1 value too large" error when calling SignedCms.ComputeSignature

Hi all,

When signing large files ( > 100 MB) with SignedCMS.ComputeSignature in a .NET 3.5 SP1 application, you may get the following exception:

System.Security.Cryptography.CryptographicException: ASN1 value too large 

Note you will get the same exception on x86 and x64 systems. Why?

ComputeSignature method ends up calling CryptMsgOpenToDecode & CryptMsgUpdate CryptoAPIs on the message it just created with CryptMsgOpenToEncode & CryptMsgUpdate to do some verifications.

It turns out that CryptMsgUpdate API has a limit on the size of the data that we can pass to it when decoding (when used along with CryptMsgOpenToDecode). This limit is 1024 * 1000 * 100 bytes (around 100MB), and it was set starting on Windows XP SP2 and Windows Server 2003 SP1. This applies to all versions of Windows after that, regardless of being x86 or x64.

When signing large files with SignedCms, we end up passing more data to CryptMsgUpdate than allowed when decoding, so we fail with the "ASN1 value too large" error.

Now, CryptMsg API supports streaming of data to be able to work with large amounts of data (check CryptMsgOpenToEncode & CryptMsgOpenToDecode & CMSG_STREAM_INFO Structure for details). The issue is that current version of .NET doesn't use streaming when calling CryptMsg API. This feature will be considered in a future version of the Framework.

So summing up, if we want to CMS sign large amounts of data with .NET, we will have to p/invoke CryptMsg API and use it in streaming mode ourselves. With this approach I've been able to sign 1GB files (and even bigger in x64 systems). I've posted a C# sample which shows how to do this: How to call CryptMsg API in streaming mode (C#).

Regards,

Alex (Alejandro Campos Magencio)