When connecting to an SSL-enabled web site with Internet Explorer, the client and server must negotiate an SSL session. The server sends its list of trusted root certificates to the client in the form of a non-encrypted record. The server requires that the client have a digital certificate for authentication, the client is able to select one that corresponds to a root certificate trusted by the server.
The problem appears when the root certificate list sent by the server exceeds the 16k size limit. This is defined in RFC 2246 (http://www.ietf.org/rfc/rfc2246.txt). When the list exceeds the 16k limit, the server sends several packets. Now, Internet Explorer relies its security to the operating system via Schannel. Schannel on Windows XP / Server 2003 and Windows Vista / Server 2008 is not able to process those packets by default. So when Internet Explorer tries to connect to the SSL-enabled web site, the connection fails.
This issue won’t happen on Windows 7 / Server 2008 R2, as they support record fragmentation in Schannel. They can manage the rest of the packets sent by the web server with the rest of the trusted root certificate list.
Fortunatelly we recently released an update to overcome this limitation for older versions of the OS:
Note this update should be available as a High Priority update in Windows Update, too.
Alex (Alejandro Campos Magencio)