Last month, I worked on an incident where a customer couldn't access a web site from Internet Explorer 9 using TLS 1.2. Accessing the same web site was working fine using TLS 1.0. I first thought the issue was similar to the one described in this blog: http://blogs.msdn.com/b/ieinternals/archive/2011/03/25/misbehaving-https-servers-impair-tls-1.1-and-tls-1.2.aspx
However, after gathering a network trace, I couldn't see any RESET coming from the server so we were hitting another scenario than the one from above blog.
Activating SCHANNEL ETL tracing and SCHANNEL event logging (as shown here: http://support.microsoft.com/kb/260729) was showing a cryptic SSL fatal alert event 36888 "The following fatal alert was generated: 40. The internal error state is 252":
Since the issue could easily be reproduces, I debugged it and found out that the TLS 1.2 handshake failure was caused by extra security checks made on signature algorithms used in the server certificate chain. Specifically, the following certificate chain was sent in the server's HELLO and the use of MD5 algorithm was causing the handshake failure:
TLS 1.2 allows the ability to specify the signature algorithms that an end point is willing to accept (http://tools.ietf.org/html/rfc5246#section-188.8.131.52.1 ). This is specified in the client/server hello extension.
Here's the client hello for our IE9/TLS 1.2 scenario:
Unfortunately, neither Netmon nor Wireshark provides parsing for the signature_algorithms field but you can do "manual parsing" using the RFC:
Type: signature_algorithms (0x000d)
Data (16 bytes)
04:01 sha256(4) rsa(1)
05:01 sha384(5) rsa(1)
02:01 sha1(2) rsa(1)
04:03 sha256(4) ecdsa(3)
05:03 sha384(5) ecdsa(3)
02:03 sha1(2) ecdsa(3)
02:02 sha1(2) dsa(2)
As you can see md5 is not listed. This is due to the fact that schannel.dll restricts the use of md5 for security reasons:
[abstract of http://tools.ietf.org/html/rfc5246]
MD5 [MD5] is a hashing function that converts an arbitrarily long
data stream into a hash of fixed size (16 bytes). Due to
significant progress in cryptanalysis, at the time of publication
of this document, MD5 no longer can be considered a 'secure'
Currently, TLS 1.1 & 1.2 are not enabled by default in Internet Explorer but this may likely change in future versions.
So, be aware that MD5 is no longer considered secure and it's probably time to get rid of your certificates with MD5 signature to avoid problems in the future!