When wrong permissions affect cipher being used for TLS connection

  • Article

The information from this blog POST has been compiled with the help of the following engineers:

  • Daniel JUE – SNCF
  • Mathieu CLAUDEL – KNS
  • Marc BOUCHARD - Microsoft Corporation

I recently worked on an interesting SSL issue which manifests itself by the inability to negotiate a TLS connection using RSA with 2048 bits key exchange when connecting to a specific set of IIS 7.5 servers. The problem's symptoms were different depending on the client computer and browser version used:

  • On a client computer running Windows 7 and Internet Explorer 9, SSL connection could be established but only using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013) cipher suite. Under Internet Explorer's File/Properties menu, the following details were provided:

    The same client could connect without any problem to any other IIS server using TLS_RSA_WITH_AES_128_CBC_SHA (0x002f) which could be checked in Internet Explorer:

  • On a client computer running Windows XP with IE6, SSL connection could just not be established and the generic "Page cannot be displayed" error was shown : the XP client was simply failing during the SSL handshake

Since the issue was only affecting a couple of servers, we first though the problem was due to bad configuration and we started to check if specific cipher suites were disabled on the affected servers (see How to Restrict the Use of Certain Cryptographic Algorithms and Protocols in Schannel.dll). It tuned out no cipher suites were disabled and the affected server were just using default SCHANNEL settings.

One interesting thing pointed by our customer is that the SSLDiag report was showing this error when trying to display the PrivateKey size :

SERVER SSL PROTOCOLS
PCT 1.0 : Enabled
SSL 2.0 : Enabled
SSL 3.0 : Enabled
TLS 1.0 : Enabled
…

#CertName :
#Version : 3
#You have a private key that corresponds to this certificate.
#Signature Algorithm : sha256RSA
Error : PrivateKey.KeyExchangeAlgorithm PrivateKey.KeySize Keyset does not exist

Unfortunately, the error above was pretty cryptic but I should have probably paid more attention to it ("Keyset" refers to the certificate's private key)…

After gathering some SCHANNEL ETL traces, my colleague Marc pointed that the KeySpec for the certificate was invalid. This could also be seen in the certutil output for the certificate :

================ Certificate 2 ================
…
Subject:
    CN=xxxxxx…

CERT_KEY_PROV_INFO_PROP_ID(2):
    …
    Provider = Microsoft Enhanced Cryptographic Provider v1.0
    ProviderType = 1
    Flags = 20
    KeySpec = 0

The expected value for a SSL Certificate is to have a KeySpec of 1 which means AT_KEYEXCHANGE as explained in the SCENARIO 3 section of this excellent article : Error HRESULT: 0x80070520 when adding SSL binding in IIS.

At this point, we tried to reimport the certificate from a PFX file but this didn't help: the certificate could be imported without error but the KeySpec was still invalid (0 value). We then tried to force the KeySpec during import using "certutil -importPFX PFXFile AT_KEYEXCHANGE" and we also started a PROCMON trace to check for potential issue during the certificate import. Bingo! We could see a couple of ACCESS_DENIED during access to C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys. After checking the permissions of this directory, we saw that the default permissions were not the same as on a working server. After fixing the permission issue, the certificate import succeeded and PROCMON didn't show any ACCESS_DENIED errors. We then stopped HTTP.SYS (net stop http) and restarted IIS (net start w3svc) and, finally, we could connect to the server using TLS 1.0 with RSA 2048 bits key exchange.

The permission issue on C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys was preventing to get the right KeySpec (AT_KEYEXCHANGE) for the RSA private key. Therefore, RSA couldn't be used for key exchange but could still be used for signature which is why TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA was used.

We hope the above blog may help other customers hitting similar issue.

Best wishes for the upcoming year from the French IIS Support team!

Emmanuel Boersma