Troubleshooting SSL related issues with IIS

In this post I am going to discuss troubleshooting SSL server certificate related issues in IIS. Please remember I am no SSL expert...I am an IIS Asp.Net guy and will discuss issues with regard to SSL with IIS.

We have seen a lot many times that the issues could have been resolved by our customers if they were aware of some targeted troubleshooting techniques.

Here I will be walking through the steps which you can follow yourself before requiring to call Microsoft PSS.

Problem Description

The most common error associated with SSL not working for an IIS website is "The page cannot be displayed" or "Cannot find server or DNS Error".

Troubleshooting steps to be followed:

First thing first: You notice the above error when you browse to your website over SSL (HTTPS). Do you also see the same behavior when you access the website over HTTP? If yes then there is a different problem you need to address first. SSL doesn't come into picture when we access a site over HTTP.
So if you are facing the problem in both the scenarios ensure you first troubleshoot why the request is not reaching the IIS server...maybe a DNS configuration etc.

My topic today is SSL with IIS so I won't be focusing on HTTP.

We have the SSL port listed in IIS mmc as shown below. By default it's 443. You can configure it to listen on a different port.

image

Now, also check whether the issue is happening for all the users, i.e. External and internal users to the network. If the issue is happening only for let's say external users check whether the requests are coming over a firewall like ISA etc. At times Firewalls may block the SSL port. You may want to check the firewall settings to unblock the SSL port. Also firewalls like ISA requires certificate to be published on it as well apart from IIS server. Ensure that the certificate that we have installed on ISA is valid. In case of ISA, if you have renewed the cert on IIS server but not on ISA users will still see the old certificate and not the new certificate. Ensure that you replace the old certificate on ISA with the new certificate. Also you may want to check if the security gateway (if any) such as NFUSE (Citrix) controllers is still referencing the old certificate.

Also at times if you see a wrong certificate being displayed for your Web Site, ensure that you do not have kernel-mode SSL enabled on your web server.Although this improves the performance but I recommend to have it disabled when you have multiple SSL enabled-websites on your server. Kernel-mode SSL is only recommended if you have just one site, mostly static content and no client certificates. In all other cases you should use User-mode SSL.

Please find this link to enable/disable kernel-mode SSL. If it is set to 1 it means Kernel-mode SSL is enabled, if absent or set to 0 it means disabled and user-mode SSL will be used. Ensure that you stop and restart http service as shown below after doing the changes. No reboot is required for the above change.

>net stop http

>net start http

>net start w3svc

[There has been few concerns from people as to why this is suggested. I
do not have a proper answer to it at this point, but I have seen such
issues in some support incidents where disabling Kernel mode SSL had
resolved the above problem]

------------------------------------------------------------------------------------------------------------------------------------------------------------------

Now once you have isolated the issue to be only with HTTPS and all users let's proceed with the following steps:

Run netstat -an (or fport.exe) for IIS 5 (and netstat -ano for IIS 6) and verify whether IIS is listening on the SSL port.

When you start Internet Information Services (IIS) 6.0 on Microsoft Windows Server 2003, IIS binds to all IP addresses on the server, not just the IP addresses that are assigned to Web sites. This is because of socket pooling enabled for IIS by default.

On IIS 6.0, if the website is listening on "All unassigned" IP address or some specific IP address (like 192.134.123.209 etc) and SSL port 443, verify that Local address entry is 0.0.0.0:443 in netstat output.

> netstat -ano

Active Connections

  Proto  Local Address          Foreign Address        State           PID
  TCP    0.0.0.0:21             0.0.0.0:0              LISTENING       2396
  TCP    0.0.0.0:25             0.0.0.0:0              LISTENING       2396
  TCP 0.0.0.0:80 0.0.0.0:0 LISTENING 4
  TCP    0.0.0.0:81             0.0.0.0:0              LISTENING       4
  TCP    0.0.0.0:83             0.0.0.0:0              LISTENING       4
    ...................

  TCP 0.0.0.0:443 0.0.0.0:0 LISTENING 4
  TCP    0.0.0.0:445            0.0.0.0:0              LISTENING       4
  TCP    0.0.0.0:446            0.0.0.0:0              LISTENING       4

Here the PID should show you which process is listening on that port.
Now let's say I have configured my website to run on a specific SSL port and that port is being used by a different process, then website will not start up.

Also you won't see the netstat output showing anything like this depending upon the port being used:

TCP    0.0.0.0:80             0.0.0.0:0              LISTENING       4

TCP    0.0.0.0:443           0.0.0.0:0              LISTENING       4

You can run netstat -ano again and see the PID corresponding to the process which is listening on that port. Stopping or disabling that service/process should allow Websites to be up and listening on the desired port. For IIS 5.x you can run Fport.exe to find out which process is listening on a specific port.

The first thing that I check for (strictly my personal preference) when we have issues with SSL is MACHINEKEYS. Yes, this is a very important (I prefer to call strategic) part of your checklist while dealing with SSL issues. In my personal experience it has resolved a lot of issues for our customers. By default machinekeys folder should have permission for Administrator, System and Everyone according to this KB278381. Ensure that we inherit the above permission for all child objects under the machinekeys folder, that way all the machine keys have the necessary permissions.

This is done by clicking the "Advanced" button on the security settings page, Clicking the check box "Replace permissions entries on all child objects...." and
Apply. This will propagate the settings to the contents in the folder.

Another reason for the above error ('The page cannot be displayed') can be if you do not have a private key corresponding to the SSL server certificate. You can check that by opening the IIS mmc -> <<YourWebSite>> ->Properties ->Directory Security ->Secure Communications ->View certificate ->General -> "You have a private key that corresponds to this certificate" as shown below.

image

If you do not see the above then SSL won't work for your website.
You can try running the following command to recover the private key from the certificate thumbprint.

To associate the certificate with its private key, we run the following command:

> certutil -repairstore my "c9 66 67 68 49 97 88 7f 05 b8 89 b7 b3 f7 37 c8 c4 da 5f 16"

Note: "c9 66 67 68 49 97 88 7f 05 b8 89 b7 b3 f7 37 c8 c4 da 5f 16" is the Thumbprint of the missing certificate. You can see this if you double-click on the .cer file, choose the Details tab and select Thumbprint.

image

Make sure you use the 2003 version of certutil.exe (with the associated certadm.dll and certcli.dll), or you will not have access to the repairstore command.

If the above command does not get back the private key or for some reason it still does not work you may prefer to get a new certificate from the CA.

One of the best tools that Microsoft has come up with is SSL Diagnostics to troubleshoot SSL related issues. Just download it to the IIS server and run it and if there is an error due to configuration of IIS or certificate it will show up. You can also use this tool to issue a test certificate for your website to check whether the problem was occurring because of your server certificate or some other IIS configuration related issues. Things that can be shown quickly through this tool includes on "no private key", "SSL port being used by a different process", "machine keys not having enough permission", "IP address conflicts" etc.

Another scenario can be wherein if you view the certificate, it will state that "You have a private key that corresponds to this certificate," and if you run SSL Diagnostics, you'll get the error as below:

#WARNING: You have a private key that corresponds to this certificate but CryptAcquireCertificatePrivateKey failed

When you initially go through the IIS Certificate Wizard to create a new certificate, a file is added to the MachineKeys folder. This is the private key.

If you add permissions to the new file in the MachineKeys folder before you process the pending request in the IIS certificate wizard, the certificate will work properly. If you've already gone past that point and got a new certificate, then necessary permissions should be added to the MachineKeys folder and a new certificate request must again be submitted.

Now consider a scenario wherein you are able to access the site through SSL Diagnostics tool (through 'Simulate SSL Handshake') for the same certificate but not through IE. What I mean is your SSL handshake gives no error etc in SSL Diagnostics although when you access the website through IE over SSL it still fails. In that case there could be an IE setting that you may want to check for such a behavior. Check this out KB811834.

Now one last scenario. At times we see customers coming up with requests wherein they want to use the same server certificate for multiple websites.

This is not possible. You cannot use the same certificate for multiple websites on the same server. Check this KB187504.

Now there is a workaround in IIS 6.0 Windows 2003 SP1, wherein you can use Wildcard certificates to get a similar functionality. Here is the link.

If you have a Load balanced or clustered environment you can export the same certificate to all the servers and install them on the related websites. It should work fine.

Hope this helps....let me know if there are any questions around the topic.

Cheers!!!