Kerberos troubleshooting from IIS perspective
Hi All,
This is my first posting in the blog.
I really had to take enough courage to start blogging, but with some help from one of my mentors in MS, I am finally here.
I hope people really get benefitted from the articles that I post in here.
Today, I am going to talk about how to implement Kerberos authentication for IIS. I have chosen this topic after a lot of consideration. This topic has always evoked a state of anxiety and fear among Web administrators and MS PSS support engineers alike. Also this has been a pain for us since a lot of calls that we receive are related to Kerberos authentication failure and causes a lot of labor and revenue loss to MS and customers.
So here it goes...
So what exactly is this Kerberos, first time when I heard of it I thought it must be some mystical word related to enchantment and what not. I went and looked into the dictionary and found something similar. It meant a fierce three-headed dog figure from Greek mythology that guarded the gates of the underworld. Kerberos protocol, similar to the dog figure has three main sections: client, server and an intermediary called Key distribution centre (KDC).
So how exactly is this Kerberos protocol work:
There are numerous articles that you can find which will give you an insight as to how Kerberos protocol works, so instead of explaining some redundant stuffs here which might confuse you a bit more (like the way it did to meJ), I will be very lucid and straight in my explanation, and concentrate more on troubleshooting than getting into various jargons associated with it. Let me know if you need articles on the topic and I can post it here.
Simply speaking, a client requests a ticket (or token) from an Intermediary called Key Distribution centre (KDC) for accessing any service registered with it. In our case to access a web service, it looks for an authenticated token to access the web services, and once getting a unique short-term session key from the KDC, directly contacts the IIS server hosting the web service. IIS in turn receives the token its own session key and since it is authenticated by the KDC with which the IIS service has been registered authenticates the client to access the application running on it. Remember authentication and authorization are two different terms. A client might be authenticated still might not be able to access the resource because of lack of authorization to access it.
To understand the details of how a Kerberos protocol works, I recommend reading MS knowledge base or Technet.
Before you configure Kerberos authentication for your site, I recommend having these tools handy:
SETSPN (For adding, listing, deleting SPN entries for a domain)
KERBTRAY (For checking the Kerberos ticket used by the client to access a web server. It gives you information as to which ticket is being used by the client to access the IIS server, and whether the ticket is capable of delegation).
I will take up a scenario where you want to implement Kerberos delegation to work in this architecture. It is also called double-hop since client's credentials are hopped twice from the client to the IIS web server to the backend SQL server to access a resource. I suggest there are very good articles present elsewhere on Microsoft site where you can get in depth information on how Kerberos authentication works. I have concentrated more on troubleshooting, so you can skip to the next section J.
We assume client, IIS server and the backend SQL server in the same domain. The same scenario will also work if you have the components in different domains but they are mutually trusted both ways.
As a troubleshooting process, start with only Basic authentication enabled on IIS server and then test from a client machine to see if that works successfully. If it works, we are good to proceed further with Windows integrated authentication as the only enabled authentication on IIS (make sure that we do not have Anonymous authentication selected in the IIS mmc console).
The following checklist gives you an insight of the basic configuration required for Kerberos to work in double hop scenario from IIS perspective.
Non-NLB Scenario
IE : IIS : SQL Server
===============================
IE-IIS-Share
{All using default accounts, for eg. In IIS 6.0, app pool running under Network service or Local system.}
--------------------------------------------------------
IE:
- Add the URL to "Local Intranet Zone"
- Enable Windows Integrated Authentication
- Automatic logon with current username and password or, Automatic logon only in Intranet Zone
--------------------------------------------------------
IIS:
- Only "Windows Integrated Authentication" is checked.
Type in > cscript adsutil.vbs get w3svc/ntauthenticationproviders (You need to run this from <system drive>/inetpub/adminscripts)
- Make sure that this command shows > Negotiate, NTLM; or there is no value set.
Else type in > cscript adsutil.vbs set w3svc/ntauthenticationproviders Negotiate, NTLM
- Make sure to cross check the same at individual website/virtual directory level by using the command > cscript adsutil.vbs find ntauthenticationproviders.
SPN:
- http/<computer-name>:<port> <iis-computer-name>
http/<FQDN> <iis computer-name>
CAUTION: Putting the <port> has been dicey. At times it works and at times it doesn't (only my own experienceJ). I will recommend not using it when you set the SPN.
Domain Controller (DC):
- Under Active Directory Users and Computers -> <Domain> -> Computers, Select the IIS server, right click ->Properties->Delegation tab
Make sure that "Trust this Computer for delegation to any service (Kerberos only) is selected, or else "Trust this computer for delegation to specified services only" is selected.
(You will get this option if the Domain functional level is Windows Server 2003 only).
The 1st option is more generic and is good while you are implementing it for testing the first time.
When the 2nd option is checked, you can go ahead with any of the options: "Use Kerberos only" or "Use any authentication protocol". In such a case make sure that you are selecting the right service Type running on the backend service for which you need delegation. Let's say if you have SQL server running at the backend to which you want the IIS to delegate the credentials, we need to add service type as "MSSQLSVC" and default port as 1433.
<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
IE-IIS-Share
{App Pool running under Domain account}
--------------------------------------------------------
Domain account configuration:
- Account is trusted for delegation
- Is a member of IIS_WPG group on the local IIS computer
- Has "Act as a part of Operating System"/"Impersonate a client after authentication" privileges.
IE:
- Add the URL to "Local Intranet Zone"
- Enable Windows Integrated Authentication
- Automatic logon with current username and password or, Automatic logon only in Intranet Zone.
IIS:
- Only "Windows Integrated Authentication" is checked.
Type in > cscript adsutil.vbs get w3svc/ntauthenticationproviders (You need to run this from <system drive>/inetpub/adminscripts)
- Make sure that this command shows > Negotiate, NTLM; or there is no value set.
Else type in > cscript adsutil.vbs set w3svc/ntauthenticationproviders Negotiate, NTLM
- Make sure to cross check the same at individual website/virtual directory level by using the command > cscript adsutil.vbs find ntauthenticationproviders.
SPN:
- Add the following:
http/<computer-name>:<port> <domain user account>
http/<FQDN> <domain user account>
Remove the following:
http/<computer-name>:<port> <iis-computer-name>
http/<FQDN> <iis-computer-name>
CAUTION: Putting the <port> has been dicey. At times it works and at times it doesn't. I will recommend not using it when you set the SPN.
Domain Controller (DC):
- Under Active Directory Users and Computers -> <Domain> -> Computers , Select the IIS server, right click ->Properties->Delegation tab
Make sure that "Trust this Computer for delegation to any service (Kerberos only) is selected, or else "Trust this computer for delegation to specified services only" is selected.
(You will get this option if the Domain functional level is Windows Server 2003 only).
The 1st option is more generic and is good while you are implementing it for the first time.
When the 2nd option is checked, you can go ahead with any of the options: "Use Kerberos only" or "Use any authentication protocol". In such a case make sure that you are selecting the right service Type running on the backend service for which you need delegation. Let's say if you have SQL server running at the backend to which you want the IIS to delegate the credentials, we need to add service type as "MSSQLSVC" and default port as 1433.
If you still encounter problems try checking for duplicate SPNs.
- To find duplicate SPNs
Use command prompt to execute this command on dc:
ldifde -f <filename> -d "<dc=domain-netbiosname,dc=primary-domain>" -l serviceprincipalname -r "(serviceprincipalname=<serviceprincipalname-to-check-for-duplicates>)" -p subtree
e.g. if the domain name is test.abcd.com:
ldifde –f C:\log.txt -d "dc=test, dc=abcd, dc=com"-l serviceprincipalname –r "(serviceprinicpalname=http/test.abcd.com)" -p subtree
NLB Scenario
--------------------------------------------------------
Now we will talk about a scenario where IIS server is a part of Network load balancer (NLB). First thing, make sure that Kerberos is supported by your NLB (hardware or software). The settings for Kerberos are a bit different when you configure it for IIS servers running as NLB nodes. Here you don't have to set SPNs for individual IIS nodes; rather you need to set an SPN entry for the Virtual IP or alias of the Load balancer. It can be hardware or software.
IE-IIS-Share
--------------------------------------------------------
{All using default accounts}
--------------------------------------------------------
IE:
- Add the URL (here the URL is the NLB's virtual URL or alias) to "Local Intranet Zone"
- Enable Windows Integrated Authentication (Internet Options->Advanced->Security)
- "Automatic logon with current username and password" or "Automatic Logon only in Intranet Zone"
IIS:
Type in > cscript adsutil.vbs get w3svc/ntauthenticationproviders (You need to run this from <system drive>/inetpub/adminscripts)
- Make sure that this command shows > Negotiate, NTLM; or there is no value set.
Else type in > cscript adsutil.vbs set w3svc/ntauthenticationproviders Negotiate, NTLM
- Make sure to cross check the same at individual website/virtual directory level by using the command > cscript adsutil.vbs find ntauthenticationproviders.
- Setup the IIS Servers for delegation as mentioned in the above steps.
- Only "Windows Integrated Authentication" is used in IIS.
- If it's an NLB environment, we need to run the IIS application under an App pool running with domain user account.
- The domain user account should be trusted for delegation in the Active Directory.
From Active Directory Users and Computers, go to the properties of the IIS User (Domain user account). On the Delegation Tab, select "Trust this user for delegation to any service (Kerberos only)"
- The domain user account should be a part of IIS node's IIS_WPG group.
- We need to set the above settings for all the IIS nodes in the NLB.
- Also add the same host header entry for the NLB URL in all the IIS nodes in the IIS manager console -><website>->Properties->Web Site-> Advanced tab.
SPN:
- For IIS:
http/<Netbios name of the NLB> <Domain user account>
http/<FQDN of the NLB> <Domain user account> (If we are running the site on port 80, otherwise http/<FQDN of the NLB>:<Port> <Domain user account>
CAUTION: Putting the <port> has been dicey. At times it works and at times it doesn't. I will recommend not using it when you set the SPN even when your site is running under a different port other than the default 80.
We need to make sure that we do NOT have SPN entries set for http/<FQDN> <iis computer-name> for any of the IIS nodes.
Command: >Setspn –A http/<FQDN of the NLB> <Domain user account>
When we are using Constrained delegation make sure that the backend service is listed in list of services, for e.g. in case we are connecting to SQL server at the backend add MSSQLSVC in the Machine/user properties->Delegation tab.
Note: When we are accessing the web application from a client machine, I suggest installing Kerbtray on the client machine and checking for the "OK as Delegate" option in the attributes section for the corresponding SPN. If it is selected it means the ticket can be used for delegating credentials from the IIS server to the backend server. If it is not then it means there are some issues with the settings in IIS or somewhere else. Using Netmon trace is always a good idea to figure out what tickets are being used or looked for by the Client when accessing the Web application.
In case you face any issues related to Kerberos authentication failure, do the following to understand more from the event logs
- Make sure that we have enabled Kerberos logging according to https://support.microsoft.com/default.aspx?scid=kb;EN-US;262177 on all the IIS nodes.
- Apart from that, you also need to make sure that on all the IIS servers, these two settings are enabled.
Start->Programs->Administrative tools->Local security Policy->Security settings->Local Policies->Audit Policy->Audit account logon events->"Success, Failure".
Start->Programs->Administrative tools->Local security Policy->Security settings->Local Policies->Audit Policy->Audit logon events->"Success, Failure".
NOTE:
At times making sure all the above changes are done properly doesn't help, and in such cases make sure that we purge all the kerberos tickets using Klist or Kerbtray. In fact if possible logoff and re-login to the client machine from where you are testing the web application for kerberos authentication so that the client is issued a fresh ticket.
Additional Info:
================
You might see this error in the event logs in DC:
Event Type: Error
Event Source: KDC
Event Category: None
Event ID: 11
Date: 4/1/2002
Time: 1:40:14 PM
User: N/A
Computer: ComputerName Description:
There are multiple accounts with name host/mycomputer.mydomain.com of type 10.
This might be because of Duplicate SPNs. There are two or more computer accounts that have the same service principal names (SPNs) registered. Please refer to KB 321044 for further info. Remember, you can use Ldifde to check for duplicate SPNs as mentioned above.
Few Links:
For configuring backend SQL server to accept Kerberos authentication go through this link: https://support.microsoft.com/kb/319723/en-us
Also a good article on IIS Kerberos authentication https://support.microsoft.com/kb/907272/en-us and https://support.microsoft.com/?id=929650
How to configure an Asp.Net application for a delegation scenario: https://support.microsoft.com/kb/810572/
PS: Remember to test whether delegation is working fine, you need to access the website URL from a workstation (client) browser and not from IIS server itself.
Reason: IIS uses NTLM credentials when accessing the backend when request reaches it from the local server if, Kerberos fails and there is only single hop involved. NTLM will work with single hop and hence if you access a site locally from the IIS web server it is a single hop and not a double hop scenario.
One simple mantra to be remembered always: You can have multiple different SPNs registered under an account but not the other way, i.e. you should not have the same SPN registered under multiple accounts because it leads to duplicate SPN issue.
***************************************************** Addition to the blog
Think about a scenario where https://server/app1 and https://server/app2 are running inside a network service & a domain user identity respectively .
The SPNs requested will be http/server in both the cases, and since we can’t have duplicated SPNs it won’t work. We need to then either use the same server process identity or dedicated host headers.
Again,if you are using two websites with same name but different ports like https://server:80 and https://server:81; by default IE will request a ticket for the same SPN HTTP/server.
We would then need an hotfix for the client machines, https://support.microsoft.com/kb/908209.
If you have two websites https://application1 and https://application2 that are both DNS aliases (CNAMEs) of say myserver DNS host, IE will request the ticket for SPN HTTP/myserver for both the above websites. By default IE does't use port for sending the ticket and just the SPN name like, http/mysite:99, It ignores the port part of it.
Then you would need client fix https://support.microsoft.com/kb/911149, or use a different DNS HOSTs rather than CNAMEs.You might well go ahead with using a host headers for the websites.
[Something to add here with regard to using IP Addresses to access a site...
There is another confusion that people have while dealing with Kerberos authentication. At times you may want to use IP addresses to access a website and still want Kerberos authentication to work. Now in a general scenario this will not work because Kerberos requires SPN's to recognize a service like HTTP etc. You can however make it work by adding SPN's in the form: http/10.0.1.25 (website's IP) etc. This may or may not work.
However we do not recommend the above way to make Kerberos work for your site using IP addresses. The reason being that SPN's should ideally be names like http/<somename> and not http/<some IP address>.
Let's consider a scenario wherein users belong to domain2 and the Web server is part of domain1. Also let's assume we have mutual trust between domain1 and domain 2. When using IP addresses, client will look for SPN HTTP/10.0.1.25 (assuming this is website's IP on domain1) in domain2 (client's local domain).
Now you may get into an issue wherein domain2 will not give any referral back to to the client to look into domain1 for the SPN. This can occur if IP address is being used to look for a service. In such a case even after adding SPN's for IP addresses, Kerberos won't work and will fall back to NTLM.
]
***Update: Regarding confusion around Port entry in SPNs, check this https://technet.microsoft.com/en-us/library/cc263449.aspx#section4
Happy troubleshooting…and in case you still face issues, Microsoft Product Support Services (PSS) is always there to help you!
Feel free to shoot me a question if you have any confusion or need some assistance.
*Check the following link for my other posts related to Kerberos.