All About Kerberos “The Three Headed Dog” with respect to IIS and Sql
Kerberos Understanding and troubleshooting
Steps to be followed while dealing with Troubleshooting Kerberos related issues.
Prerequisites:
1) Kerberos can only work in intranet because client needs to interact with the Domain controller directly.
2) Kerberos can be used in internet only if you have a public facing active directory. But public facing active directory is not recommended due to security concerns.
3) Server , client and dc must be in the intranet.
Tools that come handy:
SETSPN (For adding, listing, deleting SPN entries for a domain)
KERBTRAY (on client machine)- For checking the Kerberos ticket used by the client to access a web server. It gives you information as to which ticket is being used by the client to access the IIS server, and whether the ticket is capable of delegation.
Deleg Config(server side tool) that adds a application under the side you are trying to browse which gives you information if the website is working for single hop and double hop
Klist(client side) to check which all tickets I am using.
Netmon
How Kerberos Works:
Overview in general:
Single domain Scenario
.gif "clip_image001")
1) Client will contact kdc saying that it’s a user and it needs a ticket to get tickets (TGT)
2) KDC will give the TGT to the client by encrypting with the client machine password hash.
3) When client wants a service ticket, it will produce its TGT(for authorisation) and request TGS.
4) Based on auth the service ticket is given to the client.
5) Then the client passes its service ticket to the network server or resource machine for authentication.
6) After authentication, normal client server response request process continues.
Multiple Domain Scenario
Both Child1 and Child2 are in the same forest with the same parent domain R2dom.local.
Administrator of the Child domain (CHILD1) login to a member server (CH1-Mem) in CHILD1 domain.
After login in the user tries to access \\r2dom-ch2-Mem1 . R2dom-ch2-Mem1 is a member server in Child2 domain.
--> I have used Network monitor to analyze and understand how Kerberos authentication would work, when accessing resource across domain.
Below you see that the Administrator is getting the required Kerberos tickets when accessing resources across domain.
| 10.2.1.2 = DC of CHILD1.R2DOM.LOCAL10.1.1.2 = DC of R2DOM.LOCAL10.3.1.1 = Dc of CHILD2.R2DOM.LOCAL10.10.10.1 = CH1-Mem in CHILD110.10.10.1 10.2.1.2 KerberosV5:TGS Request Realm: CHILD1.R2DOM.LOCAL Sname: cifs/r2dom-ch2-Mem1 --> KrbApReq: KRB_AP_REQ (14) Ticket: Realm: CHILD1.R2DOM.LOCAL, Sname: krbtgt/CHILD1.R2DOM.LOCAL |
** Here you see a Kerberos TGS request being sent to the local domain (CHILD1.R2DOM.LOCAL) DCs for a SPN cifs/r2dom-ch2-Mem1. Local domain TGT sent in the TGS request.
| 10.2.1.2 10.10.10.1 KerberosV5:TGS Response Cname: Administrator --> Ticket: Realm: CHILD1.R2DOM.LOCAL, Sname: krbtgt/R2DOM.LOCAL |
** Here the local domain DC returns a TGT of the Parent domain R2DOM.LOCAL. This is like a referral being sent to the client as the local domain does not have the right to issue a Kerberos Ticket for cifs/r2dom-ch2-Mem1.
| 10.10.10.1 10.1.1.2 KerberosV5:TGS Request Realm: R2DOM.LOCAL Sname: krbtgt/CHILD2.R2DOM.LOCAL --> KrbApReq: KRB_AP_REQ (14) Ticket: Realm: CHILD1.R2DOM.LOCAL, Sname: krbtgt/R2DOM.LOCAL |
** Now the client machine send a TGS request to the Parent domain R2DOM.LOCAL, requesting for a TGT of another of its Child domain where the cifs/r2dom-ch2-Mem1 resides. When sending this request to the parent domain, the client uses the TGT of the Parent domain received in the earlier referral from local DC.
| 10.1.1.2 10.10.10.1 KerberosV5:TGS Response Cname: Administrator --> Ticket: Realm: R2DOM.LOCAL, Sname: krbtgt/CHILD2.R2DOM.LOCAL |
**Parent domain sends the TGT of the child2 domain to the client. This can also be taken as a referral to the CHILD2 domain.
| 10.10.10.1 10.3.1.1 KerberosV5:TGS Request Realm: CHILD2.R2DOM.LOCAL Sname: cifs/r2dom-ch2-Mem1 -->KrbApReq: KRB_AP_REQ (14) Ticket: Realm: R2DOM.LOCAL, Sname: krbtgt/CHILD2.R2DOM.LOCAL |
** Eventually the Child sends a TGS request for the SPN cifs/r2dom-ch2-Mem1 to the DCs in domain CHILD2.R2DOM.LOCAL who is authorized to issue a ticket for the server (r2dom-ch2-Mem1) in its domain. This time the client uses the CHILD2 domains TGT to make request.
| 10.3.1.1 10.10.10.1 KerberosV5:TGS Response Cname: Administrator --> Ticket: Realm: CHILD2.R2DOM.LOCAL, Sname: cifs/r2dom-ch2-Mem1 |
** Finally the client gets the Kerberos Ticket for cifs/r2dom-ch2-Mem1, which will help the Administrator user access the shares on r2dom-ch2-Mem1 in domain CHILD2.R2DOM.LOCAL.
The workstation the user 'Administrator" is using to access the resource across domain, also needs a similar ticket.
Tickets for r2dom-ch2-Mem1 is requested from local domain, who returns the error KDC_ERR_S_PRINCIPAL_UNKNOWN, as the r2dom-ch2-Mem1 computer is not a part of the local domain (CHILD1.R2DOM.LOCAL)
| 10.10.10.1 10.2.1.2 KerberosV5 KerberosV5:TGS Request Realm: CHILD1.R2DOM.LOCAL Sname: r2d-ch2-Mem1 10.2.1.2 10.10.10.1 KerberosV5 KerberosV5:KRB_ERROR - KDC_ERR_S_PRINCIPAL_UNKNOWN (7) |
Below you see that the client computer is getting the required Kerberos tickets when accessing resources across domain.
| 10.10.10.1 10.2.1.2 KerberosV5 KerberosV5:TGS Request Realm: CHILD1.R2DOM.LOCAL Sname: cifs/r2dom-ch2-Mem1 10.2.1.2 10.10.10.1 KerberosV5 KerberosV5:TGS Response Cname: CH1-Mem$ 10.10.10.1 10.1.1.2 KerberosV5 KerberosV5:TGS Request Realm: R2DOM.LOCAL Sname: krbtgt/CHILD2.R2DOM.LOCAL 10.1.1.2 10.10.10.1 KerberosV5 KerberosV5:TGS Response Cname: CH1-Mem$ 10.10.10.1 10.3.1.1 KerberosV5 KerberosV5:TGS Request Realm: CHILD2.R2DOM.LOCAL Sname: cifs/r2dom-ch2-Mem1 10.3.1.1 10.10.10.1 KerberosV5 KerberosV5:TGS Response Cname: CH1-Mem$ |
Kerberos from IIS perspective , how it works:
1) Client sends a HELLO to the server
2) Server replies with a 401(sub status 2) Unauthorized telling the client that it requires to authenticate itself before accessing the page using www authenticate header (negotiate provider).
3) Client requests the Domain controller saying I am trying to access a particular website(say bing.com by specifying it as http/bing.com) from a particular server.
4) Domain controller looks through its SPN (service principal name) table for the particular application or site which will have the entries as below.
| Service| Name of the service registered| registrar name (account name) Eg: HTTP| bing.com| Chiranth |
This means that bing.com is registered for the account Chiranth
Once it finds an entry in the table then DC will encrypt the token with the password hash of the account and send it to the client
5) The Client now sends the token to the server.
6) The server uses the account’s password hash to which the spn is registered(will use machine account if you have kernel mode authentication checked or will use application pool credentials if you have useapppoolcredentials as true in IIS (system.webServer/security/authentication/windowsAuthentication) to decrypt the token. If the server is able to successfully decrypt the token the requested page is given back.
NOTE : To register SPN use the command
| setspn –a http/[fqdn server name] domain\account name |
Specific Scenario when cx wants to register the host or machine spn to domain account, follow the below steps:
| setspn –a http/<computer-name> <domain user account>setspn –a http/<FQDN> <domain user account> |
1) When you use kernel mode authentication we use Host SPN’s password for encryption and decryption of the token by DC and server
2) So if you register an SPN for the account name check to useapppoolcredentials in the configuration editor of windows authentication.
Why Kerberos and not NTLM:
· Kerberos authentication offers the following advantages over NTLM authentication:
· Mutual authentication. When a client uses the Kerberos v5 protocol for authentication with a particular service on a particular server, Kerberos provides the client with an assurance that the service is not being impersonated by malicious code on the network.
· Delegation support. Servers that use Kerberos authentication to authenticate clients can impersonate those clients and use the client's security context to access network resources.
· Performance. Kerberos authentication offers improved performance over NTLM authentication.
· Simplified trust management. Networks with multiple domains no longer require a complex set of explicit, point-to-point trust relationships.
· Interoperability. Microsoft's implementation of the Kerberos protocol is based on standards-track specifications recommended to the Internet Engineering Task Force (IETF). As a result, the implementation of the protocol in Windows 2000 lays a foundation for interoperability with other networks where Kerberos version 5 is used for authentication.
Basic Troubleshooting steps:
What should we do when we see that we are getting 401 unauthorized or windows authentication failure.
Client Side
1) Check if Windows Integrated Authentication is enabled on the browser. Automatic logon with current username and password or, Automatic logon only in Intranet Zone.
2) Check if the URL is on Intranet Zone Sites.
Server side
3) Check if Basic Authentication is working. If it is not working, then it is not Kerberos issue.
4) If basic works, check for NTLM and if it doesn’t work, we should involve AD.
5) In a successful scenario when Kerberos works you will see a 401.2 and 200
6) If it is going over NTLM we will see 401.2 401.1 and 200 in fiddler.
![clip_image001[7]](https://msdntnarchive.blob.core.windows.net/media/MSDNBlogsFS/prod.evol.blogs.msdn.com/CommunityServer.Blogs.Components.WeblogFiles/00/00/01/61/50/metablogapi/0602.clip_image0017_17811991.png "clip_image001[7]")
7) Also to check if the token is Kerberos or NTLM, the Kerberos ticket usually starts with Y and usually NTLM token starts with TLRM.
| Kerberos Ticket eg: Authorization: Negotiate YIIQiwYGKwYBBQUCoIIQfzCCEHugMDAuBgkqhkiC9xIBAgIGCSqGSIb3EgECAgYKKwYBBAGCNwICHgYKKwYBBAGCNwICCqKCEEUEghBBYIIQPQYJKoZIhvcSAQICAQBughAsMIIQKKADAgEFoQMCAQ6iBwMFACAAAACjgg6sYYIOqDCCDqSgAwIBBaEcGxpGQVJFQVNULkNPUlAuTUlDUk9TT0ZULkNPTaIeMBygAwIBAqEVMBMbBEhUVFAbC2NoaXJhbnRoOHIyo4IOXTCCDlmgAwIBEqEDAgEQooIOSwSCDkewzlFPUeEgCJxtUmS7i1lNBiKYS9OCRPEHdyUytqKqs9SY |
| NTLM Token eg: Authorization: NTLM TlRMTVNTUAADAAAAGAAYAIIAAACUAZQBmgAAAA4ADgBYAAAADAAMAGYAAAAQABAAcgAAAAAAAAAuAgAABYKIogYBsR0AAAAPnYnxQFaTOYGkUPDVgZ7SckYAQQBSAEUAQQBTAFQAYwBoAGkAcgBhAG0AVAAtAEMASABJAFIAQQBNAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAN6Op+WbvZB0Tchmb777NmIBAQAAAAAAAB5e5pMbts4B1nm7EWkuiDMAAAAAAgAOAEYAQQBSAEUAQQBTAFQAAQAWAEMASABJAFIAQQBOAFQASAA4AFIAMgAEADQAZgBhAHIAZQBhAHMAdAAuAGMAbwByAHAALgBtAGkAYwByAG8AcwBvAGYAdAAuAGMAbwBtAAMATABjAGgAaQByAGEAbgB0AGgAOAByADIALgBmAGEAcgBlAGEAcwB0AC4AYwBvAHIAcAAuAG0AaQBjAHIAbwBzAG8AZgB0AC4AYwBvAG0ABQAkAGMAbwByAHAALgBtAGkAYwByAG8AcwBvAGYAdAAuAGMAbwBtAAcACAAeXuaTG7bOAQYABAACAAAACAAwADAAAAAAAAAAAQAAAAAgAACXZ723Po2/mq+xx6VA97FdnWz3JNKjqzWGOYp01eYk9QoAEAAAAAAAAAAAAAAAAAAAAAAACQAgAEgAVABUAFAALwBjAGgAaQByAGEAbgB0AGgAOAByADIAAAAAAAAAAAAAAAAA |
8) To verify this we can copy this token value and decode it from base64 in fiddler tools option.
9) Create a test page and test with windows authentication(single hop).
10) If you see that the token sent to the server seems to be the Kerberos token and still it is not working. the First thing to check would be if you are using the right account fro decryption of the ticket n the server. the most awesome thing is if you are not using the proper account fro decryption you will get notified about this by the Kerberos system event log entries on the client itself if Kerberos
| Type : ErrorDate : Time : Event : 4Source : KerberosCategory : NoneUser : N/AComputer : xxxxxxDescription: The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server . The target name used was HOST/xxxxx or HTTP/xxxxx. This indicates that the target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Ensure that the target SPN is only registered on the account used by the server. This error can also happen if the target service account password is different than what is configured on the Kerberos Key Distribution Center for that target service. Ensure that the service on the server and the KDC are both configured to use the same password. If the server name is not fully qualified, and the target domain () is different from the client domain (xxx.xxx.xxx), check if there are identically named server accounts in these two domains, or use the fully-qualified name to identify the server. |
11) SPN configuration
a. Check if Kernel mode Authentication is enabled, if yes, then spn must be registered for machine account
b. If Kernel mode is not enabled and app pool identity is domain account then SPN must be registered for the domain account
c. If Kernel mode authentication is enabled and useapppoolcredential is set to true(as described above) and app pool credential is domain account, then the SPN must be registered for domain account.
12) To check if u have duplicate spn’s u can try the commands.
To find duplicate spn’s on server
Use command prompt to execute this command
| Setspn –x |
Or you could run the below in DC
To find duplicate SPNs
Use command prompt to execute this command on dc:
| ldifde -f <filename> -d "<dc=domain-netbiosname,dc=primary-domain>" -l serviceprincipalname -r "(serviceprincipalname=<serviceprincipalname-to-check-for-duplicates>)" -p subtreee.g. if the domain name is test.abcd.com:ldifde –f C:\log.txt -d "dc=test, dc=abcd, dc=com"-l serviceprincipalname –r "(serviceprinicpalname=http/test.abcd.com)" -p subtree |
On the server side u can run
| Setspn –l acctname |
Also you can run the below command to list the spn’s in the entire FOREST for the particular Hostname.
| Setspn –F –Q service/hostnameEg: Setspn –F –Q */chiranth8r2.com |
This will list all the service spn’s for chiranth8r2.com and also will tell you to which account it is registered.
| Note : If everything looks fine and still single hop is failing. check which account is application pool running under. if it is running under application pool identity, change it to local system and give it a try. if test with local system succeeds, then there is a hotfix from Microsoft for it to work under applicationpoolidentity. Below is the hotfix link. https://support.microsoft.com/kb/2545850 |
13) Create a virtual directory under the site pointing it to a shared location with a simple page(double hop delegation test). If the test fails follow step 13 or if it succeeds follow step 14.
14) (For double Hop) If your test page works check if the trusted for delegation setting is checked in active directory for the account to which machine is registered. If you are using kernel mode you have to trust the machine or host for delegation. To check if machine account is trusted for delegation got ‘Active directory users and computers’, right click on the machine name on go to the tab delegation and select the second option ‘Trust this computer for delegation to any service’ or the third option. If you are using app pool credentials trust the app pool account for delegation. Same above steps should be followed for app pool account in AD.
15) If trusted for delegation is already present and test with shared location succeeds, then there must be some issues with the SQL spn’s. Below are the steps we can follow.
16) How to Verify if SQL is good to except kerberos
1.First thing you would like to check is if your SQL is already capable to Kerberos or not. Simple way to verify this is to see if there is any existing connection using Kerberos. You could do this by running following command against your SQL instance and reviewing the result. If, even a single connection says Kerberos, You are good and don’t need to proceed further.
| select auth_scheme from sys.dm_exec_connections |
There could be combination of these protocols you would typically see as a result of above query.
| NTLM, Kerberos, SQL |
2. If the list does not show Kerberos, We would proceed further. Now, we have to check if Service Principal Names (SPN) for your SQL instance are configured properly or not.
In command below, I am presuming that your SQL box is called MySQLServer.
You domain is called MyDomain.
SQL Service account is SQLServiceAccount
Check the Service account of your SQL Service and run following command against setspn .
| SetSPN -L MyDomain\SQLServiceAccount |
If this is a standalone default SQL instance, It should return following two Service Principal Names (SPN).
| MSSQLSvc/MySQLServer.MyDomain.com:1433 MSSQLSvc/MySQLServer:1433 |
If this is a named instance listening on TCP port 4545, SPNs should look like this.
| MSSQLSvc/MySQLServer.MyDomain.com:4545 MSSQLSvc/MySQLServer:4545 |
If this is a Clustered default SQL instance, It should return following two Service Principal Names (SPN).
| MSSQLSvc/MySQLVirtualName.MyDomain.com:1433 MSSQLSvc/MySQLVirtualName.MyDomain.com |
If this is a Clustered named SQL instance listening on TCP port 4545, SPNs should look like this
| MSSQLSvc/MySQLVirtualName.MyDomain.com:4545 MSSQLSvc/MySQLVirtualName.MyDomain.com |
3.If you see any of the required set of the SPN missing, You need to create it explicitly. For example you run following command.
| SetSPN -L MyDomain\SQLServiceAccount |
And it returned only one SPN .
| MSSQLSvc/MySQLServer:1433 |
You have to create another required one using following command .
| SetSPN -A MSSQLSvc/MySQLServer.MyDomain.com:1433 MyDomain\SQLServiceAccount |
Please note, adding SPNs requires domain admin privileges. Once added successfully, You could confirm it using following command. SetSPN -L MyDomain\SQLServiceAccount it should return .
| MSSQLSvc/MySQLServer.MyDomain.com:1433 MSSQLSvc/MySQLServer:1433 |
4.Once you have set of SPNs in place, your SQL Server is good to accept Kerberos authentication. You have to close your existing applications or connections and open them again to initiate fresh Kerberos connections.
5.In some cases, You would see your connections still using NTLM (see point 1). This would happen most probably because of similar SPNs existing under some other Service account, These would be considered as duplicate SPNs. This can happen if you have your SQL Service account being changed recently. As a result, There would be old SPNs still present under old account. We can find duplicate SPNs by running following command using SetSPN.
SetSPN -X
In case there is same SPN registred under some other account, say OLDSQLServiceAccount, You would see it in output of SetSPN -X. You can then get rid of this duplicate SPN using following command.
| SetSPN -D MSSQLSvc/MySQLServer.MyDomain.com:1433 MyDomain\OLDSQLServiceAccount |
16) The dns entry for the url should be marked as an A record or else it will give u intermittent prompts for username and password(two types a record and c record).
17) Disable LoopBack Check if you are facing the problem only on the Server Side. (but able to browse the url from clients, as on server locally it works over ntlm)
18) You can also enable Kerberos event logging. On information about how to enable Kerberos event logging. see below or the kb https://support.microsoft.com/kb/262177
| Start Registry Editor. Add the following registry value: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\Kerberos\ParametersRegistry Value: LogLevelValue Type: REG_DWORDValue Data: 0x1If the Parameterssubkey does not exist, create it. |
Note Remove this registry value when it is no longer needed so that performance is not degraded on the computer. Also, you can remove this registry value to disable Kerberos event logging on a specific computer.
Quit Registry Editor. The setting will become effective immediately on Windows Server 2008, on Windows Vista, on Windows Server 2003, and on Windows XP. For Windows 2000, you must restart the computer.
You can find any Kerberos-related events in the system log.
19) You can check in deep by downloading delegconfig tool and put that application inside the website you are trying to browse.
20) If none of the above steps help, we can collect a netmon trace on client, server and AD simultaneously and see the communication. We can see if we are getting a service ticket(TGS) and also isolate if the issue lies in AD side, client side or server side.
Note: One of my colleagues has recently developed a tool to troubleshoot the Kerberos issue and manage the settings and information regarding that can be found in the below link
https://github.com/SurajDixit/KerberosConfigMgrIIS
SPN BLOGS and Kerberos Troubleshooting Blogs: https://blogs.iis.net/brian-murphy-booth/archive/2007/03/09/the-biggest-mistake-serviceprincipalname-s.aspx https://blogs.msdn.com/b/saurabh_singh/archive/2007/01/29/kerberos-troubleshooting-from-iis-perspective.aspx https://blogs.msdn.com/b/saurabh_singh/archive/2008/11/16/who-knows-you-may-be-missing-these-points-for-kerberos-authentication-failures-for-web-applications.aspx
Technorati Tags: authentication,iis,UI,missing section,missing,anonymous,registry,kerberos,kerberos troubeshooting,spn,fiddler,http spn,sql spn,windows authentication,negotiate,usekernelmode