Kerberos in Load Balanced Environments

A very common topic we tend to get many questions on is delegation and troubleshooting delegation usually via one of the many topics that impact this: SPN’s, Kerberos, Load Balancers, 401 errors, etc. Many times detailed environment knowledge is required to make any conclusions, so rather than documenting a complex flowchart that covers every possible scenario, I am going to draft up a fairly common environment and document the various steps required to make sure Kerberos is working for that environment. Kerberos only works properly when everything is setup correctly, so troubleshooting issues can be very frustrating and time consuming. If there is any one issue in the chain, authentication will fail and give you errors like: 
      “Login failed for user: NT AUTHORITY\ANONYMOUS LOGON”
      “Login failed for: NT AUTHORITY\SYSTEM”
      “HTTP Error 401.1 – Unauthorized: Access is denied”
If you’re troubleshooting one of these issues I suggest reading through all the steps in this article and verifying your configuration, then move on to the troubleshooting section.

A few notes regarding this post:

  • These concepts apply to any web applications/services that are referenced by a name other than their computer name in Active Directory (regardless of a physical load balancers presence), in this example I’m using Dynamics CRM as the web application
  • For other Kerberos specific questions I suggest reading the “Kerberos for the busy admin” blog article on the AskDS blog.
  • Kerberos authentication is required if you are persisting a user’s credentials or delegating the user’s credentials through more than 1 hop (this scenario is commonly referred to as “double hop”)
  • For the purpose of demonstration I am illustrating unconstrained delegation. Constrained delegation is an optional way to define which SPN’s allow delegation of credentials (essentially a more specific mapping for delegation).
  • For information on CRM 2011 setup and questions on splitting roles and distributing services for better scaling please refer to my other blog post titled: “Server Setup Frequently Asked Questions” (
    • In addition the FAQ article includes information about the MSCRMSandboxService SPN which may be required depending on your configuration.

This post has the following assumptions:

  • CRM 2011 is installed on your CRM servers: NDCRMSrv1, NDCRMSrv2, NDCRMSrv3
  • CRM 2011 is installed to databases located on the SQL Server: NDSQLSrv1 (this is for the purpose of demonstration only, CRM does support using multiple SQL servers to host the various CRM databases)
  • SSRS 2008 (or R2) is installed on your reporting servers: NDRPTSrv1 and NDRPTSrv2 and these servers are configured in a Scale-Out deployment (using the same SSRS configuration databases) instructions are here:
  • You have access (or have a process) to request changes or make changes to Active Directory
  • You have not set any SPN’s for these services in your environment and you have no duplicate SPN’s for these services or host machines.
  • Your domain functional level is 2003 or greater
  • You are using Active Directory integrated authentication for Dynamics CRM.



1) Service Principal Names (SPN’s):

First thing we want to focus on is setting our SPN’s correctly. A service principal Name (SPN) is a way to map identities in Active Directory to various services (SQL, HTTP, etc) hosted in your domain. This mapping is an essential part of issuing and decrypting Kerberos tickets within your environment. Below, in the troubleshooting section, you can see how a workstation will ask your DC’s for a Kerberos ticket via the service name which, for HTTP services, is determined by the URL of the service.

If you run into any scenarios where you believe there must be duplicate SPN’s you’re most likely trying to use more than one identity to host a single service. For example: you’ll notice in the above graphic that we’re hosting the CRM services under the hostname and with a single identity: contoso\CRMWeb. If you wanted to add a second website which would use a different port and a different identity, but would use the same URL, you have two options: 1) use a different hostname/URL to differentiate the services 2) register the SPN’s using a port number to differentiate them.

To configure the SPN’s I recommend mapping out all the various services you’re hosting. For HTTP services you’ll also want to map out the URL’s used to access the service*, for this example we have the following potential URL’s:

Potential CRM Service URL’s: Potential Report Service URL’s:
– http://crm
– http://ndcrmsrv1
– http://ndcrmsrv2
– http://ndcrmsrv3
– http://reports
– http://ndrptsrv1
– http://ndrptsrv2

*NOTE: there has been some debate on the need for registering both the Fully Qualified Domain Name (FQDN) SPN as well as the netbios name SPN, generally speaking, I’ll register both just to make sure all bases are covered. In addition, if users will only be using the “friendly” url or alias (in this instance: http://CRM) then registering the SPN’s for each server is not required – but if you plan on using the server names to access the web services (even if it’s for testing) it’s a good idea to register those as well.

Next, you’ll want to construct a table of the various services that will require an SPN. Please note: if you had more than one SQL server or any custom web applications or web services that use different URL’s those should also be added to this table.

Identity/User Hosting the Service Service Type

Service Principal Name(s) HTTP

http/ HTTP

http/ MSSQLSvc MSSQLSvc/

*NOTE: HTTP is used to identify an SPN as an “HTTP” service, this holds true for both SSL and non-SSL HTTP services. If you are registering any SPN for a web service (SSL or not) the SPN will begin with HTTP. 

2) Registering the SPN’s:

Now that we have a full list of all the SPN’s we’re going to set them up in Active Directory. The easiest way to set an SPN is to use the SetSpn utility in Windows. You will have to have elevated permissions to add SPN’s in Active Directory. The syntax to use for adding the above SPN’s is as follows: SetSpn –S {spnToAdd} {AD Object}. An example of adding the SPN and the SQL SPN would be:
  SetSpn -S HTTP/ contoso\crmweb
  SetSpn -S MSSQLSvc/ contoso\sqlservice

Repeat this process for each SPN, I’ve found it helpful to have an Excel spreadsheet of my SPN’s and account names and check them off as I add them. You’ll notice that I’ve documented using –S instead of –A, –S is a new option that is used to check for duplicates before adding the SPN – if there are duplicates you’ll receive an error and the SPN will not be added. 

3) Configuring CRM’s IIS Server for Kernel Mode Authentication:

IIS 7 and 7.5 can take advantage of Kernel Mode authentication and CRM configures it’s website to use it. Kernel Mode auth is desired as it improves authentication performance and prevents authentication problems with application pools using custom identities. However, when using a host name that does not match the server name(s) we need to tell IIS to use the Application Pool’s identity (as it’s common on all your servers) and not the system identity this allows the Kerberos tickets to be decrypted by any server behind the load balancer. To do this we’re going to set the UseAppPoolCredentials property in the IIS configuration.

Check the value of the webservers WindowsAuthentication configuration
%systemroot%\System32\inetsrv\appcmd.exe list config -section:system.webServer/security/authentication/windowsAuthentication

Set “UseAppPoolCredentials” in the webservers configuration file:
%systemroot%\system32\inetsrv\appcmd.exe set config -section:system.webServer/security/authentication/windowsAuthentication -useAppPoolCredentials:true /commit:apphost

3.1)Kerberos Auth Persistence (optional)

Configuring persistence of Kerberos credentials will reduce the challenge/responses on the same TCP session resulting in fewer 401’s and better performance. If you analyze a Fiddler trace of communication between the user and the server you’ll notice that most successful responses from a webserver are proceeded by at least one 401, even when the requests are for the same TCP session between the client and server. By setting the AuthPersistNonNTLM property to “true” the webserver will persist the credentials of the user over the existing TCP sessions – dramatically reducing the number of 401’s for the user. If you’re looking in Fiddler you will also notice a Persist-Auth header in the response indicating auth was persisted for that request. If you’re interested in another tweak for better performance through compression of WCF traffic be sure to check out this blog posting: “Enable WCF Compression to Improve CRM 2011 Network Performance

Check the value of the webservers WindowsAuthentication configuration
%systemroot%\System32\inetsrv\appcmd.exe list config -section:system.webServer/security/authentication/windowsAuthentication

Set “authPersistNonNTLM” in the webservers configuration file:
%systemroot%\system32\inetsrv\appcmd.exe set config -section:system.webServer/security/authentication/windowsAuthentication -authPersistNonNTLM:true /commit:apphost

4)Configure your accounts to allow for Kerberos Delegation:

To make sure delegation will succeed in any given environment there are a couple of settings that must be in place before delegation is allowed.

  1. Any services that persist credentials must be allowed to delegate in Active Directory. For this example, we must enable the CRMWeb account to delegate. To do this
    • login to a server with credentials allowing you to edit the given service account
    • Open Active Directory Users & Computers (from the run box type: DSA.msc and press Ok)
    • Click on the root of your domain (in this case it would be
    • Click on the Find button in the toolbar of Active Directory Users & Computers: image_thumb[137]
    • Find your service account (CRMWeb) and open that objects properties
    • Click on the Delegation tab (this will only show up once the account has an SPN registered – otherwise it is hidden)
    • Make sure to “Trust this user for delegation to any service (Kerberos only)
      NOTE: if you wish to implement constrained delegation you would select Specified services only and select which SPN’s to use. I recommend setting up unconstrained delegation first, then constraining it after you’ve verified that it works

5) Troubleshooting:

  • Clock Skew – this is an illusive cause and is so obvious it’s often overlooked.  If any of the servers communicating are more than 5 minutes (Default – the policy name is: Maximum lifetime for user ticket Kerberos Policy), if the servers are out of time sync then you can get the same Kerberos errors documented in this article.  On a side note, this one has burned me several times, make this your first check just in case – maybe you’ll get lucky and this will be your silver bullet 🙂
  • Client Browser IE Configuration: Double check Internet Explorer for the “Enable windows authentication” option:
    • image_thumb1
  • How you access CRM: Kerberos authentication cannot function without connectivity to a domain controller, thus if you are accessing over the internet Kerberos auth will likely fail unless you’re on a VPN or on your corporate network. If you’re accessing CRM over the internet, CRM has a configuration specifically designed to work over the internet called “Claims Based Authentication with Internet Facing Deployment (IFD)”.
  • Check your SPN’s – Windows Server 2008 now includes SetSPN which can be used to list the SPN’s. No special permission should be required to read these SPN’s. For example: to check the CRMWeb account for SPN’s open a command prompt and type the following: setspn –L contoso\crmweb. This will list all the SPN’s found under the CRMWeb account.
  • Event Logs: Check your servers and client System & Security event logs. The System log can contain errors from KDC or Kerberos which might contain useful information and sometimes you’ll find useful failed audit entries in the Security log (be careful with these as they can be benign as well). If this doesn’t yield any results also check the Global Catalog domain controller System Event logs for KDC errors.
  • Check DNS: For each server (CRM, SQL, reports, etc) run nslookup and make sure you get the correct IP address, in addition make sure the reverse lookup is working as well (ie: nslookup {ip} should return you the correct host name).
  • Active Directory Replication: As mentioned above, the SPN’s are stored in Active Directory. If Active Directory replication is delayed or broken the SPN’s will not replicate and you can get inconsistent results. I suggest first checking with your Active Directory admin and having them check replication status and the NTDS logs. If necessary you can use a tool like LDP.exe to bind to each global catalog and check for the SPN’s, but this can be time consuming.
  • Netmon: Capture a netmon trace from your client
    • Close all of your Internet Explorer browsers
    • Start netmon 3.4 and press the New Capture button. Once the new capture tab appears, press the Start Capture button
    • From the run box on your computer type in your server URL: (for this example)
    • Once the prompt appears stop the netmon capture
    • Enter the following into the “Display Filter” box: KrbError (see in the graphic below)
    • Then press the “Apply” button in Display Filter box to filter your Frame Summary
    • Within the Frame Summary window click on the Kerberos Errors displayed. The frame summary should show you the DC name or IP that responded to the request as well as the error:


    • In the Frame Details expand the KRB_ERROR to view the details of the error, this shows exactly which service name failed along with the realm info and other details. In this scenario you would see a failure for the service HTTP/ or HTTP/crm depending on how you entered the URL into the browser. This Kerberos error is telling you that it cannot find a suitable Service Principal Name matching the service.
  • Enable Kerberos Event Logging on the server and on your client, you will likely see an error of: “KDC_ERR_PREAUTH_REQUIRED” or “KDC_ERR_S_PRINCIPAL_UNKNOWN” when you attempt to login. Please keep in mind that enabling Kerberos event logging will show you benign errors as well as valid errors to investigate, so I recommend only doing this to validate other errors or once you’ve exhausted other techniques.
  • User delegation blocked. If you have a user that is constantly prompted but other users are not prompted
      • login to a server with credentials allowing you to edit the given user account in Active Directory
      • Open Active Directory Users & Computers (from the run box type: DSA.msc and press Ok)
      • Click on the root of your domain (in this case it would be
      • Click on the Find button in the toolbar of Active Directory Users & Computers: image_thumb[143]
      • Find the account and open the properties for that user
      • Click on the “Account” tab and make sure the following checkbox is unchecked
  • Check for duplicate SPN’s:
    • Please remember, all SPN’s must be unique, if you register a duplicate SPN then neither will function properly (SPN’s are case insensitive). There are a couple of techniques you can use to check for duplicate SPN’s, I’m going to focus on the use of Windows Server 2008’s version of SetSPN. There are two approaches to checking for duplicates using SetSPN:
      1. Check for any duplicate SPN’s for the entire forest – this is a very handy option but in true enterprise environments this can take some time to run and can consume a lot of memory. To do this you run the following command at a command prompt:
        SetSpn.exe –X
      2. Search for a specific SPN across the domain for duplicates. For example, if I wanted to search my domain for any duplicates of the SPN HTTP/ I would use the following syntax:
        SetSpn.exe –Q HTTP/
  • Use DelegConfig to test Kerberos Authentication and setup: 

As always I’ll do my best to keep up with comments. If you want to keep in touch with our team you can follow us here ( as well as on Twitter, if you have a Microsoft Premier support contract and wish to work with a member of our team ask your TAM about the PFE offerings we have for Dynamics CRM, and if you want to connect with us at conferences we can be found speaking and attending Dynamics Convergence. We’ll keep any other events or opportunities to connect up to date here and on Twitter.


Sean McNellis

Comments (20)

  1. nrodri says:

    Hi Sean,

    Many thanks for this article very insightful. I would to share a command which in many cases can be more efficient than setspn.exe -X for duplicate detection. I have found that setspn.exe -X does not always return the duplicates. the command is:

    ldifde -f c:spn_out.txt -d "DC=domain,DC=com" -l serviceprincipalname -r "(serviceprincipalname=*/*)" -p subtree



  2. Thanks for sharing Nuno!  Let me know of any situations where this article was useful for you also if you have any more tips like this please do share.  


  3. Yona says:

    Hi Sean

    You're article is a great resource for setting up CRM in a load balanced environment. It helped me a lot.

    However, I'm running into issues load balancing just the Sandbox Processing Service, is it even possible? I've expanded on it here:…/load-balancing-ms-dynamics-crm-2011-sandbox-processing-service

    Thanks again.

  4. Hi Yona, thanks for reading!  The Sandbox services are not able to sit behind a load balancer – or the way they're designed doesn't lend itself well to that configuration. I believe the CRM Implementation guide contains information on this, but the Sandbox services will open a network Socket and listen on a specific port for traffic. When the sandbox service successfully comes online it calls back to CRM to register itself as "available" (so services like async and front end know that it's available). The roles that require isolation (sandboxing) will then round robin call into the sandbox services directly over that TCP channel on demand. For more information around this see this: Also be advised that you may need to setup a custom SPN for the sandbox service as well.

    Let me know if you have further questions on this – thanks!


  5. Yona says:

    Thanks for responding. However, it doesn't explain why one front-end server won't load balance to two sandbox servers (1st item in my SO question).

  6. I would make sure both sandbox services aren't throwing errors or warnings in the event logs (indicating they cannot come online properly) as this would cause what you are seeing, if you still don't see them distribute over time I would recommend opening a ticket. Thanks!


  7. Sachu says:

    Guys i am a novice to NLB & SPN. Can u people help me in gaining knowledge from basics about these relating to CRM. Please suggest your valuable answers.

  8. Hi Sachu,

    Apologies for how long it took to respond to your question.  This particular article applies to any configuration where you have double hop authentication using a webserver as a front end. If your CRM environment is using IFD+Claims auth this won't likely be of concern to you. If you're a premier customer one option would be to ask your tam to schedule some time with us and we can go over it with you over the phone or in person.  


  9. Michael Drake says:

    I ran the "appcmd.exe list config" command in Step 3 above, and saw the identical setup shown in that first command screen shot. Can you tell me where the corresponding configuration file is located? I cannot find the entries in either the machine.config or web.config under the C:WindowsMicrosoft.NETFramework or C:WindowsMicrosoft.NETFramework64 folders.

  10. medrake says:

    Above comment left by me, but was not signed in. Thanks.

  11. Hi Michael sorry it's taken so long to get back to you.  The configuration is stored in the applicationhost.config file which is usually located in c:windowssystem32inetsrvconfig.



  12. Chris K says:

    Hi Sean,

    Thanks for your excellent blogs. They're a useful source of information that would otherwise be very difficult to come by.

    I have one (hopefully final) comment relating to the sandbox service. We are looking to implement a scalable deployment architecture so looking to have multiple sandbox servers, as well as multiple web front end servers that will be deployed behind a network load balancer. In relation to your previous post that the sandbox services "call back to CRM to register itself as "available", presumably they flag themselves as available to all  servers? Does this occur as a database update or direct communication to a service?



  13. Amol says:

    Thanks for your excellent blogs.

    Question for clarfication:

    When we say constraint delegation, does it mean setting up the service account's attribute "AllowedToDelegateToAccount"

  14. Amol says:

    Thanks for the excellent blog

    Question :

    When we say constraint delegation ,it means setting the attribute values for  AllowedToDelegateToAccount  for service account right?

  15. Jochen says:

    Dear Sean,

    I'm a Little unsure how to configure the trust for Delegation in our scenario.

    We want to Setup Kerberos (works with NTLM).

    We've an IIS/ARR Setup as a Loadbalancer/ReverseProxy which forwards the requests to our CRM Server  via URL rewriting, which in turn accesses MSSQL and SSRS (the latter we don't use).

    If I create SPNs for our CRM Server (without setting delegation) no AUTH is possible anymore and we receive only 401's.

    So my question is where do we need to Setup the trust for Delegation? From IIS/AAR to CRM or vice versa?

  16. Jim Justice says:

    you can find additional information on Kerberos and Microsoft workloads on the KEMP Technologies…/kerberos

  17. Gary says:

    Hi, I have an issue with Team Foundation Server as from Linux client using Kerberose via a Load Balancer.

    Being most software developer and DBA I don't have any experience in this area. Can you recommend a resource for this specific scenario?

    thx in advance,


  18. Tim says:

    Hi Sean,
    It is a great article and it is very helpful. I am hitting a 401 issue which I cannot find the reason. My configuration is as follows:
    There is a NLB. Behind NLB, I have two servers (let say server A and server B). The service is a WCF service. The SPN setting is correct. There is no problem when we making WCF calls from remote clients.
    In server A, there is a program making WCF calls to the NLB URL every 5 minutes. I found that there are 50% of the requests failed with 401 status code. I enabled the kerberos audit in server A, and the log said the kerberos authentication failed with status code KRB_AP_ERR_MODIFIED. I tried to change the service URL from NLB to server B, everything works fine. Do anyone know why this 401 happened? Thank you.


    1. @Tim – thanks for reading! If I understand your scenario you have an NLB with server A&B behind it and server A&B have local services that call back to the NLB URL (essentially loop back?). I assume you have external clients calling into the servers, and the servers also need to call this service (behind the NLB).

      If that’s the scenario one possible resolution is to edit the Host file on each local server, add the FQDN you’re calling from the server and map that to the servers local loopback address of (this will keep the traffic on the local server for local server sourced calls). NOTE: before doing this you may need to disable the loopback check (DisableLoopbackCheck) in windows to allow for this.

      1. Tim says:

        Thank you very much. Your suggestion works fine.