401 – Unauthorized Response to Register-SPWorkflowService

Article
12/30/2014

My previous post about configuring the Workflow Manager using least privilege is a great start; however I realized the other day that I was missing something. Thinking about a more realistic scenario and indeed if I decided that I need high availability, then I'd need more than one Workflow Manager (WF) server.

I started doing research and found a great 3 or 4 part series of blogs from the well-known Spencer Harbar. In this blog series, Spencer goes through a great explanation of the certificates involved and walks through creating a highly available WF Farm using Windows Load Balancer (WLB) software. His explanation describes how the Subject Alternative Name (SAN) field is used to allow us to use virtual names easily.

I'm not going to re-hash the setup of WF in this post, but I did alter the setup in order to use SSL. My configuration for this post includes:

a fully configured SharePoint Server 2013 Farm with a host named site collection at https://intranet.contoso.lab
a WF Farm consisting of a single server and configured for HTTPS, but I want to register it within SharePoint as a virtual name in order to ease expansion in the future:

Server: WF01.contoso.lab

Virtual name: WF.Contoso.lab

So we've created a DNS A record for the virtual name of wf.contoso.lab and it's resolving to the IP address of the server wf01.contoso.lab. Next we run the powershell cmdlet to register the WF Farm with the SharePoint Farm and we expect all of this to work without a problem… because nothing ever fails the first time around, right? Riiiiiiiggghhhtt…

So we log into the SharePoint server as an appropriate account (refer to my previous post), and open a SharePoint 2013 Management Shell and execute the following:

The result?

Register-SPWorkflowService : A response was returned that did not come from
the Workflow Manager. Status code = 401:
< !DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"https://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
< html xmlns="https://www.w3.org/1999/xhtml">
< head>
< meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/>
< title>401 - Unauthorized: Access is denied due to invalid credentials.</title>
< style type="text/css">
< !--
body{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica,
sans-serif;background:#EEEEEE;}
fieldset{padding:0 15px 10px 15px;}
h1{font-size:2.4em;margin:0;color:#FFF;}
h2{font-size:1.7em;margin:0;color:#CC0000;}
h3{font-size:1.2em;margin:10px 0 0 0;color:#000000;}
#header{width:96%;margin:0 0 0 0;padding:6px 2% 6px 2%;font-family:"trebuchet
MS", Verdana, sans-serif;color:#FFF;
background-color:#555555;}
#content{margin:0 0 0 2%;position:relative;}
.content-container{background:#FFF;width:96%;margin-top:8px;padding:10px;positi
on:relative;}
-->
< /style>
< /head>
< body>
< div id="header"><h1>Server Error</h1></div>
< div id="content">
< div class="content-container"><fieldset>
<h2>401 - Unauthorized: Access is denied due to invalid credentials.</h2>
<h3>You do not have permission to view this directory or page using the
credentials that you supplied.</h3>
< /fieldset></div>
< /div>
< /body>
< /html>
HTTP headers received from the server - WWW-Authenticate: Negotiate,NTLM.
Client ActivityId : e27eaa36-7e61-4511-a767-7cc5bad42e2e.
At line:1 char:1
+ Register-SPWorkflowService -SPSite https://intranet.contoso.lab
-WorkflowHostUri ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~
+ CategoryInfo : InvalidData: (Microsoft.Share...WorkflowService:
RegisterSPWorkflowService) [Register-SPWorkflowService], AuthenticationExc
eption
+ FullyQualifiedErrorId : Microsoft.SharePoint.WorkflowServices.PowerShell
.RegisterSPWorkflowService

So basically, even though I was using all the right accounts in my least privileged configuration --- you read my previous post, right? --- I was still encountering an access denied. To save you the time and effort of all of the trouble-shooting I will cut to the chase. The following error showed up in the System Event log on my WF server:

Log Name: System
Source: Microsoft-Windows-Security-Kerberos
Event ID: 4
Level: Error
User: N/A
Computer: wf01.contoso.lab
Description:
The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server wf01$. The target name used was HTTP/wf.contoso.lab. This indicates that the target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Ensure that the target SPN is only registered on the account used by the server. This error can also happen if the target service account password is different than what is configured on the Kerberos Key Distribution Center for that target service. Ensure that the service on the server and the KDC are both configured to use the same password. If the server name is not fully qualified, and the target domain (CONTOSO.LAB) is different from the client domain (CONTOSO.LAB), check if there are identically named server accounts in these two domains, or use the fully-qualified name to identify the server.

After consulting with a few peers, they immediately suspected kernel mode auth in IIS. As a test, we disabled that option on the WF website and reset IIS (to refresh Kerberos tickets) and now when registering the SPWorkflowService in SharePoint it worked flawlessly. However, there are advantages to using kernel mode authentication in IIS 7.0 so we don't necessarily want to disable it completely.

The explanation of what's happening…

With kernel mode authentication enabled, IIS utilizes http.sys to decrypt Kerberos tickets and it will be doing so under the context of the System Account (notice the reference in the event log error above to 'wf01$'). When this happens with an application pool that is running under specific account credentials, the Kerberos tickets cannot be decrypted and used.

Instead of disabling kernel mode authentication, we have another option to allow the application pool credentials to be used to decrypt Kerberos tickets – useAppPoolCredentials. The command for this is as follows:

appcmd.exe set config "Workflow Management Site" –section:system.webServer/security/authentication/windowsAuthentication /useAppPoolCredentials:"True" /commit:apphost

After running this, with kernel mode authentication enabled on the Workflow Management Site, then the registration cmdlet should succeed. Note that if you have run the registration command previously and it failed, then you may need to add the –Force parameter to it in order to overwrite the previously failed registration.

As with the majority of the content I produce, I could not have done this alone (or at least not as quickly). Kudos out to a couple of peers:

Dean Cron (fellow PFE)
Xuehong Gan
Vishal Bajal

401 – Unauthorized Response to Register-SPWorkflowService

The result?

The explanation of what's happening…

Additional resources