Least Privilege Remote Configuration of Search for SharePoint Server 2013

  • Article

My customer was going through the process of creating scripts to configure their latest SharePoint Server 2013 (SP2013) farm and encountered an issue while provisioning the Search topology. It was quite a sticking point for them so I began some research. I've simplified the scenario in order to make the repro easier and this is my configuration:

  • DC – a Windows Server 2012 VM hosting Active Directory and SQL Server 2012 SP1
  • APP1 – a Windows Server 2012 VM with SP2013 installed, but not configured
  • WFE1 – a Windows Server 2012 VM with SP2013 installed, but not configured

The objective is to create and configure a new SP2013 farm and provision the entire new (non-default) Search topology on APP1 from Powershell scripts running on WFE1. So the steps are basically:

  1. Create farm on WFE1
  2. From APP1, join farm
  3. From WFE1, provision Search entirely on APP1

The part that makes this a bit tricky in addition to the remote provisioning is that it is a least privileged configuration. (I've mentioned before how much I love/loathe this type of setup.) So the accounts that I have are:

  • contoso\SetupAcct
  • contoso\SearchSvc
  • contoso\SearchContent

The initial pieces of creating and configuring the farm are rather straight-forward and indeed "old-hat" to a lot of us since we've been running with Powershell scripts since the inception of SP2010 so I won't re-hash that part here; however the piece that was causing a problem with this issue is the remote provisioning of Search. At the point where we create and activate the topology, Powershell returns the following:

 Exception calling "Activate" with "0" argument(s): "Timed out waiting for search service 'SPSearchHostController' provisioning timer job to complete" 

 At C:\Scripts\Provision-Search-Svc-App-and-Topology2.ps1:164 char:4 

 + $clone.Activate() 

 + ~~~~~~~~~~~~~~~~~ 

  + CategoryInfo : NotSpecified: (:) [], MethodInvocationException 

  + FullyQualifiedErrorId : TimeoutException

 

There were also errors in the event log on APP1 (the target search server):

  1. Application: hostcontrollerservice.exe

    Framework Version: v4.0.30319

    Description: The process was terminated due to an unhandled exception.

    Exception Info: System.ServiceModel.CommunicationException

    Stack:

    at Microsoft.Ceres.HostController.WcfServer.WcfService.StartService()

    at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

    at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

    at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

    at System.Threading.ThreadHelper.ThreadStart()

     

  2. Faulting application name: hostcontrollerservice.exe, version: 15.0.4521.1000, time stamp: 0x519e34b6

    Faulting module name: KERNELBASE.dll, version: 6.2.9200.16451, time stamp: 0x50988aa6

    Exception code: 0xe0434352

    Fault offset: 0x000000000003811c

    Faulting process id: 0x1288

    Faulting application start time: 0x01cf1250582d79f8

    Faulting application path: C:\Program Files\Microsoft Office Servers\15.0\Search\HostController\hostcontrollerservice.exe

    Faulting module path: C:\Windows\system32\KERNELBASE.dll

    Report Id: 9a6eb0a6-7e43-11e3-9411-00155d8d78e4

    Faulting package full name:

    Faulting package-relative application ID:

     

and in the ULS logs on APP1:

System.ServiceModel.CommunicationException: The service endpoint failed to listen on the URI 'net.tcp://localhost/ceres/hostcontroller/' because access was denied. Verify that the current user is granted access in the appropriate allowAccounts section of SMSvcHost.exe.config. ---> System.ComponentModel.Win32Exception: Access is denied

at System.ServiceModel.Activation.SharedMemory.Read(String name, String& content)

at System.ServiceModel.Channels.SharedConnectionListener.SharedListenerProxy.ReadEndpoint(String sharedMemoryName, String& listenerEndpoint) -

-- End of inner exception stack trace ---

at System.ServiceModel.Channels.SharedConnectionListener.SharedListenerProxy.ReadEndpoint(String sharedMemoryName, String& listenerEndpoint)

at System.ServiceModel.Channels.SharedConnectionListener.SharedListenerProxy.HandleServiceStart(Boolean isReconnecting)

at System.ServiceModel.Channels.SharedConnectionListener.SharedListenerProxy.Open(Boolean isReconnecting)

at System.ServiceModel.Channels.SharedConnectionListener.StartListen(Boolean isReconnecting)

at System.ServiceModel.Channels.SharedTcpTransportManager.OnOpenInternal(Int32 queueId, Guid token)

at System.ServiceModel.Channels.SharedTcpTransportManager.OnOpen()

at System.ServiceModel.Channels.TransportManager.Open(TransportChannelListener channelListener)

at System.ServiceModel.Channels.TransportManagerContainer.Open(SelectTransportManagersCallback selectTransportManagerCallback)

at System.ServiceModel.Channels.TcpChannelListener`2.OnOpen(TimeSpan timeout)

at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)

at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout)

at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)

at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout)

at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)

at Microsoft.Ceres.HostController.WcfServer.WcfService.StartServiceEndpoint()

at Microsoft.Ceres.HostController.WcfServer.WcfService.StartService()

at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

at System.Threading.ThreadHelper.ThreadStart() StackTrace:

at Microsoft.Ceres.Diagnostics.Native.dll: (sig=11fe9943-9e8b-44e9-b239-aa54f65e251f|2|microsoft.ceres.diagnostics.native.pdb, offset=A75E)

at Microsoft.Ceres.Diagnostics.Native.dll: (offset=19159)

 

There has been a lot of research and blog posting on the error from the ULS log, but all that I found simply said "to resolve reboot the server" or "to resolve add the SearchSvc account to the local administrators group". While both of these did in fact resolve the issue, they didn't go far enough to actually tell me what was happening. After extensive repros of this issue, the core of the problem can be found in the following blog:

https://blogs.msdn.com/b/joncole/archive/2010/06/10/tcp-port-sharing-access-is-denied.aspx

Basically what happens is that when APP1 joins the SP2013 farm, the config file for Smsvchost.exe gets updated to add the local WSS_WPG group to the AllowAccounts element. At this point, the SearchSvc account is not a member of that group so the security token that the Smsvchost service (TcpPortSharing) has for this group does not include it. As Jon states in his blog referenced above, basically this service (and any other service that depends on this service) needs to be restarted to get a new security token for the WSS_WPG group. My new steps to resolve this are as follows:

  • On WFE1, create the farm.

  • On APP1, join the farm and execute "net start sptimerv4" since the timer service is not started through this process.

  • On WFE1, run the search provisioning script up to the point where the Search service instance on APP1 is started.

  • On APP1, "net stop/start" each of the following:

     

    • NetTcpActivator

    • NetPipeActivator

    • NetTcpPortSharing

       

  • On WFE1, complete the Search configuration and activate the new topology.

In Powershell terms, that means to either use PS Remoting to run the net stop/start commands immediately after the call to Start-SPEnterpriseSearchServiceInstance for APP1 or to insert a pause so you can go directly to APP1 to make restart the services.