We need to talk (aka: “It’s not me, it’s you”) ...

Article
07/16/2010

Issue:

After installing an Administrative node of FAST Search Server 2010, additional Non-Administrative FAST Search Server 2010 nodes cannot communicate with the Admin node. Subsequently, the non-admin nodes to fail download the deployment.xml, and as such, fail to finish configuration successfully.

What you can expect to see:

When running the psconfig.ps1 script on the non-admin nodes via PowerShell, you may see a failure similar to the following:

Please wait while Windows configures FAST Search Server for SharePoint. Configuration may take several minutes...

An error occurred while configuring IPSec - Could not connect to the admin node.

This may be because of,

1. Invalid admin node name

2. Invalid baseport. Baseport of admin node and non-admin node must be same

3. Admin node is not up and running

4. Missing IPSec rules on admin node. If you added this host to the deployment.xml after running this script on the admin node, you need to rerun the IPSec cmdlet on the admin node

See C:\Install\FAST\log\fast-install.log for more details.

This is also evident if you simply attempt to run the Set-FASTSearchIPSec PowerShell cmdlet:

2010-07-15 04:03:19.170Z Verbose [15] SetIPSec - no.fast.middleware.SystemException: Unable to connect to the remote server ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 157.54.158.87:13390

at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)

at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)

--- End of inner exception stack trace ---

at System.Net.HttpWebRequest.GetRequestStream()

at no.fast.middleware.ConnectionManager.ExecuteMethod(HttpWebRequest request, MemoryStream dataStream)

at no.fast.dsinterfaces.nameservice.NameserverStub.Resolve(String Name, String InterfaceType, String Version)

--- End of inner exception stack trace ---

at Microsoft.SharePoint.Search.Extended.Installer.Mahasen.Common.Middleware.MiddlewareOperations.DownloadDeploymentConfig(String localhost, String nameserverhost, String middleware_port, String fileName)

at Microsoft.SharePoint.Search.Extended.Installer.Mahasen.Cmdlets.SetIPSecCommand.CreateNonAdminRules(IHostConfig hostConfig)

at Microsoft.SharePoint.Search.Extended.Installer.Mahasen.Cmdlets.SetIPSecCommand.ProcessRecord()

Root Cause (Summary):

The non-admin node(s) attempt to communicate to the admin node via port 13390, which is secured via IPSEC. If the Kerberos session encounters a failure, the IPSEC connection is never established, and the socket will not be opened.

How to Troubleshoot:

1. In these scenarios, we use the “Microsoft IPSEC Diagnostic Tool” (from https://support.microsoft.com/kb/943862) to gather additional diagnostic data specific to the IPSEC failures.

2. Once the data has been collected, open the wfpdiag.txt file, search for “ERROR_IPSEC_IKE_AUTH_FAIL,” and then you should see data similar to the following:

[2]01C0.15D4::07/15/2010-16:38:22.626 [user] |146.27.92.41|Peer failed with Windows error 13801(ERROR_IPSEC_IKE_AUTH_FAIL)

[2]01C0.15D4::07/15/2010-16:38:22.626 [ikeext] 1137|146.27.92.41|ProcessNotifyData: mmSa 0000000002F14600 cookie 71ddc30c state 2 messId 0

[3]01C0.15D4::07/15/2010-16:38:22.643 [ikeext] 1137|146.27.92.41|IKE diagnostic event:

Event Header:

Timestamp: 1601-01-01T00:00:00.000Z

Flags: 0x00000106

Local address field set

Remote address field set

IP version field set

IP version: IPv4

IP protocol: 0

Local address: 146.27.92.39

Remote address: 146.27.92.41

Local Port: 0

Remote Port: 0

Application ID:

User SID: <invalid>

Failure type: IKE/Authip Main Mode Failure

Type specific info:

Failure error code:0x000035e9

IKE authentication credentials are unacceptable

Failure point: Remote

Flags: 0x00000002

Multiple MM failures present

Keying module type: Authip

MM State: Second roundtrip (SSPI) packet sent

MM SA role: Responder

MM auth method: Unknown

Cert hash:

0000000000000000000000000000000000000000

MM ID: 0x0000000000000471

MM Filter ID: 0x000000000004bfc3

[3]01C0.15D4::07/15/2010-16:38:22.644 [ikeext] 1137|146.27.92.41|IKE diagnostic event:

Event Header:

Timestamp: 1601-01-01T00:00:00.000Z

Flags: 0x00000106

Local address field set

Remote address field set

IP version field set

IP version: IPv4

IP protocol: 0

Local address: 146.27.92.39

Remote address: 146.27.92.41

Local Port: 0

Remote Port: 0

Application ID:

User SID: <invalid>

Failure type: IKE/Authip Main Mode Failure

Type specific info:

Failure error code:0x000035ec

General processing error

Failure point: Local

Flags: 0x00000002

Multiple MM failures present

Keying module type: Authip

MM State: Second roundtrip (SSPI) packet sent

MM SA role: Responder

MM auth method: Kerberos

Cert hash:

0000000000000000000000000000000000000000

MM ID: 0x0000000000000471

MM Filter ID: 0x000000000004bfc3

[2]01C0.15D4::07/15/2010-16:38:22.644 [ikeext] 1137|146.27.92.41|Cleaning up mmSa: 0000000002F14600. Error 13801(ERROR_IPSEC_IKE_AUTH_FAIL)

Root Cause (Details):

After reviewing the data above, we can clearly see a breakdown in the IPSEC conversation. This is primarily caused by one (or both) of the following tweaks:

· A Group Policy setting that forces the Kerberos conversation to take place entirely over UDP (“MaxPacketSize”)

· A Group Policy setting that restricts the Maximum Token Size (“MaxTokenSize”)

To verify this, open an Administrative Command Prompt, and run “GPRESULT -V” (no quotes), and you should see an entry similar to the following:

GPO: Default Domain Policy

KeyName: SYSTEM\CurrentControlSet\Control\Lsa\Kerberos\Parameters\MaxTokenSize

Value: 160, 134, 1, 0

State: Enabled

This indicates that the Default Domain Policy is modifying the default settings for the MaxTokenSize, and subsequently is interfering with our ability to successfully establish a connection via IPSEC.

Resolution:

You may pursue the following options to resolve this issue:

· Exclude the FAST Search Server 2010 machines from having this setting applied to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\Kerberos\Parameters\MaxTokenSize

· The FAST Server Administrators may potentially override the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\Kerberos\Parameters\MaxTokenSize setting manually in the Registry (to a value of 0x0000FFFF), and subsequently lock it down such that future Group Policy applications do not overwrite this setting

Please refer to https://support.microsoft.com/kb/327825, “New resolution for problems with Kerberos authentication when users belong to many groups” for further information regarding the MaxTokenSize registry value.