Workflow manager Configuration failure: Step ‘Add host to Service Bus farm’ failed

I installed the Work Flow Manager 1.0 on SharePoint 2013 environment (Windows Server 2012, Microsoft SQL Server 2012 - SP1). While running the Workflow Manager Configuration Wizard, it fails at the step "Add host to Service Bus farm" - "Service Bus Services
Starting...". I am running the wizard under a domain administrator user but it keeps failing at this point every time.

I got the below event entry in the event Viewer:

Event ID: 61502
Source: Microsoft-ServiceBus

Microsoft.ServiceBus.Commands.Common.Exceptions.OperationFailedException: Starting service Service Bus Message Broker failed: Time out has expired and the operation has not been completed.
at Microsoft.ServiceBus.Commands.Common.SCMHelper.StartService(String serviceName, Nullable`1 waitTimeout, String hostName)
at Microsoft.ServiceBus.Commands.ServiceBusConfigHelper.StartSBServices(String hostName, Nullable`1 waitTimeout)
at Microsoft.ServiceBus.Commands.AddSBHost.ProcessRecordImplementation()

ServiceBus Broker service failed to start, retry count 1. Exception message: Operation timed out.. Stack Trace: at System.Fabric.FabricRuntime.Create(Action fabricExitCallback)
at Microsoft.Cloud.ServiceBus.MessageContainerHost.Ring.Join()
at Microsoft.Cloud.ServiceBus.MessageContainerHost.MessageContainerHostComponent.Open()
at Microsoft.Cloud.HostingModel.ComponentHost.OpenComponent(IComponent component, RequestTracker tracker)
at Microsoft.Cloud.HostingModel.ComponentHost.Open()

at Microsoft.ServiceBus.MessageBroker.Backend.OnStart(String[] args)

On the WFM box, the Windows AppFabric service was in stopped state. I started Windows AppFabric service as the Service Bus service is dependent on this. Now when I started the Service Bus Message Broker service and got the error below:

Windows could not start the Service Bus Message Broker service on Local Computer.Error 1053: The service did not respond to the start or control request in a timely fashion.

System.Management.Automation.PipelineStoppedException: The pipeline has been stopped.
at System.Management.Automation.MshCommandRuntime.ThrowTerminatingError(ErrorRecord errorRecord)
at System.Management.Automation.Cmdlet.ThrowTerminatingError(ErrorRecord errorRecord)
at Microsoft.ServiceBus.Commands.AddSBHost.ProcessRecordImplementation()
at Microsoft.ServiceBus.Commands.ServiceBusBaseCmdlet.ProcessRecord()

Microsoft.ServiceBus.Commands.Common.Exceptions.OperationFailedException: Starting service Service Bus Message Broker failed: Time out has expired and the operation has not been completed.
at Microsoft.ServiceBus.Commands.Common.SCMHelper.StartService(String serviceName, Nullable`1 waitTimeout, String hostName)
at Microsoft.ServiceBus.Commands.ServiceBusConfigHelper.StartSBServices(String hostName, Nullable`1 waitTimeout)
at Microsoft.ServiceBus.Commands.AddSBHost.ProcessRecordImplementation()

Decided to go for a set of WFM traces, I find the below failure in the traces, seems the failure :
System.TimeoutException: Operation timed out. ---> System.Runtime.InteropServices.COMException: Exception from HRESULT: 0x80071BFF
at System.Fabric.Interop.NativeRuntime.FabricEndCreateRuntime(IFabricAsyncOperationContext context)

at System.Fabric.FabricRuntime.NativeFabricRuntimeFactory.InitializeFabricRuntimeEndWrapper(FabricRuntime runtime, AsyncCallOutAdapter adapter)

at System.Fabric.Interop.Utility.FinishNativeAsyncInvoke[TResult](String functionTag, Func`2 endFunc, TaskCompletionSource`1 source, AsyncCallOutAdapter adapter, Boolean expectCompletedSynchronously)

--- End of inner exception stack trace ---
at System.Fabric.FabricRuntime.Create(Action fabricExitCallback)

at Microsoft.Cloud.ServiceBus.MessageContainerHost.Ring.Join()
At this point I collected the netstat information (netstat -ano| findstr 9000) for port 9000 which is the default port used by Service Bus Message Broker service. Interesting, I found some custom service listening on this port. Now what? I will make my SB listen to a different port!
I followed the steps to leave the current WFM farm. Selected the custom configuration option for configuring WFM and this time we selected port 9001 (after confirming in netstat information that no other process communicates on port 9001). The configuration completed successfully without any errors.

Hope this helps!

 

Written By
Jainath Ramanathan

Reviewed By
Chirag Pavecha

Microsoft India GTSC.