Troubleshooting App Management WCF Endpoint Failures

I was recently setting up federated search for SharePoint 2013 using Result Sources to a Remote SharePoint farm. That has already been extremely well documented by Steve Peschka, but did want to dive into the following error with the App Management service that I encountered along the way because it helps illustrate tactics for troubleshooting endpoint failures.

Personally, the message "Get-SPAppPrincipal : There are no addresses available for this application" gives me little indication as to what's actually occurring, but the corresponding message in ULS gives us many more clues:

Microsoft.SharePoint.SPEndpointAddressNotFoundException: There are no addresses available for this application.   
 at Microsoft.SharePoint.SPRoundRobinServiceLoadBalancer.BeginOperation()   
 at Microsoft.SharePoint.Administration.SPServiceApplicationProxyBase`1. ExecuteOnChannel(Boolean requireDelegation, Action`1 codeBlock)   
 at Microsoft.SharePoint.AppManagement.AppManagementServiceApplicationProxy. GetScaleOutDatabaseMap()   
  ...snipped for brevity...  

This particular SPEndpointAddressNotFoundException exception is fairly generic in that it can occur for just about any SharePoint service application WCF Endpoint (e.g. Topology, Managed Metadata, Search Query, Search Administration, Security Token Service, etc), but in this case it occurred for the App Management service application (notice AppManagement.AppManagementServiceApplicationProxy in the call stack). So the root cause for this specific occurrence is actually pretty simple... I didn't have the App Management service application created (Central Admin > Manage Service Applications) and didn't have the App Management service instance started (Central Admin > Services on Server).

But I promised troubleshooting steps, so let's dig in...

When it comes to explaining the SharePoint Service Application architecture - an essential aspect for understanding Endpoint failures, no one has explained this better than Spencer Harbar. In this post he states:

"When you start a service machine instance for which there is an associated Service Application, an IIS Virtual Application will be created within the SharePoint Web Services IIS Web site. This will include the Service Application Endpoint (a WCF or ASMX). Each service application must expose a service application endpoint. The service application endpoint is only created on the machine(s) hosting the service machine instance. "

To see the current endpoints URLs for any given service application, use the following PowerShell:

$foo = Get-SPServiceApplication | where {$_.Name -like "App Management"}
foreach ($pt in $foo.EndPoints) { $pt.ListenUris }

However, when I ran this in my environment, I found that there were no endpoint URLs reported for this service application as seen below...

Likewise, if this endpoint were to exist, we would expect to see the GUID of the Service Application (in this example, d30a17fbc93945a18333ad2f2b303691 as seen in the output above) under the list of SharePoint Web Services. Again, it's missing here too (which is expected based off the output above) 

For what it's worth, if you had the endpoint URL reported in the PowerShell output, but do not see it IIS on the applicable server, run the following PowerShell (with the appropriate Service Application name) to recreate the service in IIS:

$foo = Get-SPServiceApplication | where {$_.Name -like "App Management"}
$foo.Provision()

However, if the service is missing in PowerShell (as above) as well as missing in IIS, then the service instance is likely not started on the applicable server. In this case, go to Central Admin > Services on Server...

And if needed, start the service instance... 

 Then, go back to IIS and verify the d30a17fbc93945a18333ad2f2b303691 sub-site was created... 

And then finally, go back to PowerShell to see the endpoint URLs..

It's worth reiterating that although this post focused on the App Management service application, endpoint failures can occur for just about any SharePoint service application and these steps should be applicable for those as well.

I hope this helps!!

References:

• "There are a number of components within a 'Service Application'. They are Service Instance > Service Machine Instance > Service Application Endpoint > Service Application > Service Connection (aka Service Application Proxy) > Consumers (Web Applications)" (Spencer Harbar: In a Nutshell: SharePoint 2010 Service Applications)
• In WCF, " [a]n endpoint describes in a standard-based way where messages should be sent, how they should be sent, and what the messages should look like. A service can expose this information as metadata that clients can process to generate appropriate WCF clients and communication stacks." (MSDN: Fundamental Windows Communication Foundation Concepts)