Troubleshooting Endpoints on a WCF Web Role

Over the past couple of weeks, I’ve been working on enhancing a Windows Azure example application from our recent TechJam event series. I’m planning to post that code soon (and dedicate a few blogs posts to it), but there was one aspect of the application (specifically detailed in Step 5 below) that I struggled with for a few days and so feel merits its own coverage.  If you’re well versed with the topic of internal and external endpoints on Windows Azure roles, you may want to jump right to the meat of the post!

  1. The scenario
  2. Internal endpoint configuration
  3. Service implementation
  4. Client implementation
  5. Gotchas
  6. Resolution

The Scenario

My application includes a Windows Azure Web Role that houses two public WCF Services and two other Worker roles.  In an attempt to centralize some of the configuration, I decided to add a third service to the Web Role that only the two Worker Roles would use, and to do so I’d expose an internal endpoint on the Web Role.  The architecture looks something like below, where svc1 and svc2 are the two public WCF services, and svc3 is the ‘new’ service accessible only via the internal endpoint. (I’ve included a pared down version of this architecture – with one versus two Worker Roles - as an attachment to this post). Sample Service Architecture

The tricky part here was getting ALL of these services working at the same time.  There’s plenty of advice about setting up services on internal endpoints (see, for instance, Alfredo Delsors’ blog post and episode 12 of Cloud Cover), but little on the gotchas of mixing internally and externally accessible services. 

I’m going to take this opportunity to re-cover some of the ground in the resources I mentioned above, so skip down to the ‘gotchas' if you don’t need the review.

.

Internal Endpoint Configuration

Creating a new internal endpoint for the Web Role is straightforward either via the Role property pages or directly in the ServiceDefinition.csdef file; for example, here I’ve defined TWO internal HTTP endpoints in addition to the default input endpoint, which I renamed from Endpoint1 to ExternalHttpEndpoint.  Note, you can define up to five internal endpoints per role.

Endpoint configuration

 

Service Implementation

Creating a new WCF Service in existing Web Role

This is the easy part, you simply create a new WCF Service in your Web Role project, leaving you with a Service1.svc, IService1.cs, and Service1.svc.cs files in which to code the service interface and implementation.

New service dialog

 

Listening on the Internal Endpoint

With no other changes, the new service will work just fine, but be accessible to the public since it’s using the default ServiceHostFactory behind the scenes.  To have the service accessible on only the internal endpoint, we need a custom ServiceHost implementation that sets up the service to be bound to only the internal endpoint address.  Here’s one option, specifically for a service named Service1 with an interface of IService1 (by the end of the article, this will evolve to a more general implementation):

 using System;
using System.ServiceModel;

using System.ServiceModel.Activation;

using Microsoft.WindowsAzure.ServiceRuntime;
namespace Service
{
    public class MyServiceHostFactory : ServiceHostFactory
    {
        protected override ServiceHost CreateServiceHost(Type serviceType, Uri[] baseAddresses)
        {
            String internalEndpointAddress =
                string.Format("https://{0}/Service1.svc",
                RoleEnvironment.CurrentRoleInstance.InstanceEndpoints["InternalHttpEndpoint"].IPEndpoint);
            var host = new ServiceHost(typeof(Service1), new Uri(internalEndpointAddress));
            host.AddServiceEndpoint(typeof(IService1), new BasicHttpBinding(), String.Empty);
            return host;
        }
    }
}

The code to determine the internal endpoint is highlighted, and in this case presumes there’s an HTTP endpoint named InternalHttpEndpoint define on the role instance hosting the service.  With that endpoint, we simply instantiate a new ServiceHost instance with an endpoint address corresponding to the internal port and the standard naming convention for a WCF service.

Pointing IIS to the custom ServiceHost implementation

The only thing left is to tell IIS to use the custom ServiceHostFactory implementation, which is done in the markup for the Service1.svc file shown below:

 <%@ ServiceHost Language="C#" Debug="true" 
                Service="Service.Service1" 
                CodeBehind="Service1.svc.cs" 
                Factory = "Service.MyServiceHostFactory"
%>

 

Client Implementation

For the most part, the client implementation is no different from any other WCF client, but there are a few things to note:

  1. Since the service is hosted on an internal port, the only ‘clients’ that can access it are other roles deployed in the context of the same Windows Azure cloud service deployment.  Referring to the architecture diagram at the beginning of the post, this means only instances of Worker Role 1 and Worker Role 2 can reach the service.  In particular, you cannot access a service on an internal endpoint from a role in a different Windows Azure-hosted application; you’ll have to access it via an input endpoint, just as an external client would.
  2. The endpoint address is not static.  The Windows Azure Fabric controller handles both the assignment of IP address and port when starting roles within Windows Azure, so you will need to access the endpoint programmatically, and we’ll get to that shortly.
  3. Internal endpoints are not load balanced.  If you once again refer to the opening diagram, you’ll see that external requests come through a published address, something like myapp.cloudapp.net:8080, where 8080 is the port number assigned to the input endpoint as part of the Web Role configuration.   The load balancer routes that request to one of the instances of the Web Role and translates the external address to the appropriate private IP address and port of the selected instance.  With internal endpoints, a direct connection is made to the targeted address.  With not too much code though a rudimentary load balancer can be included as part of the client implementation.

Here’s a possible client implementation targeting the service above:

    1:  var epList = (from role in RoleEnvironment.Roles["WebRole"].Instances
   2:                      select role.InstanceEndpoints["InternalHttpEndpoint"]);
   3:  var selectedAddress = epList
   4:                        .Skip(new Random(Environment.TickCount).Next(epList.Count()))
   5:                        .Select(ep => ep.IPEndpoint.ToString())
   6:                        .First();
   7:   
   8:  var factory = new ChannelFactory<Service.IService1>(new BasicHttpBinding());
   9:  Service.IService1 instance = factory.CreateChannel(
  10:      new EndpointAddress(String.Format("https://{0}/Service1.svc", selectedAddress)));
  11:   
  12:  txtResult.Text = instance.Echo("Hello");
 

The first LINQ query (Lines 1 – 2) selects the IPEndpoint information for the InternalHttpEndpoint from each of the instances of the role named WebRole.  The next LINQ statement selects a random element from that list to serve as a rudimentary load balancing scheme.  From there on out, it’s standard WCF client coding, with Line 12 setting a TextBox to the output of a simple Echo method which has been defined on the IService1 interface.

 

Gotchas

The first hurdle

You knew from the outset it wasn’t going to be that easy, right?  If you follow the steps above and run your application, you’ll find that the invocation of the internally accessible service will fail, with an exception along the lines of:

The requested service, 'https://127.0.0.1:5204/Service1.svc' could not be activated. See the server's diagnostic trace logs for more information.

More details can be found in Event Viewer’s Application logs:

 WebHost failed to process a request. 
Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/12547953 
Exception: System.ServiceModel.ServiceActivationException: The service '/Service1.svc' 
cannot be activated due to an exception during compilation. The exception message is: 
The ChannelDispatcher at 'https://127.0.0.1:5204/Service1.svc' with contract(s) 
'IService1' is unable to open its IChannelListener.. 
---> System.InvalidOperationException: The ChannelDispatcher at 
'https://127.0.0.1:5204/Service1.svc' with contract(s) 'IService1' is unable to open 
its IChannelListener
---> System.InvalidOperationException: No protocol binding matches 
the given address 'https://127.0.0.1:5204/Service1.svc'. 
Protocol bindings are configured at the Site level in IIS or WAS configuration. 

at System.ServiceModel.Activation.HostedAspNetEnvironment.GetBaseUri(String transportScheme, Uri listenUri) 
at System.ServiceModel.Channels.TransportChannelListener.OnOpening() 
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) 
at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout) 
...

From the IP address, you can tell I was running this in the local development fabric (emulator), but you’d see the same error when running in the cloud, but instead referring to a private IP address internal to the Windows Azure data center where your application is running (something like 10.2.100.3). 

The port number referenced is indeed the internal HTTP port, something you can confirm when debugging, by adding the following to the custom ServiceHost implementation:

       foreach (var a in RoleEnvironment.CurrentRoleInstance.InstanceEndpoints)
         Debug.WriteLine("Role Endpoint: " + a.Key + 
                   " " + a.Value.IPEndpoint.ToString());

 

The root cause of the error is the setting in the web.config which is added by the WCF Web Role template in Visual Studio (and thanks to Alfredo Delsors’ blog post for noting that):

    <serviceHostingEnvironment multipleSiteBindingsEnabled="true" />

Now, why does that setting matter here?  If you add a bit more debug code to your custom CreateServiceHost method

      foreach (var a in baseAddresses)
        Debug.WriteLine("Base Address: " + a.AbsoluteUri);

you’ll find that with the multipleSiteBindingsEnabled=true, the baseAddress array argument contains only one endpoint, the one corresponding to the input endpoint, so the internal endpoint that you want to bind to is unavailable.

So, change the value of multipleSiteBindingsEnabled to false, and you’re good to go!  Well, sort of…. The internal service works fine, but…..

The second hurdle

When accessing the external WCF services now, you’ll see a different exception:

 WebHost failed to process a request. 
Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/12547953 

Exception: System.ServiceModel.ServiceActivationException: The service '/ExternalService.svc' 
cannot be activated due to an exception during compilation. The exception message is: This 
collection already contains an address with scheme http. There can be at most one address 
per scheme in this collection. If your service is being hosted in IIS you can fix the 
problem by setting 
'system.serviceModel/serviceHostingEnvironment/multipleSiteBindingsEnabled' to true or 
specifying 'system.serviceModel/serviceHostingEnvironment/baseAddressPrefixFilters'.

For the WCF-savvy that’s a fairly common error which occurs because each service is allowed only one base address for each transport .  Here transport == http.

Two remedies are suggested in the exception message:

set multipleSiteBindingsEnabled to true: well, we know this won’t work because it breaks the invocation of the service on the internal port

specify baseAddressPrefixFilters : this is a configuration option added in .NET 3.5, but as far as I can tell is not something you can programmatically access, and since we don’t know what the port number will be for the Windows Azure role endpoints, it’s not an option.

 

Resolution

At first it seemed like a Catch-22, but the remedy isn’t that hard:

  1. Set multipleSiteBindingsEnabled set to false (the default), and
  2. Create a custom ServiceHostFactory for all of the services on the Web Role, including the ‘normal’ external ones.

The one aspect of this I didn’t like though is having multiple ServiceHostFactory classes that differed very slightly, namely the name of the port and the service interface and implementation class.  I opted for a more general implementation by introducing a convention in my code and endpoint naming:

  1. Web Role endpoints are named <endpoint-prefix>HttpEndpoint
  2. ServiceHostFactory derived class is named <endpoint-prefix>ServiceHostFactory

An abstract implementation of ServiceHostFactory manages the convention, and I simply need to define concrete classes with names complying with that convention.   Here’s my implementation:

 using System;
using System.ServiceModel;

using System.ServiceModel.Activation;

using Microsoft.WindowsAzure.ServiceRuntime;
namespace Service
{
    // ServiceFactory uses following conventions:
    // 1.  Subclass of factory is named <endpoint-prefix>ServiceFactory
    // 2.  Azure role class has endpoint with name <endpoint-prefix>HttpEndpoint
    public abstract class AzureServiceHostFactory : ServiceHostFactory
    {
        protected override ServiceHost CreateServiceHost(Type serviceType, Uri[] baseAddresses)
        {
            String factoryClass = this.GetType().Name;
            String endpointName = 
                String.Format("{0}HttpEndpoint", 
                        factoryClass.Substring(0, factoryClass.IndexOf("ServiceHostFactory")));
            String externalEndpointAddress =
                string.Format("https://{0}/{1}.svc",
                RoleEnvironment.CurrentRoleInstance.InstanceEndpoints[endpointName].IPEndpoint,
                serviceType.Name);
            var host = new ServiceHost(serviceType, new Uri(externalEndpointAddress));
            var basicBinding = new BasicHttpBinding();
            foreach (var intf in serviceType.GetInterfaces())
                host.AddServiceEndpoint(intf, basicBinding, String.Empty);
            return host;
        }
    }
    public class ExternalServiceHostFactory : AzureServiceHostFactory { }
    public class InternalServiceHostFactory : AzureServiceHostFactory { }
}

and in keeping with my convention, I have two endpoints defined on my Web Role:

Role Endpoint configurations

 

Take a look at the sample project if you want to tinker around more, and let me know if you see any opportunities for a cleaner implementation.  I’m wondering if some dependency injection might eliminate the sole reliance on configuration here, but that was a bit out of the scope of what I wanted to cover for now.