Scenario 5 : My website is throwing HTTP Error 503. The service is unavailable when browsed over the default cloud service domain url (*.cloudapp.net)

Referring to my blog on Azure Cloud Service Troubleshooting Series, this is the 5th scenario of the lab. Please make sure you have followed the lab setup instructions for Super Convertor application as per this, to recreate the problem.

Symptom

I am getting HTTP Error 503 response while browsing my cloud service application url (https://cloudservicelabs.cloudapp.net/), although my web role 'SuperConvertor' is in running state. Rebooting or Re-imaging the role instance is not resolving the issue.

Service Unavailable

HTTP Error 503. The service is unavailable.

Troubleshooting

Usually when you get 50x errors in your application, it means something is broken at the server side. 503 Service Unavailable server error response code indicates that the server is not ready to handle the request. You must be thinking why suddenly a newly deployed cloud service application suddenly started throwing this error. Is the application crashing ? Is the request reaching the IIS server ? Is the server under high load ?

Generally when I am confused with all of these questions, I follow the basic troubleshooting as I would have done while dealing with an On-Prem IIS server. So I RDP to the web role instance and browse the application locally. Before browsing the site locally, out of curiosity I checked the Application and System event viewer logs to negate any possibility of IIS ApplicationPool crash  or any other application related exceptions. It was absolutely clear !

Next I checked the IIS logs present under C:\Resources\directory\{Deployment ID}.SuperConvertor.DiagnosticStore\LogFiles\Web to check if we can get more information about the HTTP 503 error like sub-status code, time taken to execute the request, etc. It came to my surprise that the directory is absolutely empty, no IIS log has been generated so far !

This means the request is not reaching to the IIS at all. As per IIS architecture, we know that HTTP.sys listens for HTTP requests from the network, passes the requests onto IIS for processing, and then returns processed responses to client browsers. By default, IIS provides HTTP.sys as the protocol listener that listens for HTTP and HTTPS requests and any error at HTTP.sys level is logged in this directory - D:\Windows\System32\LogFiles\HTTPERR. So let's see what we can find in HTTPErr log -

 #Software: Microsoft HTTP API 2.0
#Version: 1.0
#Date: 2018-08-13 03:12:38
#Fields: date time c-ip c-port s-ip s-port cs-version cs-method cs-uri streamid sc-status s-siteid s-reason s-queuename
2018-08-13 03:25:22 293.217.138.127 12052 10.1.2.5 80 HTTP/1.1 GET / - 503 - N/A -
2018-08-13 03:25:22 293.217.138.127 20463 10.1.2.5 80 HTTP/1.1 GET /favicon.ico - 503 - N/A -

Clearly if you see the above log, HTTP 503 is thrown from HTTP.sys level and the client request is getting rejected from there itself without reaching the IIS. Now we are going to browse the site locally from IIS and see what happens - I get an error - This page can’t be displayed. One thing I noticed is that the IIS website was having a binding like below, which means in order to access this particular website we need to access over the custom domain name (www.cloudservicelabs.com)

Websites are accessed by every client using bindings. A typical binding for Websites is in the form of IP:Port:HostHeader. It is a mechanism which tells the server how this site can be reached. The next question that will come to your mind is that from where does this custom hostname came from ?

ServiceDefinition.csdef is the place where you can configure the bindings for your webrole, and here is what I can see for this application:

<WebRole name="SuperConvertor" vmsize="Standard_D1_v2">
<Sites>
<Site name="Web">
<Bindings>
<Binding name="Endpoint1" endpointName="Endpoint1" hostHeader="www.cloudservicelabs.com"/>
</Bindings>
</Site>
</Sites>

In real world scenario, in order to access your cloud service application over a custom hostname, you need to have a DNS configured for this host header corresponding to the cloud service VIP. For now, you can delete the hostHeader attribute from Binding element and redeploy your cloud service solution to resolve the issue.

I hope you have got an idea how to troubleshoot issues related application accessibility in Azure Cloud Service.

Happy Learning !