The improved guide on How to Turn on or off the Report Server Web Service (SSRS 2008 R2)

A couple of weeks ago, during a visit to a customer whose reporting platform, which is architected on top of SQL Server Reporting Services 2008 R2, I was checking to evaluate its health state, I lived a very surreal experience which I don’t wish to anyone.

The situation was like this: Given the distribution of the reporting activity I identified their users were demanding via interactive requests vs subscription-based requests, I proposed to them a given scale-out design where some SSRS front-ends would implement multiple roles, and others would just take one or two specific roles. Those roles we were planning to segregate were the Windows Service, the Web Service, and the Report Manager.

Once we had agreed on the new scale-out layout, I pointed them to the documentation (How to: Turn Reporting Services Features On or Off) so that they could begin implementing the plan by disabling certain features, from those servers where we had planned to do so.

A side note here: The good thing about acting, testing, and measuring effects on non-production environments, before you repeat those exact same steps over “the real thing” (i.e. the one environment in production, on which your whole business relies) is that if something unexpectedly fails, it doesn’t hurt too much. Well, this time it hurt more than we would have liked to. Simply because it took us a not-so-little while to determine the reason why we ended up stuck with a non-working service.

That topic in the documentation mentions the following (pay especial attention to the highlighted part):

You can turn off report server features that you do not use as part of a lockdown strategy for reducing the attack surface of a production report server. In most cases, you will want to run Reporting Services features concurrently to use all of the functionality provided in Reporting Services. However, depending on your deployment model, you can disable the features that you do not require. For example, you can enable only the background processing if all report processing is configured as scheduled operations. Similarly, you can run just the Report Server Web service if you only want interactive, on-demand reporting.

The procedures in this topic show you how to turn off Reporting Services features. Features can be configured in different ways, such as by editing the RsReportServer.config file directly or by using the Surface Area Configuration for Reporting Services facet of Policy-Based Management in SQL Server Management Studio.

Having read the instructions, the proceeded with the plan.

In one of the SSRS servers, they over-acted from what we had planned, and not only turned off the Web Service, but also the Windows Service. According to our plan, on that server they should have only disabled the Windows Service. Only after configuring the whole farm of SSRS servers is that they noticed the additional unplanned change they had made on that one front-end. So they decided to revert it so that this server was one of those that provided Web Service functionality. But, guess what? After “undoing” the change, it still didn’t work.

By their own, they tried as many options as they could think about, in an attempt to bring the web service back to life. From modifying over and over the rsreportserver.config file to force the recycle of the application; to using the Services (services.msc) console to stop the whole SQL Server Reporting Services service hosting the whole stack of SSRS features, including the web service; to even rebooting the whole server (that’s where desperation normally ends up taking you to .)

It still didn’t work. So, at that point they decided to involve me.

After they gave me the high level description of what they had done and the current status, first thing I did was opening the How to: Turn Reporting Services Features On or Off topic I had pointed them to follow, and asked if those were the instructions they had followed or if they had utilized any other “more creative” of their own. As obedient IT Pros they were, they had used what was suggested in the page.

Then, I asked them to show me what were they trying to do to bring the web service back accessible.

One of them edited rsreportserver.config and showed me that the IsWebServiceEnabled element was currently set to false, so they replace it with a true, and saved the file. After completing that action, the log file showed that the application was being recycled because of that change in the configuration file. So, we posted an HTTP request to a URL routed to (and handled by) the web service and waited a bit for the application to be fully loaded. What we saw, though, was what they had already seen and didn’t seem like the expected behavior: In response to our request, the server came back with an HTTP error code 500, whose description said “Reporting Services is unavailable”.

I couldn’t explain it either so, I decided it was time to debug it. After having debugged it and analyzed the relevant parts of the source code of SSRS, the conclusion was that:

If you turn OFF the Web Service using the WebServiceAndHTTPAccessEnabled property (or for that matter the SetServiceState method of the MSReportServer_ConfigurationSetting WMI class which is what it uses behind the scene) which is part of the Surface Area Configuration for Reporting Services facet, accessible for the UI of SQL Server Management Studio, and then try to turn it ON using the supposedly alternative method, consisting in editing the rsreportserver.config file to set the IsWebServiceEnabled element to true, it will not work and, what is worst, you will have the perception that you have been left in an impasse.

So, my strong recommendation here is: if you use the SetServiceState WMI method (whether it is invoked by the facet or through any other mean) to disable the Web Service feature, use the exact same mechanism to re-enable it.

Now, if you are interested in the troubleshooting details, the story went like this:

If you switch off the web service using the facet:

It invokes the WMI method which sets the Configuration->Service->IsWebServiceEnabled configuration to False in rsreportserver.config, but also re-writes Global.asax so that the application inherits from a different class (Microsoft.ReportingServices.WebServer.StoppedApplication) whose BeginRequest event handler, if invoked, would write an HTTP response with an status code 500 and “Reporting Services is unavailable” as the description. The reflected managed code of that class looks like this:

namespace Microsoft.ReportingServices.WebServer

{

    public class StoppedApplication : System.Web.HttpApplication

    {

        private void Application_BeginRequest(Object sender, EventArgs e)

        {

            WriteServiceUnavailable(Response);

            CompleteRequest();

        }

        internal static void WriteServiceUnavailable(HttpResponse response)

        {

            response.Status = "500 - Service unavailable";

            response.StatusCode = 500;

            response.Write(WebServRes.ReportingServiceUnavailable);

        }

    }

}

Because the rsreportserver.config file has been modified (as I said, the SetServiceState method modifies the IsWebServiceEnabled element), the web app domains are eventually cycled, and that triggers the update of the HTTP Endpoint:

ReportingServicesService!CUnmanagedCallback::CreateHttpEndpoint+0xec

mscorwks!CLRToCOMWorker+0x19a

CLRStub[StubLinkStub]@2890ee7

ReportingServicesLibrary!Microsoft.ReportingServices.Library.ServiceAppDomainController.SetWebConfiguration(RunningApplication, Boolean, System.String)+0x2f4

02cff598 0416287b ReportingServicesLibrary!Microsoft.ReportingServices.Library.ServiceAppDomainController.SetWebConfiguration()+0x52

02cff5b0 0201d7ca ReportingServicesLibrary!Microsoft.ReportingServices.Library.ServiceAppDomainController.CycleWebAppDomains(Recycle)+0x8b

02cff61c 0201d12a ReportingServicesLibrary!Microsoft.ReportingServices.Library.ServiceAppDomainController.ServiceMaintenanceInternal()+0x1ba

02cff648 72bf7036 ReportingServicesLibrary!Microsoft.ReportingServices.Library.ServiceAppDomainController.ServiceMaintenance()+0x22

Updating the HTTP Endpoint, meaning that the HTTP.sys request queue stops routing requests targeting the URL of the Web Service.

httpapi!HttpRemoveUrl

ReportingServicesService!CHttpSysApi::UlRemoveUrl+0x59

ReportingServicesService!HttpSrvConn::UpdateListener+0x29

ReportingServicesService!SNIUpdateListener+0x11d

ReportingServicesService!HttpEndpoint::RemoveUrl+0x22

ReportingServicesService!HttpEndpoint::CreateEndpoint+0x589

ReportingServicesService!HttpEndpointCreateEndpoint+0x49

ReportingServicesService!CUnmanagedCallback::HttpEndpointCreateEndpointThread+0x84

ReportingServicesService!SOS_Task::Param::Execute+0xcb

And the consequence of such action is that, from that point forward, any requests the HTTP driver receives on that URL are responded with an HTTP error 503, without ever reaching the BeginRequest handler of the HTTP application implemented in Microsoft.ReportingServices.WebServer.StoppedApplication.Application_BeginRequest. So, the experience of the user who attempts to reach the URL of the Web Service is that it gets the following response:

If the user tries to access the Report Manager, and the Report Manager is enabled, it fails like this:

Because Report Manager depends on the Web Service as shown below:

0:055> !pe

Exception object: 0af9ff70

Exception type: System.Net.WebException

Message: The remote server returned an error: (503) Server Unavailable.

InnerException: <none>

StackTrace (generated):

<none>

StackTraceString: <none>

HResult: 80131509

KERNELBASE!RaiseException+0x58

mscorwks!RaiseTheExceptionInternalOnly+0x2a8

mscorwks!JIT_Throw+0xfc

System_ni!System.Net.HttpWebRequest.GetResponse()+0x5247ae

System_Web_Services_ni!System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(System.Net.WebRequest)+0xd5

System_Web_Services_ni!System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(System.Net.WebRequest)+0x5

ReportingServicesWebUserInterface!Microsoft.ReportingServices.UI.Global+RSWebServiceWrapper.GetWebResponse(System.Net.WebRequest)+0xe

System_Web_Services_ni!System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(System.String, System.Object[])+0xad

Microsoft_ReportingServices_Diagnostics_53190000!Microsoft.SqlServer.ReportingServices2010.ReportingService2010.IsSSLRequired()+0x20

ReportingServicesWebUserInterface!Microsoft.SqlServer.ReportingServices2010.RSConnection2010.SetConnectionProtocol()+0x45

ReportingServicesWebUserInterface!Microsoft.SqlServer.ReportingServices2010.RSConnection2010+SoapMethodWrapper`1[[System.__Canon, Method(Boolean)+0x3f

ReportingServicesWebUserInterface!Microsoft.SqlServer.ReportingServices2010.RSConnection2010.ListItemTypes()+0x57

ReportingServicesWebUserInterface!Microsoft.ReportingServices.UI.Global.SecureAllAPI()+0x12

ReportingServicesWebUserInterface!Microsoft.ReportingServices.UI.ReportingPage.EnsureHttpsLevel()+0x1b

ReportingServicesWebUserInterface!Microsoft.ReportingServices.UI.ReportingPage.ReportingPage_Init(System.Object, System.EventArgs)+0x57

CLRStub[StubLinkStub]@166afe2

System_Web_ni!System.Web.UI.Control.OnInit(System.EventArgs)+0x64

System_Web_ni!System.Web.UI.Page.OnInit(System.EventArgs)+0xd

System_Web_ni!System.Web.UI.Control.InitRecursive(System.Web.UI.Control)+0x14e

System_Web_ni!System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)+0x17b

System_Web_ni!System.Web.UI.Page.ProcessRequest(Boolean, Boolean)+0x84

System_Web_ni!System.Web.UI.Page.ProcessRequest()+0x51

System_Web_ni!System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)+0x16

System_Web_ni!System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)+0x32

App_Web_wnrsat_u!ASP.pages_folder_aspx.ProcessRequest(System.Web.HttpContext)+0x5

System_Web_ni!System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+0xb6

System_Web_ni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+0x4c

System_Web_ni!System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(System.Exception)+0x133

System_Web_ni!System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object)+0x7c

To this point, the experience would be exactly the same no matter how the user turned off the Web Service, whether he used the WMI method (SetServiceState) or whether he manually edited the rsreportserver.config file.

Now, the problem appears when he tries to turn it back on. If the user turned the Web service off using the WMI method (that is what the facet uses behind the scenes), and he turns it back on using the same WMI method, the initial behavior is restored (i.e. the config file is modified to set IsWebServiceEnabled to True, which eventually triggers the cycling of the web app domains, and the Global.asax is re-written to inherit from the original Microsoft.ReportingServices.WebServer.Global class.

But, and this is where the problem comes, if the user turns the Web service off using the WMI method, and tries to turn it back on by manually editing the rsreportserver.config file to set IsWebServiceEnabled to True, that will add the URL of the Web service back, so that HTTP.sys begins routing any incoming requests to the web service. This is fine, only that the request handler that will receive new requests going forward is the one implemented in the Microsoft.ReportingServices.WebServer.StoppedApplication class. Only the implementation of the SetServiceState method of the WMI provider overwrites the contents of the Global.asax so that it points back to the original class Microsoft.ReportingServices.WebServer.Global. Since manually editing the config file misses that part of the process, what would happen is that new requests will end up being processed by that handler which, as could be read in its code above, simply responds with an HTTP 500. What the user would see when accessing the web service is this:

With that evidence, the user could suspect he made some mistake while turning the web service on through the config file. But he opens it, and it shows fine. Even the log file shows that the setting is properly set:

library!DefaultDomain!1568!06/27/2012-13:47:19:: i INFO: Initializing IsWebServiceEnabled to 'True' as specified in Configuration file.

However, it doesn’t quite work. So, as a second option, they may try to enable it from the facet (via SSMS), and they discover that the facet says it is already enabled (because its value is actually taken from the value of the IsWebServiceEnabled node in the config file):

No trace is left in any log file that could help the user understand what the root cause of the problem is. And that, if they want to solve it in a supported way, they must use the WMI method. If they want to use the facet exposed via SSMS, that implies setting it to False again, and then back to True.

What could be the justification for this different behavior then?

Going through the history of the relevant WMI provider file, a guess can be taken at the answer. The method that modifies global.asax dates back to SSRS 2005, when the web services was an IIS application. There was no way for IIS to honor the <IsWebServiceEnabled> element. So the guess here is that <IsWebServiceEnabled> was simply a flag for the WMI provider. The real logic of disabling web service was in the global.asax change. However, from SSRS 2008 on the web server was hosted in the SSRS exe, so it could honor the <IsWebServiceEnabled> element. The bit of history about global.asax is probably lost in the times.

So, the bottom line is that SAC/WMI makes additional modification to global.asax. It should be better documented, and ideally the WMI provider should avoid changing global.asax.

A bug has been filed to track this problem and improve, in the short term, the documentation, and in a longer term, the implementation.