Reporting Services HTTP 401 (Unauthorized) - Host Headers require your attention

In a previous post, I talked about issues with repeated login prompts when trying to access either Report Server or Report Manager using your browser. 

Recently, during an upgrade of our internal test server we identified a new type of HTTP 401 (Unauthorized) message.  This post provides a description of the problem and the solution.   

Symptoms:

When all of the following symptoms apply, it could be this issue:

A. When accessing Report Manager or Report Server you use a URL that looks like http(s)://<foo>/reports. The name <foo> is NOT the computer name of the computer on which Report Server and Report Manager are deployed.  However, DNS or the machine's hosts or lmhosts files (WINDOWS\system32\drivers\etc) are configured to send requests to <foo> back to the report server computer.

B. When accessing Report Server directly (e.g. http(s)://<foo>/reportserver) you get no problems - you can navigate the report server namespace and you can view all reports.

C. However when you access Report Manager (e.g. http(s)://<foo>/reports), you get what looks like the Report Manager UI but where you expect to see a list of reports, you get an error saying HTTP 401 (Unauthorized). If using SharePoint integrated mode, the error can appear on the SharePoint UI pages.

D. The ReportServerUrl element is specified in the rsreportserver.config file and is set to a URL that looks like http(s)://<foo>/reportserver.

The symptoms are most likely to occur when Reporting Services is deployed in a scale-out deployment or when Reporting Services is accessed using a user friendly name that does not match the machine name of the computer.

Solution:

<foo> is considered a "Host Header".  It is an alternate name for the computer on which SSRS is installed.  You will need to add the NetBIOS and Fully Qualified Domain Name (FQDN) for <foo> to the list of BackConnectionHostNames stored in the Windows Registry.  Casing does not matter. Use the steps in Method 2: Specify host names in KB 896861, with the following adjustment.   Step 7 of the KB article says "Quit Registry Editor, and then restart the IISAdmin service."  Instead, just reboot the computer.   This is the correct solution for security reasons that I'll explain later in this article.

For example, if <foo> is a Windows machine name like “contoso”, then it likely also can be referenced in FQDN form as “contoso.domain.com”. You will need to add both representations to the list in BackConnectionHostNames.

Cause:

Report Server and Report Manager communicate using a network connection.   When configured for a scale-out deployment of Reporting Services, typically a single virtual server name is given to the deployment and the underlying machine names are never used by end-users.   The same can happen even for non-scale-out deployments but I would consider it less common.  For example, you might call the server www.contoso.com or on the internal network simply "contoso".    These are called Host Headers.  We have only observed the issue in this post when using a Host Header; we have not seen it when using just the machine name. 

By default, Report Manager is configured for Windows authentication. In this mode, it impersonates the user making the request and uses that user's credentials to connect to the Report Server.  Because the network request is local, the authentication between RM & RS can succeed without running into a double hop authentication issue.  In the case described above, it fails mysteriously.  Let's see why...

The exact authentication mode used by both RS and RM is defined in rsreportserver.config in the <AuthenticationTypes> element.  When <AuthenticationTypes> includes <RSWindowsNTLM/> then RM & RS can use NTLM authentication.   NTLM is generally a destination agnostic authentication scheme.  What this means is that the client does not have mutual authentication and cannot be sure which machine it authenticated to. 

When you perform NTLM authentication, you can optionally specify an intended destination in the form of an SPN.   This helps mitigate authentication reflection attacks. Not all clients (e.g. browsers) or APIs packages (e.g. .Net Framework) do this by default.  The specific change we observed is that the .Net Framework 3.5 SP1 now defaults to specifying the Host Name used in the request URL in an SPN in the NTLM authentication package.  So if you make a URL request to https://<foo>/reportserver then the SPN “HTTP/<foo>” is added to the authentication information.  This is good news from a security perspective, but has an unfortunate consequence.

As an aside - slide 24 of Jesper Johansson’s PowerPoint slide deck (download.microsoft.com) on authentication shows how these reflection attacks work.

You might think that since the user accessing Report Manager accessed https://<foo>/reports and then the report manager accessed https://<foo>/reportserver, everything should just work.  However, Windows has a special case when you do NTLM authentication on the local computer (i.e. loop back connection) that affords protection against authentication reflection attacks. Part of the NTLM authentication process centers on a challenge issued by the destination computer and sent back to the client computer.   Windows keeps track of the challenges it issues.  If Windows receives a challenge it itself generated Windows will fail the authentication unless the connection is a loop back connection. 

To determine whether the connection is a loop back connection, Windows reads the destination contained in the authentication information supplied by the client.  If the destination is not specified or maps to the local machine then Windows allows the authentication to proceed.   In our failing case, <foo> is neither the machine name nor the loop back IP address nor the machine's IP address, so Windows fails the authentication requests.  This causes the requests between Report Manager and Report Server to fail with 401 (Unauthorized). 

To address the issue, we need to tell Windows that <foo> is actually an alternate name for the local computer.  The correct way to do that is to use the BackConnectionHostNames registry entry as described in Method 2: Specify host names in KB 896861.   Solution Method 1: Disable the loopback check will result in a less secure system, as it disables the protection against reflection attacks. It is better to constrain the set of alternate names to only those you expect the machine to actually use.