Why System.Net.WebException comes up post .NET Framework 3.5 SP1?

In the recent past I have come across several issues where our customers have started running into System.Net.WebException while trying to access a web resource (web service, webpage etc) hosted locally on the same machine using some HostName. This issue started coming up after installing SP1 for .NET Framework 3.5 . The exception details are similar to the following:

Server Error in '/TestApplication' Application.


The remote server returned an error: (401) Unauthorized.

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.Net.WebException: The remote server returned an error: (401) Unauthorized.
Source Error:

The source code that generated this unhandled exception can only be shown when compiled in debug mode. To enable this, please follow one of the below steps, then request the URL: 1. Add a "Debug=true" directive at the top of the file that generated the error. Example:   <%@ Page Language="C#" Debug="true" %> or: 2) Add the following section to the configuration file of your application: <configuration>    <system.web>        <compilation debug="true"/>    </system.web> </configuration> Note that this second technique will cause all files within a given application to be compiled in debug mode. The first technique will cause only that particular file to be compiled in debug mode. Important: Running applications in debug mode does incur a memory/performance overhead. You should make sure that an application has debugging disabled before deploying into production scenario.

Stack Trace:

 [WebException: The remote server returned an error: (401) Unauthorized.]   System.Net.HttpWebRequest.GetResponse() +5313085   ASP.default_aspx.btnCallService_Click(Object sender, EventArgs e) +60   System.Web.UI.WebControls.Button.OnClick(EventArgs e) +111   System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument) +110   System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String eventArgument) +10   System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument) +13   System.Web.UI.Page.RaisePostBackEvent(NameValueCollection postData) +36   System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +1565

Version Information: Microsoft .NET Framework Version:2.0.50727.3053; ASP.NET Version:2.0.50727.3053

After investigating this further we found that while accessing web-resource NTLM Authentication was being used. To reproduce the issue, create a simple Test Web-Site running on some different port. Also add a hostname to the site as www.test.com and add an entry for it in your Host file (located at C:\Windows\System32\drivers\etc). Add a simple asp page in this Test Web Site which displays the current time. The code for it will look like below -

 <%=Now()%>

In the Default Web Site add a page called default.aspx. The code for that will look like below -

 <%@ Page Language="C#" %>
<%@ Import Namespace="System.IO" %>
<%@ Import Namespace="System.Net" %>
<script runat="server">

    protected void btnCallService_Click(object sender, EventArgs e)
    {
        WebRequest request = WebRequest.Create("https://www.test.com:81/Time.asp");
        request.Method = "GET";
        request.Credentials = CredentialCache.DefaultNetworkCredentials;
        WebResponse response = request.GetResponse();
        StreamReader reader = new StreamReader(response.GetResponseStream());
        string responseFromServer = reader.ReadToEnd();
        Response.Write (responseFromServer);
        response.Close();
    }
</script>
<body>
    <form id="form1" runat="server">
        <asp:Button ID="btnCallService" runat="server" onclick="btnCallService_Click" 
            Text="Invoke Web Service" />
    </form>
</body>
</html>

Now browse the Default.aspx and click on button and expected behavior is it should show the current time in the browser. But instead we end up getting exception mentioned above.

The reason for the exception is post .Net Framework 3.5 SP1 Windows authentication defaults to NTLM authentication when the CustomTargetNameDictionary property is not set. NTLM authentication includes a challenge issued by the destination computer and sent back to the client computer. When a computer receives a challenge generated by itself, the authentication will fail resulting in exception mentioned as above unless the connection is a loop back connection. You can find additional details in the article.

There are two possible methods for a server application to work around this change. The recommended approach is to map the host name used in the request URL to the BackConnectionHostNames key in the registry on the server. The BackConnectionHostNames registry key is normally used to map a host name to a loopback address. To specify the host names that are mapped to the loopback address and can connect to Web sites on a local computer, follow these steps -

1. Click Start, click Run, type regedit, and then click OK.

2. In Registry Editor, locate and then click the following registry key:

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0

3. Right-click MSV1_0, point to New, and then click Multi-String Value.

4. Type BackConnectionHostNames, and then press ENTER.

5. Right-click BackConnectionHostNames, and then click Modify.

6. In the Value data box, type the host name or the host names for the sites (the host name used in the request URL) that are on the local computer, and then click OK. In our case it would be www.test.com.

7. Quit Registry Editor, and then restart the IISAdmin service and run IISReset.

A less secure work around is to disable the loop back check, as described in this article. This disables the protection against reflection attacks. So it is better to constrain the set of alternate names to only those you expect the machine to actually use.

I hope this helps!