502 - Web server received an invalid response while acting as a gateway or proxy server

Context:

There is time where you may use an Application Request Routing (ARR) with a claims base integrated Web Application using WIF.

In this case, the user should possess a SAML token granted by an IDP like ADFS before accessing the Web App.

However, the following exception occurs when the ARR module tries to redirect the call to a different application:

502 - Web server received an invalid response while acting as a gateway or proxy server.

There is a problem with the page you are looking for, and it cannot be displayed.

When the Web server (while acting as a gateway or proxy) contacted the upstream content server, it received an invalid response from the content server.”

To understand what happened, let’s investigate a FREB log:

Investigation:

From the FREB traces we saw that the ARR module was trying to read the entity body of the request.

However, BytesReceived returned “0” which caused the 502 exception to occur:

155.

NOTIFY_MODULE_END

ModuleName="ScriptModule-4.0", Notification="REQUEST_ACQUIRE_STATE", fIsPostNotificationEvent="true", NotificationStatus="NOTIFICATION_CONTINUE"

19:15:47.104

156.

NOTIFY_MODULE_START

ModuleName="ApplicationRequestRouting", Notification="EXECUTE_REQUEST_HANDLER", fIsPostNotification="false"

19:15:47.104

157.

i

ARR_REQUEST_HEADERS_START

19:15:47.104

158.

i

ARR_REQUEST_HEADERS_END

19:15:47.104

159.

i

GENERAL_READ_ENTITY_START

19:15:47.104

160.

i

GENERAL_READ_ENTITY_END

BytesReceived="0", ErrorCode="Reached the end of the file.

(0x80070026)"

19:15:47.104

161.

i

ARR_RESPONSE_HEADERS_START

19:15:47.104

162.

r

MODULE_SET_RESPONSE_ERROR_STATUS Warning

ModuleName="ApplicationRequestRouting", Notification="EXECUTE_REQUEST_HANDLER", HttpStatus="502", HttpReason="Bad Gateway", HttpSubStatus="3", ErrorCode="2147954402", ConfigExceptionInfo=""

19:17:47.495

163.

r

SET_RESPONSE_ERROR_DESCRIPTION Warning

ErrorDescription="The operation timed out

"

19:17:47.495

164.

NOTIFY_MODULE_END

ModuleName="ApplicationRequestRouting", Notification="EXECUTE_REQUEST_HANDLER", fIsPostNotificationEvent="false", NotificationStatus="1"

19:17:47.495

165.

NOTIFY_MODULE_COMPLETION

ModuleName="ApplicationRequestRouting", Notification="EXECUTE_REQUEST_HANDLER", fIsPostNotificationEvent="false", CompletionBytes="0", ErrorCode="The operation completed successfully.

(0x0)"

19:17:47.495

166.

NOTIFY_MODULE_END

ModuleName="ApplicationRequestRouting", Notification="EXECUTE_REQUEST_HANDLER", fIsPostNotificationEvent="false", NotificationStatus="NOTIFICATION_CONTINUE"

19:17:47.495

How can we explain that?

According to MSDN, the request entity body was null at the time ARR tried to read it because it was previously read by another module.

IIS does not maintain a copy of the request after it has been read. Therefore, it is recommended that only the handler for an HTTP request should read the request entity.

The HttpRequest.InsertEntityBody method overload provides IIS with a copy of the request entity if it was previously read by ASP.NET. This method overload is useful for cases where ASP.NET has read the entity request and you want to reuse the existing request data.”

Thus, we need to look for the module that read the entity body prior to ARR module. I checked the activities that occurred before ARR attempted to the Entity Body and saw this:

66.

NOTIFY_MODULE_START

ModuleName="WSFederationAuthenticationModule", Notification="AUTHENTICATE_REQUEST", fIsPostNotification="false"

19:15:47.104

67.

AspNetPipelineEnter

Data1="System.IdentityModel.Web.WSFederationAuthenticationModule"

19:15:47.104

68.

i

GENERAL_READ_ENTITY_START

19:15:47.104

69.

i

GENERAL_READ_ENTITY_END

BytesReceived="804", ErrorCode="The operation completed successfully.

(0x0)"

19:15:47.104

70.

GENERAL_REQUEST_ENTITY

Buffer= “Some Contents"

19:15:47.104

71.

AspNetPipelineLeave

Data1="System.IdentityModel.Web.WSFederationAuthenticationModule"

19:15:47.104

72.

NOTIFY_MODULE_END

ModuleName="WSFederationAuthenticationModule", Notification="AUTHENTICATE_REQUEST", fIsPostNotificationEvent="false", NotificationStatus="NOTIFICATION_CONTINUE"

19:15:47.104

73.

NOTIFY_MODULE_START

ModuleName="AnonymousAuthenticationModule", Notification="AUTHENTICATE_REQUEST", fIsPostNotification="false"

19:15:47.104

So the WSFederationAuthenticationModule Module read the Entity Body prior to ARR Module.

This is a known System.Web bug. Unfortunately, there is no official hotfix created by Microsoft for this issue.

Workaround:

1- Use the new Specification called OWIN if you would like to couple ARR and WIF in the same Web App.

2- Call HttpRequest.InsertEntityBody to ensure that you give back the entity body to IIS after it is read by WSFederationAuthenticationModule when using System.Web

You can add the Entity Body back after it is used by WSFederationAuthenticationModule as follows:

add Application_BeginRequest in the Global.asax's Application Begin request as follows:

 

    1:  
    2: void Application_BeginRequest(Object sender, 
    3: EventArgs e) {
    4: HttpApplication application = (HttpApplication)sender;
    5:  
    6:  HttpRequest request = application.Context.Request;
    7:  
    8: if (request.InputStream.Length > 0)
    9:  {
   10: // This will ensure that IIS gets back the entity body after it is 
   11: read by WSFederationAuthenticationModule 
   12: request.InsertEntityBody();
   13:  }
   14:  }

 

caveat:

Calling HttpRequest.InsertEntityBody may cause a Heap corruption in IIS 7.0 or 7.5.

Therefore, you will need to install the following hotfix in order to avoid a Heap issue: Heap corruption occurs when a module calls the InsertEntityBody method in IIS 7.5