Application Request Routing – Part 2(Reverse Proxy and Troubleshooting ARR, URLRewrite Issues)

In the last blog we had discussed how we can leverage ARR as a load balancer. Over here in this blog we will see how we can use ARR as a reverse proxy when the content server is not exposed to the outside world. We will also discuss how we can go on about troubleshooting errors and issues like 502.3 which is the most common error while using ARR. You can use even server farms to configure as reverse proxy by adding a single server to the server farm.

Before we go on about how we can configure ARR as reverse proxy Lets check some of the options available in ARR Reverse proxy.

Go to inetmgr UI and click on the server name and you will find the option ‘Application Request Routing Cache’. Below is how the UI looks.


Below are the options and its intended purpose.

1) Cache Configuration:

This option lets you configure proxy cache settings for your ARR server.


Cache clean-up interval: By default it is 5 minutes. This setting specifies the time in interval in minutes, at which the cache will check for files or invalidate the cache files and delete it if requires.

Disk usage high threshold: This setting specifies the maximum percentage of configured disk space for cache content to be used before the cached files are deleted to make up more space.

Disk usage low threshold: This setting specifies percentage of configured disk space that will be reached once files are deleted.

Byte range segment size: This is similar to chunking. This setting specifies the maximum size of the chunks that be created by segmenting a single request. Default is 256kb.

Compression: This is used to compress responses into a format that the servers are capable of handling such as gzip.

2) Cache control rules:

This section lets you configure rules to manage the cache control behavior.



You can create rules to be executed when either cache control directive is not present in the headers of the response sent by server or the requests or you can have the rule execute always independent of the cache control header by selecting the item in the drop down

Do not cache: you can select this if you don’t want specific contents to be cached based on the rule.

Cache: select this option to cache based on the condition.

Host name: this specifies for which host name the content should be either cached or not.

URL: this is a very good option where you can specify a frequently accessed content which can be cached.

For eg: you can choose to cache images i.e., */images/*.jpg all the image contents which is definitely a good rule to improve performance.

3) Add Drive and Add secondary drive:

Using this setting you can change your disk cache location to a required custom drive may be D:\ and also limit the amount of space you can use.


4) Server Proxy Settings:



Reverse rewrite host in response headers: This option might not be of much value over here but it’s a very important setting while having reverse proxy. Imagine the internet exposed URL is and you have the backend servers’ contoso1 and contoso2. Consider an example where you have a redirect status set and in the response location tag is set to http://contoso1/redirectedpage.aspx in web server in respect to the request forwarded from ARR server and this has to be notified to the end client. The client doesn’t know who contoso1 is. So before sending the complete response to the client, the ARR server rewrites the host name in the location tag as

Setting up ARR as a reverse proxy


Imagine you have a site with external URL and the actual content is hosted on IIS server which is not exposed to the internet world and internal/ intranet URL is http://contososerver/.*

So over here we will forward the requests to the ARR server and inturn internally route it to the backend server. Below is how the configuration will look like.

At the server level go to Application Request Routing Cache->Server proxy settings




When you specify the server you want to reroute the requests ARR will automatically create inbound rules for you as below in the URLRewrite module at server level as below.



<rule name=”ARR_server_proxy” enabled=”true” patternSyntax=”Wildcard” stopProcessing=”true”>

<match url=”*” />

<action type=”Rewrite” url=”http://contososerver/{R:0}” />




You can modify the rules on the go based on the requirement. Also you can add multiple inbound rules to reverse proxy the requests to different backend servers based on different conditions such as Hostnames.

Note: It is not mandatory to mention the reverse proxy in the server proxy settings. If you want to use ARR as a reverse proxy you just need to make sure that in server proxy settings ‘enable proxy’ is checked and then you can create your own custom url rewrite inbound rule either at the server level or at site level in IIS

Troubleshooting errors and issues with ARR

The most common error you run into when using ARR is the 502.3 error.

The best tools that you can use to troubleshoot ARR and 502.3 errors are below.

1) FREB tracing(Failed request tracing)

2) Fiddler tracing (you can install fiddler from

3) Winhttp tracing and netmon traces

FREB tracing:

This is one of the beautiful tool inbuilt in IIS which will get you more info on why you are getting 502.3 errors. You can configure failed request tracing at the default web site which is running on port 80 at the site level as below.

Go to Failed request tracing at site level->Edit site tracing->Enable


Again Failed request tracing at site level->Add




Try and reproduce the issue. When you encounter a 502.3 error next time you will have the freb xml files in C:\inetpub\logs\FailedReqLogFiles\W3SVC1. Open the xml file in internet explorer. If you are unable to open it in IE, make sure you go to and check internet options->security->internet->scripting is enabled.

Below is one of the scenarios when you get a 502.3 error and what you see in FREB

502.3 “The connection with the server was terminated abnormally”

– This happens if a request is in progress to a content server and the application pool serving that request terminates suddenly (for example due to a web application on the content server throwing an unhandled exception on a non-request thread).

You can investigate this further to check if there were any issues on the content server and investigate why the connection was terminated. You can even collect netmon traces.

Fiddler tracing

If you want to see how the requests are routed to the backend server from ARR and to see what the exact headers are forwarded to the content server and if you are getting any errors in HTTP status codes then Fiddler tracing would be a good bet. You can attach Fiddler to ARR server and get the requests going out of ARR and the responses coming to the ARR server from the backend server.

To attach fiddler to the ARR server to record requests follow the below steps.

1) Install fiddler on the ARR server.

2) Fiddler runs on port 8888, so you need to attach ARR to route the requests to backend server through port 8888 so that fiddler can record the requests and responses. To do that go Application Request Routing Cache at the server level ->Server Proxy settings and add proxy server value as localhost:8888 as below.



Consider you are getting a 404 error from the backend server to ARR server, this is how the fiddler trace will look like. You can see the successful and failure requests and responses obtained from the backend server as below.


WinHttp tracing:

Once the ARR server routes the request and gets the response back from the backend server, ARR then repackages the response to send it back to the client. For this purpose of repackaging ARR uses WinHTTP interface. If you don’t see much info in the FREB traces or fiddler traces then there might be a possibility that request is failing at WinHTTP level. To find out exactly if it is failing at WinHTTP level and where exactly it is failing, you can enable WinHTTP traces. Collecting WinHTTP traces and the way to run the trace differs on the platform and the version of the operating system we are using on the ARR server. You can refer Richard Marr’s blog where he gives a very good description on how to collect these traces. Below are the steps how we can collect winhttp traces from Richard Marr’s blog.

Windows 20081. Start the Tracing. From a command prompt run the following command:

netsh winhttp set tracing trace-file-prefix=”C:\TEMP\WinHttpLog” level=verbose format=hex state=enabled max-trace-file-size=1048576000

2. Recycle the IIS Application Pool.

3. Reproduce the issue.

4. Stop the Tracing. From a command prompt run the following command:

netsh winhttp set tracing state=disabled

5. Review the trace with Notepad or any Text editor.

NOTE: The Identity of the IIS application pool will require write access to the  log location  c:\Temp in this example:

This type of tracing is process bitness specific, so if you are looking at a 32 bit process running from 64 bit OS, you need to use: c:\windows\syswow64\cmd.exe, rather than using the regular 64 bit cmd.exe (start a run a cmd.exe)

Windows 2008 R2

Method 1

This method will output the Winhttp API calls , but not raw data for network communication. From a command prompt run the following command:

1. Start the tracing

netsh winhttp set tracing trace-file-prefix=”C:\Temp\Test3″ level=verbose format=hex

netsh winhttp set tracing output=file max-trace-file-size=512000 state=enabled

2. Recycle the IIS Application Pool.

3. Reproduce the issue.

4. Stop the Tracing. From a command prompt run the following command:

netsh winhttp set tracing state=disabled

5. Review the trace with Notepad or any Text editor.

NOTE: The Identity of the IIS application pool will require write access to the log location c:\Temp in this example:

This type of tracing is process bitness specific, so if you are looking at a 32 bit process running from 64 bit OS, you need to use: c:\windows\syswow64\cmd.exe, rather than using the regular 64 bit cmd.exe (start a run a cmd.exe)

Method 2

To get the raw data communication at network layer and the Winhttp  Api calls.

1. Start the tracing: From a command prompt run the following command:

netsh trace start scenario=InternetClient capture=yes report=yes

Note the etl file location for example:

  Trace File:         C:\Users\<your user name>\AppData\Local\Temp\NetTraces\NetTrace.etl

2.Recycle the IIS Application Pool.

3. Reproduce the issue.

4. Stop the tracing: From a command prompt run the following command:

netsh trace stop

5. Read the Trace by opening it in Netmon 3.4.

Method 3

The ETW format for winhttp API is available on windows 2008 R2 and win7 via the Event Viewer

1.  Open event viewer. Go to “View” menu –> make sure “Show Analytic and debug logs” is checked.

2. Open “Applications and Services logs” — > Open “Microsoft” — > Open “Windows –> Winhttp –> Diagnostic.


3. Highlight “Diagnostic” under Winhttp tree and right click mouse, then click “enable log”.

4.  Reproduce the issue then you can review the logs.

Below is one of the sample winhttp output which shows an error 502.3 which is happening because of timeout at ARR level. The ARR server is waiting for the data from the backend server and timing out as below.

15:57:26.662 ::Completing WinHttpReadData() with error ERROR_WINHTTP_TIMEOUT (12002); Request


Handle = 1C8457F0, Context = 1C5F0E70


15:57:26.662 ::sys-recver processing WebReceiveHttpResponseEntity completion


(error-cdoe = ? (0x5b4), #bytes = 0, overlapped = 1B775260)


15:57:26.662 ::sys-recver failed to read data; error = ? (1460)


15:57:26.662 ::ERROR_WINHTTP_FROM_WIN32 mapped (?) 1460 to (ERROR_WINHTTP_TIMEOUT) 12002


15:57:26.662 ::sys-req completing a read-data call(error = ERROR_WINHTTP_TIMEOUT(12002),cbRead =0)


15:57:26.662 ::usr-req 1C9B4F80 received OnReadData() callback; error = ERROR_WINHTTP_TIMEOUT 


(12002), ulNumberOfBytesTransferred = 0, dwptrContext = 1C380160

Netmon traces will further help you to get the exact timeout. As a workaround for this issue you can increase the timeout value for ARR at the server proxy settings or investigate why it is taking a long time in the backend server.

Hope this one helps Smile


Comments (23)

  1. Lind says:

    Thank you for this blog post! I have a problem with ARR that you might be able to help me with! I'm trying to redirect requests through my IIS with ARR to a tomcat service by using URL Rewrite. The pattern it should match is jft.generatepdf.localhost/* and I've tested it against an expected URL String (it matches). The problem is that I'm getting page not found (404) when trying to access it. It worked before when I tried with the pattern .*, but that I can't do as I have 2 more services to use on the same port (port 80). Do you have any ideas?

  2. @Lind: if I am correct jft.generatepdf.localhost is the hostname. We cant check for Hostname in the pattern section. your pattern section should be either *(if wildcard matching) or .* (if you are using regular expressions) then under the conditions section you can check the hostname in {HTTP_HOST} matches jft.generatepdf.localhost. The explanation what we need to check in each section of a inbound rule is in below link…/url-rewrite-part-2-inbound-rules.aspx

    Please let me know in case oa ny follow up questions.

  3. Lind says:

    @Chiranth Ramaswamy Thank you for your reply! I'm in desperate need of fixing this so really appreciate it:) I assume that I have to add jft.generatepdf.localhost to my hostfile while I'm using my local machine for testing right?

    So adding:   jft.generatepdf.localhost

    should be required`?

    I'm about to test this and I'll get back to you with the results!

  4. Lind says:

    @Chiranth Ramaswamy, I tried this but with no luck. This is my setup for the URL Rewrite:

    Name: Tomcat

    Matches the pattern using Regular Expression

    Pattern: *


     Input: {HTTP_HOST}

     Type: Matches the pattern

     Pattern: jft.generatepdf.localhost

    Action type: Rewrite

    Rewrite URL: http://localhost:8080/{R:0}

    In the host file I added: jft.generatepdf.localhost

    I still get page not found when trying to access with this string:


    Any ideas?

  5. @Lind: Please provide me the below information.

    URL or hostname mapped to IIS: xxxxxx(Binding for the site in IIS)

    Direct URL when you hit the TOMCAT server:xxxxx

  6. Lind says:

    @Chiranth Ramaswamy

    Binding to IIS: I have no binding set up (that I know of) to the tomcat service (let's call it generatepdf). Is this the same type of binding you can set up for a "native" iis service?

    Direct URL When I hit the Tomcat Server: Sorry for my lack of knowledge here but do you mean an example I can use to access the service through tomcat directly without the proxy?

    If that's the case here it is:


  7. @Lind: Create a site in IIS and have a binding for the iis site mapped to IP,PORT and the hostname. Lets keep the hostname as jft.generatepdf.localhost. After doing this follow the below steps.

    1) In ARR server proxy settings at server level check enable proxy check box

    2) Disable the other rules that you might have created.

    3) In hosts file add the entry to map jft.generatepdf.localhost to the IP you are using in the site. probably bind it to a specific ip rather than loopback address (

    4) At the site level go to URL rewrite->Add inbound rule->Blank rule

    For below steps I am assuming we using 'regular expressions'

    5) Pattern section should be .*

    6) Under conditions->add condition->Condition Input will be {HTTP_HOST}and the pattern section will be jft.generatepdf.localhost

    7) Action type will be Rewrite

    8) Action URL will be http://localhost:8080/{R:0}

    9) Now try to access jft.generatepdf.localhost/…/GeneratePDFServlet locally and see if it works

    let me know in case of any queries

  8. Lind says:

    @Chiranth Ramaswamy I Thank you a million times over! you found the missing piece!

    However, I have two more services on the IIS (native IIS services) that I need to be able to access through the same port (port 80). When I added bindings for them to port 80, something crashed with the setup. It still seemed to access the tomcat service through IIS as intended but the db access that I perform did no longer work.

    I don't expect you to know how to solve that problem as it is most likley highly individual but maybe you know if this is supposed to be possible? (meaning to have 2 native IIS Services on the same port as this proxy using bindings?)

  9. @Lind: you can run multiple IIS services on the same port provided you have a unique combination of IP:PORT:Hostname. so if you want the IP and port to be the same then you need to add different hostnames for different services

  10. Lind says:

    @Chiranth Ramaswamy, Ok then I know it is possible, I just need to find out what might be causing my issues. I'm using unique hostnames so the problem is probably elsewhere then.

    Thanks again for all your help! I've been trying to get this working for over a week without success so I am ever so grateful for you taking time to help me!

  11. null zero says:

    Hi Chiranth,

    I want to do more or less the same thing as Lind but slightly different.

    I want to use reverse proxy for and still serve from the same server (the one configured with ARR).

    When configuring the inbound rule on site level, I can't choose action type "Route to Server Farm".

    On the other hand if I have the default rule active (Pattern * and Route to Server Farm), it tries to redirect requests for to the server farm.

    How should I set this up? Is this possible?

    Thank you in advance!

  12. in the default rule you need to add one condition in the conditions section of the rule. So the condition will be {HTTP_HOST} matches

    When you have this condition the reverse proxy will only kick if the incoming request has the hostname or else I will serve the content from your default

  13. Geetha says:

    We have a website running on port1 and another website running on port2 on the same IIS server. We want to able to rewrite/redirect request on port1 to the app running on port 2 on the same server.

    Say for e.g, client entered url on port1 is (note mypage 1 does not exist in the app running on port1) :

    and we want the request to be serviced by the app running on port 2, i.e.:

    We want this to be transparent to the user. So as far as the user is concerned, the url in his browser should always point to port1. He should not know about the existence of the app on port2.

    I installed ARR on our IIS server to be able to use Reverse Proxy feature using the outbound rule. But I am having problems with the inbound rule itself.

    I tried with a Url Rewrite Inbound rule with Redirect Action. This works fine. However since this changes the url on the client's browser we do not want to do this. So I modified with same rule to do a Rewrite Action.  In the Rewrite Action I had specified:{R:0}

    However this does not work as I had expected. It still looks for the and since mypage1 does not exist on the app running on port1, it complains that page is not found. Please let me know, what I am doing wrong, or if this is not the correct way to do this.

  14. @Geetha: The rule that you have sounds correct to me. Again briefly lets say that is the site 1 running on port 80 and is the one running on port 9445. At the site level of the one running on port 80 this is how you rewrite inbound rule would look like.

    <rule name="">

                       <match url=".*" />


                           <add input="{HTTP_HOST}" pattern="" />


                       <action type="Rewrite" url="{R:0}" />


    Also the second thing to consider which I suspect is causing the issue and the rule is not working is below.

    When you want to rewrite request to a different site within the server or to a different server you need to enable reverse proxy in ARR. To do this, you need to Application Request Routing Cache in IIS manager at the server level->Server Proxy settings->Check Enable Proxy option.

    Also as an added point I see that you want to rewrite the requests for a specific app within the site. Lets assume app1 is the app running under the site2 and if request comes as…/somepage, you need to rewrite the rule as below with a minor change.

    <rule name="">

                       <match url="app1/.*" />


                           <add input="{HTTP_HOST}" pattern="" />


                       <action type="Rewrite" url="{R:0}" />


    Hope this helps.

    Let me know in case if you have any further queries

    1. Louis says:

      Hello sir
      I’m new to reverse proxy. We need to setup IIS reverse proxy to WAP server thru 443.
      Setup now: Matches the pattern, using: regular expressions. Pattern: *
      Conditions Match all (HTTPS) type matches the pattern Pattern off
      Server variables; action type redirect Redirect url: https:// ip address (WAP server) R:0
      Redirect type: found 302. Thats the inbound rules for ssl

      Do I need to have another inbound rule?
      Do we need any rules at the site level?

      1. @Louis: the existing rule redirects the requests coming over HTTP to the proxy server to HTTPS and looks like we are redirecting it over https and to a specific ip address. If we want to have a reverse to hide the WAP server then we need to have a inbound rule with action type as rewrite making the ARR server act as a bridge between server and the actual client

  15. RaviA4 says:

    Hi Chiranth, Thank you for this post, it’s very informative.

    I’m working on setting up reverse proxy to render content from an internet hosted Web Sphere on to an internet hosted IIS application.

    URL rewrite 2.0 was installed already on IIS before I started looking into reverse proxy implementation. So I extracted the package and installed ARR 2 and Web Farm Management V1 only.

    Enabled the proxy and put in the URL Rewrite rule:

    It always throws http 404 error. On key thing I noticed is I created the URL Rewrite rule at the website level and not at server level. I have seen the part you highlighted in yellow in the post and I’m going to try to move the rule up to server level. Beside that, anything else that may cause the 404 ?

    I do have a follow up question on moving the rule up to server level as well: My client uses a shared Web hosting environment and on this server there are total 70 websites hosted. So I wanted to make sure that both ARR and the server level URL rewrite rule are triggered only for my website and not for other 69 websites. I initially put the the URL Rewrite rule at my website level for this purpose, haven’t figured out how ARR impacts all other 69 sites. Any suggestions on this?

    Thank you for your help!

    1. @Ravi4: You can keep the rules at site level. i have updated the info on the blog as well. You need to make sure that the reverse proxy is enabled in server proxy settings at the server level. Coming to the 404 error. 404 can be thrown because of 2 reasons. 1) IIS site throws this if the resource cannot be found. If this is the case this will be logged in IIS logs. You can collect Failed request traces to see who and why 404 is being thrown. 2) HTTP.sys also can throw 404 error even if before it reaches IIS. This might happen if http.sys is not able to find the site based on the hostname. This will be recorded in httperr logs in C:\Windows\System32\LogFiles\HTTPERR

      1. @Ravi: if you want to keep the rule at server level and want that to work for only one site, you can add a condition to check the host name with {HTTP_HOST} parameter matching the hostname of the site on IIS for which you want that to execute

        1. RaviA4 says:

          Hi Chiranth,

          Thanks. Firstly, a correction to the terminology used in the question. It’s not at site level, I was trying to use it at Application level. Our environment has 500+ Applications configured under Default Web site on one server.Secondly, Yes application level rules worked. I was using a wrong pattern to test it.

          My application is TestRP and I was trying to use the pattern “/toWAS/TestServlet” which is wrong. It worked at Default web site level.

          Since I’m at TestRP application level already, the module inspects anything after /TestRP/ but my Test URL doesn’t have it. I changed it to “/TestRP/toWAS/TestServlet” and it worked perfect.

          Now, I have been dealing with intermittent http 400 errors. Following the debugging steps outlined in your blog and other IIS blogs, will post an update.

          HTTP/1.1 400 Bad Request
          Content-Type: text/html;charset=ISO-8859-1
          Content-Language: en-US
          Server: Microsoft-IIS/7.5
          X-Powered-By: ASP.NET
          Date: Tue, 22 Nov 2016 18:35:44 GMT
          Content-Length: 28

          Error 400: Invalid Request

  16. Eternal says:

    Hi Chiranth, thanks for few very useful articles, but I need some help.
    I have two servers, serverA and serverB. ServerA running multiple applications on different ports. ServerB on port 8088 running a web service which path look like http://servername:8088/servicename/methodname.svc. I want to create reverse proxy on serverA, that client send request on serverA and Server rewrite url on server level to serverB. Client doesn’t know that serverB exist.
    Well, on ServerA in Server Proxy Settings I was enable Proxy option, and in URL Rewrite (on server level) I was make a inbound rule to rewrite url if pass the regex: “servicename/methodname.svc”. I test it there and regex pass the full path to service.
    After that, I was traying to call the service, but I had response as Page not found.
    After all, I made the same service on ServerA and call service again, ServerA make call to ServerB and ServiceB give response to serverA, so everything works as well. But I want to remove service from ServerA. Can you help me?

    1. @Eternal: if possible could you let me know all the entries in the rule. For your requirement, the pattern seems to be correct. ALso you can have it as servicename/methodname.svc.* and in the action url for rewrite the action url would be http://servername:8088/{R:0}. Also you can add an extra condition such as {HTTP_HOST} to match some hostname if the service is coming on a particular hostname

      1. Eternal says:

        All the entries are this two, rule to rewrite and rewrite URL as you can see on next link:
        Did I missing something?

Skip to main content