HOWTO: Diagnose one cause of 503 Service Unavailable on IIS6


I recently got this question about encountering a 503 Service Unavailable error on 64bit Windows. The author chose to uninstall .NET Framework 1.1 as the resolution, but I think there are some better alternatives…


Question:


I was delighted to run across your treatment of ISAPI Filter ‘service unavailable’ woes; it gave me the courage to try to get to the bottom of my particular problem. You might (or might not) want to post something of the solution I found in my own situation:


On Windows XP x64 happily running IIS6, the installation of the .NET Framework 1.1 kills the IIS6 with the ‘service unavailable’ message returned; uninstalling the .NET Framework returned things to the status quo ante.



I am trying to find the blog you wrote on the 503 error and cannot find it. The google link goes to a page that does not have the information on it. I am trying to get Win XP x64 to work with VS 2003 ASP.net 1.1 and I am getting the 503 error and I cannot find any information on what to do about it. Any help would be appreciated.


Answer:


Actually, I think your situation has a better solution than uninstalling .NET Framework 1.1. If my guess is right, you should be able to run .NET Framework 1.1 on 64 bit IIS6.


If you check your Event Log entries, you will likely find several of the following entry, followed by another entry declaring that the Application Pool is being disabled. This disabling is what causes the 503 Service Unavailable to be returned.

Event Type: Error
Event Source: W3SVC-WP
Event Category: None
Event ID: 2268

Description:
Could not load all ISAPI filters for site/service. Therefore startup aborted.
Data: 0000: c1 00 00 00


The reason why you are getting this event is straight forward:



  • On 64bit Windows, the “bitness” (i.e. 32bit or 64bit) of a process must match the bitness of the DLLs loaded by that process. In other words, a 64bit EXE can only load 64bit DLLs, and 32bit EXE can only load 32bit DLLs.
  • By default, IIS6 on 64bit Windows runs with 64bit W3WP.EXE worker processes
  • .NET Framework 1.1 has ASP.Net implemented through 32bit ISAPI DLLs.

What is happening when you install .NET Framework 1.1 on IIS6 on 64bit Windows is that while IIS6 runs W3WP.EXE as 64bit, you are configuring it to load some 32bit ISAPI DLLs. This does not work and leads to the event log entry. Since the ISAPI DLLs are loaded for every request, this failure immediately happens again and again, thus triggering the “Rapid Fail Protection” health monitoring check of IIS6. This leads to this Application Pool being taken offline and a 503 Service Unavailable response being sent.


One way to fix this issue is to:



  1. Change IIS6 run W3WP.EXE as 32bit

  2. *** IMPORTANT *** Then restart the Application Pool that returns the 503 error since it is stopped. You cannot fix any 503 error without restarting the Application Pool.

Changing IIS6 to run W3WP.EXE as 32bit allows the 32bit ISAPI DLLs installed by .NET Framework 1.1 for ASP.NET to load and run inside of it. This is done by running the following commandline:

CSCRIPT %SYSTEMDRIVE%\Inetpub\AdminScripts\adsutil.vbs SET W3SVC/AppPools/Enable32bitAppOnWin64 1

This command switches IIS6 into running WOW64 (i.e. 32bit compatibility) mode on 64bit Windows on-the-fly so that IIS6 can immediately run 32bit ISAPI DLLs… unless that Application Pool is ALREADY returning 503 errors, in which case you MUST restart the Application Pool to have the bitness switch take effect. It makes sense because a 503 error means the Application Pool is offline and not running, so you must restart it to have setting changes take effect.


You can do this by either:



  • Restarting the Application Pool in question

  • Restarting IIS
    NET STOP W3SVC /y & NET START W3SVC

  • Reboot the server
    SHUTDOWN -r -t 0

Now, I cannot guarantee that this works for you because you may have other applications that must run as 64bit, in which case you have a conflicting need to simultaneously run 32bit and 64bit code in IIS6, which is not allowed.


See KB 895976 for more details.


I am also going to write up a more complete explanation of WOW64 and running 32bit applications on IIS6 in Windows Server 2003 SP1 (same codebase as Windows XP 64bit Edition) in the near future. I have answered this question to many people for the past year now, and I want to answer it once and for all. 🙂


//David

Comments (37)

  1. Adelino says:

    Hi. That script didnt make it work for me. Same error. 🙁

  2. David Wang says:

    Adelino – Is your 503 actually caused by what I describe? Because the resolution I describe is specific to that cause, and it definitely works because it addresses the fundamental issue.

    Of course, with bitness issues, it is possible for many, many variations and no single approach will work. You just have to know what you are configuring.

    If you cannot get it working, you may want to consider moving back to x86 because you definitely need to learn and understand x64 in order to use it correctly, especially during this 32-64bit transition phase…

    //David

  3. Eric-Goz says:

    Hello folks, well I had the same problem and the above mentioned fix did not work me. After digging around you must download and install the Microsoft .NET Framework Version 2.0 Redistributable Package (x64). This also updates IIS 6.0.

    After this update I have no problems running VS 2003 and any .NET projects under XP64.

    The URL for the download is here:

    http://www.microsoft.com/downloads/thankyou.aspx?familyId=b44a0000-acf8-4fa1-affb-40e78d788b00&displayLang=en&oRef=

    I hope this helps, good luck.

  4. David Wang says:

    Eric-Goz – I suspect the reason it did not work for you is because when you changed bitness to 32bit, you had other ISAPIs that depended on (or is configured) to being 64bit. But, that is an expected conflict, as I mentioned in the blog entry.

    When you run the native x64 redistributable, everything runs as 64bit, so everything is happy.

    FYI: .Net Framework redistributable does NOT update IIS6.

    //David

  5. Guillaume Lachance says:

    I wanted to report that this fix worked great for me but <b>NEEDED A REBOOT</b>.

    I’m now running Microsoft Visual Studio.NET 2003 without a glitch (debugging and all) with Framework 1.1.

  6. David Wang says:

    Guillaume – Glad to hear that. However, the fix itself should not require a reboot to take effect:

    – Toggling IIS bitness takes effect immediately

    – To rescue an Application Pool that went into 503 Service Available, just restart the Application Pool from the IIS Manager UI

    So, recovery from this issue should be a matter of changing one configuration property, restarting the afflicted Application Pool(s), and things are good to go.

    //David

  7. Dirk Nielsen says:

    David:

    Your fix works perfectly. You definitely must stop and restart the Application Pool (not recycle) for the change to take effect and lose the 503 Error.

    Dirk

  8. David Wang says:

    Strange. I have answered this question recently on how to diagnose a common cause of &quot;503 Service Unavailable&quot;…

  9. David Wang says:

    One of the cooler but definitely unsung feature of the IIS7 Server Core is PreConditions. So, that’s…

  10. DVJ says:

    Dave:

    Thanks for a good solution. I was running Win2k3 x64 with both .NET 1.1 and 2.0, and had the server automatically download and install MS fixes. Big mistake! The fix for 1.1 broke my 2 code!

    Solution: removed 1.1 and re-installed from distribution 2.

    Shame on Microsoft for issuing "patches" that break the existing software (remember XP SP2?)

    Again, thank alot,

    DVJ

  11. David.Wang says:

    DVJ – glad things worked out for you.

    Personally, I would not beret Microsoft for issuing "patches" that break the existing software because I can tell you that Customer expectation of MS patches are very unrealistic.

    1. On the one hand, they want the patches delivered yesterday.

    2. On the other hand, they want the patch tested against everything in the world to not break anything.

    Tell me how anyone can fulfill those two expectation simultaneously. No, Open Source doesn’t do it, either — they patch fast and make no compatibility guarantees — if you want the fix, you have to migrate forward. No AppCompat story at all.

    Basically, since .NET 1.1 was released prior to x64, you had to configure several "hacks" to make it work — a new release of the .NET 1.1 Redist, configure IIS6 to run in WOW64 mode, and make sure no conflicting ScriptMaps with ASP.Net 2.0 — and if you happen to get a bitness mismatch (real easy on x64 with various 32bit compatibility hacks), you have a real problem.

    In other words, consider .NET 1.1 on x64 an edge case, and since you are involving WOW64 and fairly delicate configuration to begin with, I would not be surprised that your "problem" was a simple misconfiguration assumption and easily fixable.

    I really do not fault Microsoft because we really do try our best to have things work and am glad to fix issues. But for some reason, people expect perfection and the impossible from Microsoft, but we are human, too…

    //David

  12. The following links to .NET resources have been collated over time with the assistance of colleagues.&amp;nbsp;…

  13. Magellan says:

    Your solution resolved my inability to run VS2005 on x64 too.    Everything was fine until I installed Pinnacle that required .Net1.1.    Thanks.

  14. Brett Anthony says:

    David a big thanks to you, your cscript line worked perfectly. All I had to do was restart the default apppool. Your a legend, this saved me quite a bit of "hair tearing".

    Many Regards

  15. David.Wang says:

    Another 10K entry… yes, WOW64 and 503 are truly confusing.

    //David

  16. Hunter says:

    Hey i’m running Windows Vista with the same problem.(32 bit) Is there a way to fix this?

  17. David.Wang says:

    Hunter – without more information, your issue cannot be diagnosed nor fixed.

    I suggest that you ask your questions on IIS-related newsgroups to focus on determining what is actually wrong. You can access it via web-based reader:

     http://www.microsoft.com/windowsserver2003/community/centers/iis/default.mspx

    Or NNTP News Reader on msnews.microsoft.com server for newsgroup:

     microsoft.public.inetserver.iis.

    //David

  18. Dan says:

    Thanks! This fixed the error and no reboot required!

  19. Ravi Prakash says:

    My case of 503 problem is little bit different. On production we have depolyed applications developed in Framework 1.1 and its working fine since more than a year When we changed a piece of code in application and re released the application, after two days of successfull run it started getting 503 error. In HTTPError log there are entries mentioning connection pool abandoned. When application is reverted back with the older(which was running fine) version it again started working fine and there is no 503. This problem is coming only when we deployed new version of our application that too new versions runs perfectly for few days. We have done the rigorus code analysis also to ensure there is no infinite loop or memory leak exists.

    I accept different reasons of this 503 error but it must behave uniformely. Why it is coming only when new version is deployed.

    In both the version nomrally the simultaneous users accessing sytem are crossed 80,000 some times

    Any suggestions will be welcomed….

  20. Ravi Prakash says:

    My case of 503 problem is little bit different. On production we have depolyed applications developed in Framework 1.1 and its working fine since more than a year When we changed a piece of code in application and re released the application, after two days of successfull run it started getting 503 error. In HTTPError log there are entries mentioning connection pool abandoned. When application is reverted back with the older(which was running fine) version it again started working fine and there is no 503. This problem is coming only when we deployed new version of our application that too new versions runs perfectly for few days. We have done the rigorus code analysis also to ensure there is no infinite loop or memory leak exists.

    I accept different reasons of this 503 error but it must behave uniformely. Why it is coming only when new version is deployed.

    In both the version nomrally the simultaneous users accessing sytem are crossed 80,000 some times

    Any suggestions will be welcomed….

  21. Ravi Prakash says:

    My case of 503 problem is little bit different. On production we have depolyed applications developed in Framework 1.1 and its working fine since more than a year When we changed a piece of code in application and re released the application, after two days of successfull run it started getting 503 error. In HTTPError log there are entries mentioning connection pool abandoned. When application is reverted back with the older(which was running fine) version it again started working fine and there is no 503. This problem is coming only when we deployed new version of our application that too new versions runs perfectly for few days. We have done the rigorus code analysis also to ensure there is no infinite loop or memory leak exists.

    I accept different reasons of this 503 error but it must behave uniformely. Why it is coming only when new version is deployed.

    In both the version nomrally the simultaneous users accessing sytem are crossed 80,000 some times

    Any suggestions will be welcomed….

  22. Anil Kumar Sharma says:

    Hi,

         I also received 503 problem and log file (C:WINDOWSsystem32LogFilesHTTPERRhttperr1.log) is as follows:

    #Software: Microsoft HTTP API 1.0

    #Version: 1.0

    #Date: 2007-06-05 04:33:50

    #Fields: date time c-ip c-port s-ip s-port cs-version cs-method cs-uri sc-status s-siteid s-reason s-queuename

    2007-06-05 04:33:50 13.205.105.18 1042 13.205.105.18 80 HTTP/1.1 GET / 503 1 AppOffline DefaultAppPool

    2007-06-05 04:33:57 13.205.105.18 1043 13.205.105.18 80 HTTP/1.1 GET /certsrv/ 503 1 AppOffline DefaultAppPool

    2007-06-05 04:34:18 127.0.0.1 1044 127.0.0.1 80 HTTP/1.1 GET /certsrv/default.asp 503 1 AppOffline DefaultAppPool

    2007-06-05 04:34:46 127.0.0.1 1045 127.0.0.1 80 HTTP/1.1 GET /certsrv/default.asp 503 1 AppOffline DefaultAppPool

    2007-06-05 04:34:46 13.205.105.18 1046 13.205.105.18 80 HTTP/1.1 GET / 503 1 AppOffline DefaultAppPool

    2007-06-05 04:34:50 13.205.105.18 1047 13.205.105.18 80 HTTP/1.1 GET /certsrv 503 1 AppOffline DefaultAppPool

    2007-06-05 04:35:32 13.205.105.18 1049 13.205.105.18 80 HTTP/1.1 GET /certsrv 503 1 AppOffline DefaultAppPool

    2007-06-05 04:36:31 127.0.0.1 1051 127.0.0.1 80 HTTP/1.1 GET / 503 1 AppOffline DefaultAppPool

    2007-06-05 04:37:21 127.0.0.1 1052 127.0.0.1 2077 HTTP/1.1 GET / – 2 Connection_Dropped MSSharePointAppPool

    2007-06-05 04:37:52 127.0.0.1 1053 127.0.0.1 80 HTTP/1.1 GET / 503 1 AppOffline DefaultAppPool

    2007-06-05 04:38:52 127.0.0.1 1054 127.0.0.1 2077 HTTP/1.1 GET / – 2 Connection_Dropped MSSharePointAppPool

    2007-06-05 04:45:44 127.0.0.1 1058 127.0.0.1 80 HTTP/1.1 GET /certsrv/ 503 1 AppOffline DefaultAppPool

    2007-06-05 04:47:58 127.0.0.1 1060 127.0.0.1 80 HTTP/1.1 GET /certsrv/ 503 1 AppOffline DefaultAppPool

    2007-06-05 04:47:59 127.0.0.1 1061 127.0.0.1 80 HTTP/1.1 GET /certsrv/ 503 1 AppOffline DefaultAppPool

    2007-06-05 04:48:00 127.0.0.1 1062 127.0.0.1 80 HTTP/1.1 GET /certsrv/ 503 1 AppOffline DefaultAppPool

    2007-06-05 04:49:57 127.0.0.1 1064 127.0.0.1 80 HTTP/1.1 GET /certsrv/ 503 1 AppOffline DefaultAppPool

    #Software: Microsoft HTTP API 1.0

    I installed the Windows server 2003 (with sp1) in standard installation. I installed IIS server and then certificate services. Restarted the system and tries to connect http:\localhostcertsrv, but in vein.

     Finally a quick and clever notice is caught.  For simplicity i did the dump in harddisk to increase speed of installation and as a backup, so that i did not require to insert Windows server cd everytime.

    Also, I did not require R2 so i didnt installed it. But when installing IIS services from control panel, i am asked to provide some file , i browsed to cd location but some file is not there, so alternatively these file found in R2 cd dump and picked up from there (!!!!NOTE NOTE NOTE ….root cause) .

    So when i tries to run http:\localhostcertsrv, i got service unavailable error. Everything looks fine but nothing +ve happens.

    After so many tries , i uninstalled the IIS & Certificate. And when installed again , i insert the CD in drive and provide file from there (Voila, now the file it is not able to locate from hardisk(so provided from R2 Dump) is in CD !!!).

    AND EVERYTHING WORKS FINE THEN….. 🙂

    Anil

  23. Tom G says:

    At first, I tried everything that was mentioned above, and it didn’t work for me either.  I was trying to install CF as a Multiserver Configuration.  

    What worked was when I installed CF as a Server Configuration.

    Thanks David for your blog!

  24. simon says:

    I was also having a problem with running CF on the 64 bit version of windows 2003.  The script by itself did not work (although it does have to be run)

    What got it going for me was the link that Eric-Goz suggested. I downloaded it, ran it, selected rebuild, rebooted and now all is well.

    Thanks guys

    P.S. Here’s the link again

    http://www.microsoft.com/downloads/thankyou.aspx?familyId=b44a0000-acf8-4fa1-affb-40e78d788b00&displayLang=en&oRef=

  25. Aeron says:

    The version of RPC over HTTP proxy that ships with x64 version of server 2003 has 64 bit DLL’s, and thus breaks IIS running in 32 bit mode. Would it be possible to copy the 32 bit DLL’s for RPC proxy from a 32 bit server 2003 install and overwirte the 64 bit ones, or would that be a bad idea?

    Thanks in advance…

  26. David.Wang says:

    Aeron – mixing binaries between different OSes is unsupported. Overwriting binaries (especially mixing bitness) is also unsupported.

    Unless Microsoft PSS says it is supported (in which case you should be contacting them, not me), I will not be able to help you figure out if it is possible or not.

    Personally, I would focus my attention on getting my software to support pure 64bit, instead of trying to hack unsupported 32bit workarounds together.

    //David

  27. pankaj bhandari says:

    Hi dear , i’m facing the same problem, one of our CRM server giving slow response or almost down for two to three minutes and got up automaticaly . as i’ve observed the same error logs generated on HTTPERR log file which shows there are same kind of messages in which one of message like "Connection_Dropped DefaultAppPool " is more responsible for this at the same time have observed more then one w3wp.exe  processes are running with high memory usage which could be the main cause of down . So pls give me vital clues to solve this problem pls drop yur suggestion on this

    pantig81@gmail.com

  28. David.Wang says:

    pankaj – There are MANY things that can cause a 503 Service Unavailable, so you have to troubleshoot with procedures described in my "AppPool Crash Diagnosis" or "IIS6 – Troubleshooting Guide" links from this blog to make further progress.

    Connection_Dropped indicates that either a browser closed the connection (like closing the web page), or a server-side ISAPI closed the connection (or crashed). Such crashes will be logged in the Event Log, and if enough of them happen, will cause the Application Pool to go offline and cause 503 Service Unavailable to be returned for that Application Pool. This will be logged in HTTPERR log file and require manual intervention to restore.

    High memory do not necessarily mean 503 is returned.

    If you are using ASP.Net, then it can also return a 503 Service Unavailable if it is out of worker threads to handle requests. However, such shortage is often intermittent because as prior threads finish their work, ASP.Net gets its threads back and stops sending 503s.

    Since you say that you get 503s/slow response for 2-3 minutes and then it recovers automatically without you doing anything — IF the 503s are for ASP.Net pages, then I would suspect the thread shortage issue.

    Now, the ASP.Net 503 thread shortage issue can be fixed in one of three ways:

    1. Reduce the incoming traffic rate

    2. Increase the number of Processors on the server

    3. Fix the ASP.Net code to serialize less

    Adding threads is possible but not the right option here because it will drive your CPU and memory utilization even higher. You already said that memory utilization is high, so you cannot add more threads because it will cause other issues from high memory utilization.

    //David

  29. c8to says:

    I think this fix should not be recommended. When you say it is a better solution than uninstalling the incompatible .NET 1.1, i don’t think this is the case.

    While the fix may certainly work to get IIS running in 32bit mode on a 64 bit install, and allow the 32bit .NET 1.1 to work, imagine the future incompatibilities when someone comes along and finds this completely non standard setup. It would be a maintenance nightmare…

    you do mention this briefly in the blog post, but i think you should emphasize to only do this if you really need to run the 32bit mode.

  30. David.Wang says:

    c8to – I agree with you that running purely 64bit is preferred and cleaner approach.

    However, running 32bit .Net 1.1 on 64bit Windows is also supported by Microsoft because customers request/require it. It is a case-by-cases basis.

    In this blog entry, I am simply describing what was going on with a misconfiguration. I have insufficient information to determine nor advise any case-by-case recommendation.

    //David

  31. shmuel brody says:

    shmuelbrody@yahoo.com

    my computer is a hp pavilion ze4900

    last year i did something to my internet explorer settings and now internet explorer and any other browser that i use except firefox will only give me a 503 error message

    i am not a computer tech and cant read the posts that you put out

    they make no sense to me.

    how ever i must get internet explorer working

    i reset all the settings in tools advanced

    i reinstalled the windows explorer from the disk

    and still i get 503

    i dont know what to do

    could you please help me and send the info tomy email address

    as i will never find you site again

    im not good at this at all

    thanks

    shmuel

  32. CMZK says:

    EASY SOLUTION:

    IIS7 -> APPLICATION POOLS -> ASP.NET 1,1 -> SET APPLICATION POOL DEFAULTS -> ENABLE 32-BIT APPLICATIONS -> TRUE.

    THIS WILL WORK!!!!

  33. David.Wang says:

    CMZK – yes, that is one possible solution, but only for a certain problem, and not all 503s. You need to specify to which problem your solution is for…

    //David

  34. Sam says:

    I followed this procedure :

    IIS7 -> APPLICATION POOLS -> ASP.NET 1,1 -> SET APPLICATION POOL DEFAULTS -> ENABLE 32-BIT APPLICATIONS -> TRUE

    Even after doing this change also the application is getting stopped after some time.

  35. John Timms says:

    I found another method that works for IIS 7.0 and helped me at

    http://blog.evonet.com.au/post/2008/08/22/503-service-unavailable-errors-when-IIS-running-in-32-bit-mode-on-64-bit-Windows-Server-2008.aspx

    The problem was the access ODBC driver.

  36. nishant says:

    I am also facing this 503 service unavailable error. WHen I remove IIRF filter then error is  gone. coul dthis error be related to IIRF filter