Thoughts on Application Pool Recycling and Application Availability


Question:


I am running IIS 6.0. We are currently trying to incorporate our application pools to recycle every 2 hours. The problem is everytime the the pool is recycled and the process is killed everyone connected to that webpage loses all connectivity and they have to log back into their applications that they were running. I have read about overlapping but i do not know where to go and check to see if it is still set to the default (overlapping) or if it is set to non-overlapping.


Does anyone have any suggestions on how to keep our recycle process going without our clients getting kicked out of their applications


thank you for any help


Answer:


When you recycle an application pool, HTTP.SYS holds onto the client connection in kernel mode while the user mode worker process recycles. After the process recycle, HTTP.SYS transparently routes the new requests to the new worker process. Thus, the client never “loses all connectivity” to the server – the TCP connection is never lost – and never notices the process recycle. Also, your issue is not related to overlapped recycling (enabled by default) because that does not involve connectivity nor state.


I believe your problem is with the applications running in your application pool that store state within the process, such as whether a user is logged in or not. Everytime the process recycles, that state is automatically lost… which is by-design since that is what a process recycle accomplishes. As a result, your users “lose all connectivity” and “have to log back into their applications” to re-establish that lost state. The only way to fix this is for your applications to store its state outside of the IIS6 worker process such that it is friendly to being recycled.


The following blog entry talks more about what is going on:



Now, lots of people have wondered why IIS does not provide facilities to somehow “save” the in-process session state prior to recycling and then “restore” it after the recycle, so that all applications run by IIS are automatically friendly to being recycled. The astute reader should realize that this request has several problems, most prominently:



  1. One reason to recycle a process is to proactively get rid of potentially bad/stale state that leads to crashes, hangs, or memory leaks. If IIS simply saves and restores all state across the process recycle, both good and bad, then it defeats the purpose of the recycle.
  2. Assuming that IIS has some magical way to determine the bad state from the good state and only toss away the bad state, what happens to the user application that depend on both the good and what WAS the bad state? How can IIS determine a “good” value for that bad state to prevent introducing logical errors into the user application?

    One can argue that IF IIS can determine the good value, then the application should never get the bad value to begin with… and recycling is not necessary so this discussion is moot. 😉

Thus, in order for you to keep your process recycling AND not have your clients getting kicked out of their applications, you must make sure they are architected to work with recycling by storing its state outside of the IIS worker process. Depending on the application, this may not be possible, but this is really a problem for you to solve…


Personally, I think it is strange that you are trying to establish a process that recycles the application pool every 2 hours. This sort of aggressive action usually indicates the presence of some misbehaving user code, and you usually get better bang-for-the-buck by fixing that bug (see this blog entry for the diagnostic approach). The ability to health-monitor and application pool recycle is a crutch to help you hobble along in the face of adversity, to buy you time to fix the underlying issue… and not as a primary means of running a broken application which will ultimately cause your demise.


//David

Comments (45)

  1. Brian Weis says:

    This is the reason why i will give you an example.

    EX: We have a company that has 10 users connect to a web-based application. And when we look in task manager at the W3WP.exe it shows a memory usage of around 160 MB. We had all of the users log off all within 1/2 an hr of each other but the

    memory usage does not drop. Is this memory really used up or is just waiting for another app that needs it.

    The memory does not clear up within task manager until we run a recycle.

    This is why we want to run recycling every few hours.

  2. David Wang says:

    Brian – Your intuition is on target; just missing some details. Let me explain…

    Server applications, contrary to client applications, expect to run for a long period of time. Thus, a lot of server-side application employ caching to improve performance over time, at the expense of memory, for heavily used traffic patterns which may span across users.

    For example, response to /default.aspx may be cached in memory such that after 1 user accesses it and pre-compile/caches it, the other 9 users immediately retrieves the URL without much processing. This is a huge performance win at the expense of spending memory to cache it.

    In other words, the fact that after 10 users connect to an application, log off, and after half an hour the memory still remains used is insignificant. You have to accept that this may be the expected "working set" of your application.

    What you should be concerned about is this — suppose 10 users repeatedly log on/off and use the application over a period of time doing the same sort of thing and the memory used seems to increase without bound — that is a sign of memory leak and must be addressed.

    In other words, until you determine *through experimentation* the "working set" memory of that application, you cannot distinguish between "expected memory caching" (i.e. "working set") and "aggressive memory caching" (i.e. "memory leak"). And if you cannot distinguish between the two, arbitrary numbers like 160MB or invoking periodic recycling simply does not make sense.

    It may be that for 10 users, your application’s working set is 200MB after 1 hour and stays steady thereafter — in which case you are prematurely worrying about nothing by looking at things in 30 minutes.

    Sorry, there is no magic bullet nor calculation to automatically determine the proper metrics or need for recycling – it is all specific to the application and requires human experience to decide.

    //David

  3. jason lee says:

    Hi David,

    We have built a Client Server application that uses webservices to

    handle exchanges between the Client and the Server.  (Think web-based application, but with a Win32 client instead of a browser.)  The system is stateless, so the above discussion of lost state does not apply to us.  However, we have been puzzled to find that when the worker process recycles, that often causes our client to freeze up, sometimes temporarily (1 to 5 minutes), and sometimes fatally.  It seems that either the old process is not completing its request queue before termination, or the new process is not picking up the new requests.

    To investigate this, we used a free utility called Process Explorer to

    access the threads of the worker process.  We noticed that, in those cases where the clients were unaffected by the recycling, the new worker process immediately listed a thread to the webservice DLL; in contrast, in cases where the client hung temporarily, the new worker process failed to list a thread to the webservice DLL at first, and after a few minutes, the expected thread would appear and the clients would unfreeze.  

    We also found that if we killed the worker process’s thread to the

    webservice DLL, sometimes a replacement thread would immediately appear and the client would continue undisturbed, but sometimes the thread would not reappear and the client would freeze fatally.  

    Can you tell us why recycling might cause the symptoms we’re observing?

    Thank you very much

  4. David.Wang says:

    jason – I think you need to look through the code in your custom client and web service to figure out what you are observing. What you are describing do not look like IIS issues but probably result from miscommunication between your custom application’s expectations and IIS configuration. This is because the problem is always with how the user application deals with recycling – you can always configure IIS6 to not recycle.

    For example, you say "… the old process is not completing its request queue before termination" – which is perfectly possible and by-design on a recycle. If a worker process had a long running user application and a recycle was initiated, if the application within the worker process failed to shutdown/finish within the "Recycle Wait Time Period" of 90 seconds by default (for example, a lengthy synchronous network call to SQL that took >90 seconds), IIS will go ahead and kill the worker process *without* finishing that request. Recycling makes a best effort to process the request, but if you have bad requests, IIS will kill them to be defensive. I have no idea whether your Custom Client can deal with "network data loss" in such a case, nor whether your Custom webservice allows such long requests to happen… but they can explain your temporary hangs.

    You also say "… the new process is not picking up the new requests" – which does not make sense because following a recycle, any outstanding requests to the application pool are queued and go to the new worker process and your web service decides whether to work on it or not. It is simply not possible that the new worker process is *not* picking up the requests – if that was real, lots of people would have complained about missing requests by now. So, I suspect that either your Custom Client is not making those new requests until later, or your Custom webservice is not processing them for some reason.

    Also, I ignore your observations regarding killing the worker process’s thread using Process Explorer because that is hypothetical and not real coded behavior.

    Finally, we may be looking at a startup/shutdown bug within the user application code. This happens far more frequently than people admit (because people always think their code is perfect), and it happens frequently when people run code on IIS6 which may have haphazardly worked on IIS5. This is because recycling is basically like shutdown/restart of IIS5 (except no client connection loss), and that event rarely happens prior to IIS6. But on IIS6 with recycling enabled, it can happen all the time, and startup/shutdown bugs in user code quickly show up under pressure.

    Absent more concrete information like source code, it sounds like if a work-item was lost over the recycle due to it taking too much time, the custom Client has problems dealing with it. It may make retry-requests after 5 minutes and "recover", or maybe it never recovers. But in either case, the problem is not with the recycling but how the user application deals with the recycling, and the reasons there are quite arbitrary and endless.

    //David

  5. Good says:

    Really nice thoughts!

    Do you have any ideas about intermittent performance problems? Computing resource are fine, and when slowness is found, recycling will "solve " the problem.

    Someone suggested "perhasp app pool is running out of threads", where can I find this setting or measure it then?

    Cheers

  6. SP says:

    Hi,

    We are facing a wierd problem with the worker process recycling. The web application dlls are deployed on a NAS. When the NAS connection is lost, the server gives the appropriate errors. The worker process gets recylced. The strange thing is, if we try to reconnect or refresh the web page, a new request is given to the old worker process which is getting recycled. We could observe it when we debugged the application. This causes the further requests to fails since the application does not get initialized properly. We could see the application_start event for the old worker process, which should not be the case. And when this happens all the subsequent requests fail. What might be the cause? Is this a problem with the application or the IIS giving requests to wrong process? We have tried to put in a kill call for the current process after a custom terminate routine. This seems to solve the problem. But is it a valid solution?

  7. David.Wang says:

    SP – How does losing the NAS connection trigger the recycling of the worker process? That does not sound like a built-in IIS6 behavior.

    In particular, it sounds like your triggering of the "recycling" is not known to IIS nor HTTP.SYS, so further requests still get routed to the old worker process.

    When IIS triggers the recycling of a worker process, an atomic "switch" of the Application Pool Queue happens such that new requests get routed to new worker processes while the old worker process finishes up on the existing requests. This prevents what you claim from happening.

    Right now, it sounds like you are running custom recycling-related code on the server and you are observing bugs in it, not IIS6.

    //David

  8. SoP says:

    hmm. I have a doubt with the process recycling then. I think my understanding about the process recycling is wrong.  In our case when the connection with the NAS is lost, we get an Application_End event at global.asax.cs. Once the  connection is restored an Application_start is received to the same worker process. Is this behaior correct? My understanding was when we receive Application_End event, worker process (w3wp.exe) will be killed by IIS and a new w3wp.exe will be spawned at the time of next request. Is my understanding correct? I am a novice with the IIS process recycling concepts and hence not able to understand the behavior that I am seeing. Any help will be much appreciated.

    Thanks,

    SP

  9. David.Wang says:

    SoP – What you describe sounds like Application Domain Unloading, which is an ASP.Net specific concept that has no correlation to IIS worker process recycling.

    IIS process recycling concepts are pretty simple – worker process starts up only when servicing requests, and you configure a whole bunch of metrics on when the worker process goes away – such as by time, by # requests, by CPU, on-demand, etc. These metrics are generic and know NOTHING about Application Framework specific concepts like Application_End/Application_Start, etc.

    What is happening in your case is this – When ASP.Net detects that an assembly is missing or "refreshed", it has to restart the AppDomain to reload/init that assembly – so it unloads the Application Domain (thus you get an Application_End event), and on next new request to an unloaded AppDomain, you get an Application_Start event. This is pretty much by-design behavior, and your application should work well with it. If it doesn’t, it is usually a problem in your application.

    In other words, Application_End event (an ASP.Net concept) has no correlation to worker process recycling (an IIS concept). Can you point me to the documentation that gave you this understanding?

    //David

  10. SoP says:

    Thanks a lot for this information. Not any documentation in particular. I think I got confused going through the worker process recycling and application domain concepts. Thanks a again for the information.

    SoP

  11. Liam says:

    We’ve just had heaps of trouble with what we thought was either the worker process being recycled or application_end being called and it now looks like neither were the cause of our cache issues.  for whatever reason, the cache just kept losing entries randomly – although the higher the load, the more frequent the losses.   We’ve completely given up trying to figure out why and stopped using the cache altogether in favour of a statis class initialised in global asax that exposes a hashtable…it never fails.

  12. David.Wang says:

    Liam – I do not know what cache you are referring to, but in general, caches all have insertion and eviction policies, so can you enumerate the exact insertion and eviction policies of the cache and what does the cache do when it is "full".

    In other words, with high load, maybe you are caching so many cold objects that the hot ones get evicted somehow — and this appears as "losing entries randomly" to you. What is obvious to you to cache is NOT so obvious to the cache – depends on its insertion/eviction policies. Thus, you are basically wasting time trying to figure out WHY when you cannot enumerate the insertion/eviction policy of the cache.

    So, it is not that the static class is better or never fails – it is that you happen to know its insertion/eviction policy.

    //David

  13. Last year we deployed a new underwriting system. Since the Fall of 2006 we have been steadily releasing

  14. Topcat says:

    David, assuming you are receiving a lot of concurrency or that you have a memory leak what is the highest you can safely set your application pool w3wp.exe process to recycle and still prevent out of memory errors?  I know this sounds like a dangerous question, but assuming I can’t fix the poor code, I want to know how far I can push the limit.  Let’s assume I have 32GB of physical memory.  Is the limit around 900MB physical or can I go higher?

  15. David.Wang says:

    Topcat –

    You can just set any arbitrary private-bytes-consumed limit, determined through experimentation. 900MB is just as arbitrary as 2000MB.

    Otherwise, you will have to figure out:

    1. What memory pool is the leak in and what is the limit of that memory pool

    2. What is the rate of leakage (either in terms of time or #-requests)

    And you calculate:

    #1

    —- = units to hit limit

    #2

    And you make sure to either recycle by time or by #-requests at about 80% of the limit.

    //David

  16. Jim says:

    Hi

    Great posts and helping me to understand a current problem we have.  I’m somewhat of a newbie to this btw.

    One of our Sharepoint sites (just a webapp in IIS) stops functioning after an overnight app pool recycle ie. cannot find resource or locate page.  We can resolve this by renaming web.config or moving/replacing a dll file, hitting site getting an error, renaming web.config and on next hit all seems well.  I’m guessing the rename of web.config is causing something in IIS to recompile the webapp hence why it works.

    What I’m not sure about is how to properly diagnose what may be causing this problem?  Any help gratefully received.

  17. David.Wang says:

    Jim – what Application Pool are you recycling, and do the applications in that Application Pool support recycling.

    Changing web.config or changing a DLL file in /bin causes ASP.Net to recompile that web application on next request.

    My guess is that your problem happens because of the overnight Application Pool recycling of an application that does not support it, and that causing ASP.Net to recompile the web application resets whatever internal linkages that are broken by the unexpected Application Pool recycling.

    FYI: This really has very little to do with IIS and everything to do with diagnosing the behaviors of an arbitrary application, such as Sharepoint. If you truly want to get to the bottom of things, then you should hire a professional to help – like Microsoft PSS or appropriately skilled consultant.

    //David

  18. Jim says:

    Hi

    Thanks David – I’ll take a look at the application support for recycling app pools that you mention.  The particular site which is having problems includes some workflow/infopath functionality.  We have several other MOSS sites on the same server which are fine after the overnight app pool recycle and don’t use this functionality.

    We haven’t done any separation of apps across different app pools yet but this seems a sensible next test to try.

    I’m sure you are right about Sharepoint causing the problem.  I’ll take your advice and engage a pro to help.

    many thanks

    Jim

  19. Brett says:

    David – Thank you for all of the information. I have a couple of questions. I’m kind of new at IIS  configuration so I hope they make sense.

    1) Is there a standard app pool recylce schedule that you use as part of a preventative maintenance plan? Is it necessary to recycle the pools every 8, 12, or 24 hours if you do not see issues? Can/should you base a recycle only on memory usage?

    2) Do you have a recommendation for setting memory limits for application pools? If you assume there are no known memory leak issues aside from normal wear and tear what percentage of available memory should you use? The environment I’m asking about is a SharePoint 2007 front end sever with nothing else running on the box.  

    Any insight you can offer would be appreciated.

    Brett

  20. We’re experiencing a symptom in one of our ASP .NET applications that could use some direction before analysis. The symptom is a user will start a session with the application and begin working, shortly, the performance degrades to such a slow pace that work in the session becomes impossible-or the user leaves a working session to take a break, etc. — then tries to create a new session, but the session will not connect. We’ve been restarting the application pool to address both conditons, however we know this isn’t the solution. From an administrative perspective, all I’m seeing in the logs are idle timouts, etc.

    Any advice on what else to look for to help resolve this issue over here would be very helpful.

  21. Okisan says:

    Thanks for very useful article.

    My application pool stop sometimes. I wish to restart the application pool automatically if it stop. How can I do?

    Thanks.

  22. Hi guys says:

    Just a comment about the freeze in the apps,

    if I understood, your application is connected to webservices.

    If you restart the appPool, you propably will require the web app to recompile on the first access to it, that is how .net framework works on websites, and a webservice is a kind of a website.

    I think it is normal… You application is just waiting the ready state of your webservices.

    Regards,

    André Luiz Sobreiro

    MCPD

  23. David.Wang says:

    Okisan – if the application pool stops, it means that some catestrophic error happened in the application running in the application pool. You need to follow the instructions in my "AppPool Crash Diagnosis" link from all my blog entries to diagnose the cause.

    Until you determine and resolve the crash in your application, your application pool will continue to periodically stop.

    //David

  24. Stormy says:

    IIS6 and 7 I believe come with a default recycle value set, and you should always consider increasing or extending those values based on your available memory, expected load, number of processes and processors on your server, etc. Microsoft has an algorythm for that increase, I believe, to max that out.

    If your app is poorly written…not closing objects and releasing threads, using too much memory per session, heavy user load event design, etc, you can crash the AppPool more than expected. Your only work around is to get a faster or bigger server, and bump up the app pool defaults, or look at your processes and see if threads or objects are not getting collected and disposed in a timely fashion.

  25. Richard says:

    Hi everybody

    I have a little question… In our environment we have a Web services based on XML files, when we edit an xml file for a specific web site, we apply an iisreset command to take the new configuration.

    What happens if we apply an application pool recycle on the web site? Does the web service take the new configuration?

    Thanks

  26. David.Wang says:

    Richard – the answer completely depends on how your Web Service reads configuration.

    It could require no action — as example, how changes in web.config are picked up by ASP.Net. Or it could require actions like iisreset or application pool recycl. It all depends on the Web Service’s code.

    //David

  27. Richard says:

    Ok. Thank you David. I’ll check that.

  28. terri hawkins says:

    I am new to IIS management, and could use some help please.  

    I have a web server which is running several ASP applications which have memory leaks.  I am trying to set up the application pool recycling to free the memory without having our site go down.  We established a maximum of 450K of memory before we have a problem, so I set the application pool to recycle at that point.

    Generally, this works, I can look at the event logs and see where the application pool recycled without incident (it occurs roughly 3 times per day), however, sometimes I end up having the site go down (our customers receive out of memory errors) and have to sign in and recycle it myself.

    Here is my question (although any thoughts on the above are also welcome), if I reduce the amount of memory it can use, do you think that would solve the problem? Or would it just occur more often?  Is this normal behavior for a recycle? Is it just that there are so many processes running at that time that it takes too long to startup and shutdown (the out of memory errors always seem to occur during the day).

    I was also wondering if I could leverage both the virtual and physical memory limits… is there a rule of thumb about that, or would it not help?

    Any assistance is greatly appreciated.

    terri

  29. jon says:

    Dave,

    You come off as a complete dick, chill out man-

    John

  30. jon says:

    Dave,

    Sorry, did’nt know this comment section wasn’t moderated, I retract my earilier staement.

  31. David.Wang says:

    terri hawkins – the correct solution is to fix the ASP applications to not have memory leaks. What you need to do is to force these applications to be properly stress and scale tested before being put in production, or else your life will be an endless firedrill to find a bandaid to someone else’s bugs in production. If you cannot affect the development process to improve this mess, your life will be miserable and there is no way I can help.

    Realistically, your are really not doing IIS management. You are basically baby-sitting someone else’s poor application on the server you are responsible for. And you can either keep taking that abuse, or strike back at the source of the abuse and improve application quality to truly improve your life so that you can actually do IIS management.

    Configuration Application Pool Recycling is just a temporary bandaid to attempt to work around a bug. It may/not work.

    Figuring out the Application Pool recycling metric is more an art than science — it completely depends on the behavior of the bug being worked around. The reason I say that it is an art and not science is because if you knew the nature of the leak, you would probably be able to fix the leak. It is because you don’t know the source/type of leak that you have to "guess" — hence art and not science.

    It is impossible for me to give concrete answers to your questions because they depend on the behavior of your ASP applications. You need to closely monitor the server to understand how the error comes about… but if you knew that, you probably understand the nature of the leak and would not be guessing.

    For example, if you reduce the limit for memory based recycling, they should occur more frequently, but that is no defense against an application suddenly spiraling out of control using memory. Also, what makes you say that "there are so many processes at that time that it takes too long to do X/Y"? Where’s the concrete proof? How is that related to out-of-memory?

    Hopefully, you see that everything is a possibility. It all depends on the application and the nature of the bug itself. Without diagnosing the bug, it is nearly impossible to provide a rule-of-thumb to work around the bug. If there was a rule-of-thumb, then it means that all bugs work the same way, and that is simply untrue.

    //David

  32. Dinu John says:

    I have a web application that deployed in windows 200 server IIS6.0, running in its own application pool. We are using some page level caching also, while accessing the site with more than 50 users, abrupt session expiry is happening and we could understand that the application pool recycling is happening and the cache value is also cleared. The memory usage at that time is 40%. Could you please give your thoughts on what may be the problem?

  33. Hi,

     I have the following doubt.

    Is there any way that the worker process can determine at what time it is going to be recycled  or can it calculate how many minutes are there before the app pool recycle the worker process?

    Any method that gets called or any indication that the worker process can listen to so that it come to know before hand that recylce process is to begin shortly.

    Regards

    Vishal

  34. Kirtesh Jain says:

    Hi David,

    I am running an Insurance web application on .NET 2.0  this application has 1000 s of user every day and huge volume.

    In the application we show all the data for insurance policies on a grid (paged properly)

    Now the problem is .. once in a weeks time we receive errors navigating the page containing grid :

    Errors e.g.

    1.) System. System.Web.HttpException: Multiple controls with the same ID ‘PnlState’ were found. FindControl requires that controls have unique IDs. at System.Web.UI.Control.FillNamedControlsTable(Control namingContainer, ControlCollection controls) at System.Web.UI.Control.FillNamedControlsTable(Control namingContainer, ControlCollection controls) at System.Web.UI.Control.FillNamedControlsTable(Control namingContainer, ControlCollection controls) at

    2.) System. System.Web.HttpException: Multiple controls with the same ID ‘PnlPolicyNumber’ were found. FindControl requires that controls have unique IDs. at System.Web.UI.Control.FillNamedControlsTable(Cont

    Now when we recyle the application pool the site is back available and user can browse through grid.

    Please suggest some solution so that where we can fix in application or some setting in IIS so that we do not get this errors.

  35. Cristian says:

    Hi Kirtesh,

    could be a cache problem

    Take a look at http://support.microsoft.com/kb/834608

    Cheers.

  36. Kirtesh says:

    Hi Cristian,

    I checked the url posted by you but it seems to be applicable for .NET framework 1.1 . I am using Framework 2.0 . Can I still use the hotfix ?

  37. Stuart says:

    David,

    A ton of very useful information on this post/thread (you are the top hit for "iis application pool recycling"). Thank you for taking the time to maintain the thread (I see the posts go back over three years).

    However, after digging through it all, I still have a question as I have been refactoring a moderate volume (~150k pages/day) web site to address pervasive memory issues. The recycle issue is resolved, but we have a new issue now.

    Our initial problem was multiple previous developers used Session rather liberally for everything, so our process would roll over about every 10 minutes under load because of the Session growth. This also caused a loss of state on every recycle which created a different set of problems.

    Over a period of about four months, we slowly but surely found and addressed Session offenders. In most cases, we were able to move state to move the data to the HTTP Cache. The caching design was essentially two ideas; (1) for data objects, there is now only 1 copy used by all users rather than N copies for N users and (2) collections are not populated with copies of data objects but instead contain a set of IDs that we can manually dereference to data objects in cache. This brought our memory usage well within limits and the CPU overhead seems to be negligible.

    To address the potential loss of what state was left in session, we moved to ASP State server. As a side note, our original sessions were so large, they quickly generated SQL IO errors in production I assume because it couldn’t push enough data fast enough for all the users (we found 50Mb+ sessions in the DB from that initial attempt). Anyway, we keep viewstate in Session now, along with a handful of other things, and its working great so far.

    We recently pushed the last of our caching optimizations and we ran for most of the day at 15% CPU, about 4k objects totaling 150Mb in HTTP Cache (estimated), and a process memory that stabilized at 350Mb in task manager. I considered this our new baseline.

    I would note that we have relatively aggressive HTTP Cache timeouts, so that 4k entries roll over relatively frequently. In some cases our process will spike up to 600Mb according to the Peak Memory Usage column in task manager. This seems kind of high for our caching, but I cant rule that out (i.e. I haven’t caught the process with a high value like this and checked the cache).

    With that as a back drop, on to the new problem which we really started noticing this week with the last of the code pushes.

    We run very consistent all day long with stable CPU, load, and memory, but pretty much anytime after say, 3 hours of load, the CPU can snap up to  about 80% and stay there (its immediate, not gradual). When this happens, there is no noticeable change in memory or load and the site usually becomes very slow (we get email alerts regarding CPU and timeouts). I have seen it recover from this condition before without recycling, if left alone long enough, but sometimes it eventually wanders up to 100% where I usually stop waiting to see what happens and recycle. Resetting the application pool manually snaps us right back down to our new baseline (15% CPU/350Mb process) as soon as the recycle completes.

    It seems we are accumulating something over time that is eventually showing in CPU utilization. On this note, I belabored the caching stuff because I felt we might be rolling so much through the cache that its fragmented badly enough to become expensive after some period of time, but I really have no basis for that notion.

    I suppose with the StateServer we could schedule recycles based on page hits, but I was really hoping to see the system run all day with the same app pool…

    Any thoughts on our situation?

    Thanks in advance for your time,

    Stuart

  38. Athi S says:

    Hi David,

           We have a website, which has global.asax file, which contains Seesion_End as well as Application_End events. These are firing in normal scenarios. But, if IISreset happens, it doesnt… Is it the behavior or anything wrong in our side? If its the behaviour, is there any other alternative to know when IIS is reset? Ultimately, we want to log some data (in the backend) when iisreset happens.

    Thanx in advance.

    Athi

  39. Hi David,

    Please excuse this post resurrection but we are having some issues with one of our implementations and I thought you might be able to offer some insight.

    We are running 2 web servers on a load balanced setup.

    We are getting an intermittent issue whereby the site seems to go down (on one of the web servers) and traffic is not redirected to the other. Thats a separate issue.

    The main issue is why the site is going down and its somethign i cannot fix 🙁

    We do know that simply recycling the application pool brings the site back online.

    The site is configured to recycle its application pool overnight but this issue seems to arise through the day.

    Its odd as the solution is not exposed to all users YET. Its only a handful.

    Can you offer any advice or next steps that you think i should be looking at, i would greatly appreciate that.

    Thanks

    Andy

  40. Andy Hopper says:

    Hi David,

    Please excuse this post resurrection but we are having some issues with one of our implementations and I thought you might be able to offer some insight.

    We are running 2 web servers on a load balanced setup.

    We are getting an intermittent issue whereby the site seems to go down (on one of the web servers) and traffic is not redirected to the other. Thats a separate issue.

    The main issue is why the site is going down and its somethign i cannot fix 🙁

    We do know that simply recycling the application pool brings the site back online.

    The site is configured to recycle its application pool overnight but this issue seems to arise through the day.

    Its odd as the solution is not exposed to all users YET. Its only a handful.

    Can you offer any advice or next steps that you think i should be looking at, i would greatly appreciate that.

    Thanks

    Andy

  41. Dick Fung says:

    We have a Exchange server 2003 SP2 that is running on a Windows server 2003 R2. Recently it reported the default application pool was stopped because unexpected error and the mobile users who use rcp over https cannot sync with Exchange server. Therefore I tried to set recycle the process but it make my users always disconnect from Outlook (also RPC over Https). Is any better solution like patch can solve my difficulty?

  42. Dick Fung says:

    We have a Exchange server 2003 SP2 that is running on a Windows server 2003 R2. Recently it reported the default application pool was stopped because unexpected error and the mobile users who use rcp over https cannot sync with Exchange server. Therefore I tried to set recycle the process but it make my users always disconnect from Outlook (also RPC over Https). Is any better solution like patch can solve my difficulty?

    Thanks

    Dick

  43. Vijay says:

    We have a web application and there are about 300 concurrent users and we have 4 GB RAM on the server , sometimes we feel the application slowing and it crashes and immediately it comes up and then after a few minutes it crashed again. IS it a memory constraint. Did the max numbe rof people access the pool

  44. Srinath says:

    First of all thanks for all the userful information in this blog. Not sure if someone is monitoring this blog, but im hoping someone might have an answer for my question.

    For one of the applications pools in IIS, we have the maximum used memory set to 800 MB. We have enabled webgarden with 12 processes. But, An individual worker process does not always consistently recycle when the 800 MB threshold is crossed. What could be the problem?

    Srinath

  45. Murali says:

    I have deployed 4 asp.net web applications in windows server 2012 and IIS 8, i am facing a peculiar issue of webpage not opening till i manually recycle the application pool. please help me…