ASP.NET Case Study: Lost session variables and appdomain recycles


Last night I got a question from one of the readers of the blog that went like this:

 

 

“We are facing a problem that i cannot understand, every now and than i see that my app domain is recycled (i have a log in the application_end), I check the IIS logs and i don’t see a restart of IIS and i know that no one is changing configuration (web.config).

 

I wanted to know if you know of any way that i can pinpoint the reason for that app domain to die?

 

The application pool that i am using only have the recycle every 20 minutes of idle time enabled“

 

 

 

..and I thought that since I haven’t written for a while (due to a really nice and long vacationJ) and this is a pretty common scenario I would write a post on it…

 

Before we go into the details of how to figure out why it is recycling I want to bring up two things

 

  1. What happens when an application domain is recycled

  2. What are the reasons an application domain recycles

 

What happens when an application domain is recycled?

 

In ASP.NET each individual asp.net application resides in its own application domain, so for example if you have the following website structure

 

WebSite root

          /HrWeb

          /EmployeeServices

          /FinanceWeb

          /SalesWeb

 

…where each of the subwebs HrWeb, EmployeeServices etc. are set up as an application in the internet service manager, you will have the following application domains (appdomains) in your asp.net process

 

System Domain

Shared Domain

Default Domain

Root

HrWeb

EmployeeServices

FinanceWeb

SalesWeb

 

Apart from the first three domains (in italic) which are a bit special, each of the other ones contain the data pertinent to that application (Note: this is a bit simplified for readability), specifically they contain these things worth noting…

 

  1. All the assemblies specific to that particular application

  2. A HttpRuntime object

  3. A Cache object

         

When the application domain is unloaded all of this goes away, which means that on the next request that comes in all assemblies need to be reloaded, the code has to be re-jitted and the cache including any in-proc session variables etc. are empty.  This can be a pretty big perf-hit for the application so as you can imagine it is important to not have the application domain recycle too often.

 

Why does an application domain recycle?

 

An application domain will unload when any one of the following occurs:

 

  • Machine.Config, Web.Config or Global.asax are modified

  • The bin directory or its contents is modified

  • The number of re-compilations (aspx, ascx or asax) exceeds the limit specified by the <compilation numRecompilesBeforeAppRestart=/> setting in machine.config or web.config  (by default this is set to 15)

  • The physical path of the virtual directory is modified

  • The CAS policy is modified

  • The web service is restarted

  • (2.0 only) Application Sub-Directories are deleted (see Todd’s blog http://blogs.msdn.com/toddca/archive/2006/07/17/668412.aspx for more info)

 

There may be some reasons in 2.0 that I have missed but hopefully this should cover most scenarios.

 

Specific issues

 

I want to pay a bit more attention to a few of these, which seem to be especially popularJ

 

Unexpected config or bin directory changes

 

You swear on all that is holy that no-one is touching these, but still when we start logging (as I’ll show later) the reason for the app domain recycle is a config change… how the heck can that be?

 

Elementary, Dr. Watson… something else is touching them… and that something else is usually a virus scanning software or backup software or an indexing service.  They don’t actually modify the contents of the files, but many virus scanners etc. will modify attributes of files which is enough for the file changes monitor to jump in and say “aha !, something changed, better recycle the appdomain to update the changes”.

 

If you have a virus scanner that does this, you should probably consider removing the content directories from the real-time scan, of course after carefully making sure that no-one can access and add any virus software to these directories.

 

Web site updates while the web server is under moderate to heavy load

 

Picture this scenario:  You have an application with 10 assemblies in the bin directory  a.dll, b.dll, c.dll etc. (all with the version number 1.00.00). Now you need to update some of the assemblies to your new and improved version 1.00.12, and you do so while the application is still under heavy load because we have this great feature allowing you to update assemblies on the go…  well, think again…

 

Say you update 7 of the 10 assemblies and for simplicity lets say this takes about 7 seconds, and in those 7 seconds you have 3 requests come in… then you may have a situation that looks something like this…

 

Sec 1.           a.dll and b.dll are update to v 1.00.12    – appdomain unload started (any pending requests will finish before it is completely unloaded)

Sec 2.           Request1 comes in and loads a new appdomain with 2 out of 7 of the dlls updated

Sec 3.           c.dll is updated                                    – appdomain unload started         (any pending requests will finish before it is completely unloaded)

Sec 4.           d.dll is updated

Sec 5.           Request2 comes in and loads a new appdomain, now with 4 out of 7 dlls updated

Sec 6.           e.dll and f.dll is updated                       – appdomain unload started         (any pending requests will finish before it is completely unloaded)

Sec 7.           f.dll is updated

Sec 8.           Request3 comes in and loads a new appdomain with all 7 dlls updated

 

So, many bad things happened here…

 

First off you had 3 application domain restarts while you probably thought you would only have one, because asp.net has no way of knowing when you are done.  Secondly we got a situation where Request1 and Request2 were executing with partially updated dlls, which may generate a whole new set of exceptions if the dlls depend on updates in the other new dlls, I think you get the picture…  And thirdly you may get exceptions like “Cannot access file AssemblyName because it is being used by another process” because the dlls are locked during shadow copying.  http://support.microsoft.com/kb/810281

 

In other words, don’t batch update during load…

 

So, is this feature completely worthless?  No… if you want to update one dll, none of the problems above occur… and if you update under low or no load you are not likely to run into any of the above issues, so in that case you save yourself an IIS restart… but if you want to update in bulk you should first take the application offline.

 

There is a way to get around it, if you absolutely, positively need to update under load, and it is outlined in the kb article mentioned above…

 

In 1.1 we introduced two new config settings called <httpRuntime waitChangeNotification= /> and <httpRuntime maxWaitChangeNotification= />. 

 

The waitChangeNotification indicates how many seconds we should wait for a new change notification before the next request triggers an appdomain restart. I.e. if we have a dll updated at second 1, and then a new one at second 3, and our waitChangeNotification is set to 5… we would wait until second 8 (first 1+5, and then changed to 3+5) before a new request would get a new domain, so a request at second 2 would simply continue using the old domain. (The time is sliding so it is always 5 seconds from the last change)

 

The maxWaitChangeNotification indicates the maximum number of seconds to wait from the first request. If we set this to 10 in the case where we update at second 1 and 3, we would still get a new domain if a request came in at second 8 since the waitChangeNotification expired. If we set this to 6 however, we would get a new domain already if a request came in at second 7, since the maxWaitChangeNotification had then expired.  So this is an absolute expiration rather than a sliding… and we will recycle at the earliest of the maxWaitChangeNotification and waitChangeNotification.

 

In the scenario at the beginning of this section we could have set the waitChangeNotification to 3 seconds and the maxWaitChangeNotification to 10 seconds for example to avoid the problems.

 

(I know this explanation might have been a bit confusion but I hope you catch the drift)

 

A few things are important if you fiddle with these settings

 

  1. They default to 0 if not set

  2. maxWaitChangeNotification should always be >= waitChangeNotification

  3. If these settings are higher than 0 you will not see any changes until the changeNotifications expire. i.e. web.config changes and dll changes etc. will appear cached.

 

Re-compilations

 

A common scenario here is that you have a set of aspx pages (containing some news items and what not) and you have a content editor that goes in periodically and updates the news with some new articles or other new content.   Every time you update an aspx page it has to be recompiled, because again, asp.net has no way of knowing if it was a code update or just update of some static text… all it knows is that someone updated the files.

 

If you have followed some of my previous posts you know that assemblies can not be unloaded unless the application domain is unloaded, and since each recompile would generate a new assembly there is a limit to how many recompiles you can do, to avoid generation of too many assemblies (and thus limiting the memory usage for these).  By default this limit is 15.

 

If the contents of the page is constantly updated I would recommend to dynamically get the content from a database or file rather than actually modifying the aspx pages. Or alternatively using frames with HTML pages for this content.

 

How do you determine that you have application recycles?

 

If you experience cache or session loss, it is probably a good bet, but to make sure you can look at the perfmon counter ASP.NET v…/Application Restarts.

 

How do you determine what caused an appdomain restart?

 

In ASP.NET 2.0 you can use the built in Health Monitoring Events to log application restarts along with the reason for the restart.  To do this you change the master web.config file in the C:WINDOWSMicrosoft.NETFrameworkv2.0.50727CONFIG directory and add the following to the <healthMonitoring><rules> section

 

                <add name=Application Lifetime Events Default eventName=Application Lifetime Events

                    provider=EventLogProvider profile=Default minInstances=1

Comments (112)

  1. Snorrk says:

    Great post!

  2. Jason says:

    Great post, Tess.  Thanks so much for sharing this info!

  3. Well, if you have ever wondered, Tess has a great post listing out the various reasons why. Well worth

  4.  My

     ASP.NET 2.0 Tips, Tricks, Recipes and Gotchas &quot;Highlights Page&quot; [Via:

     ScottGu ]

     Sending…

  5. Great post indeed, Thank you.

  6. Great post it helped my a lot, we were having many timeout’s and we didn’t have a clue on why they were happening. Thank’s.

  7. Yaroslav Vysotsky says:

    Great! Very useful post. Hope that will help us to solve a problem. Thanks!

  8. Below are some nice articles and links I&amp;rsquo;ve found on the web over the last week that I enjoyed,

  9. JasonBSteele says:

    Tess,

    This is a really useful post and has given me several pointers.

    However, I am having a problem where my users are loosing their session and the folowing messag eis appearing in th event log:

    Event code: 4005

    Event message: Forms authentication failed for the request. Reason: The ticket supplied has expired.

    Doing a bit of googling reveals that a few other people are having this problem but nobody has really come up with a good explanation why. Perhaps this could be the subject of a future blog? /emote Fails miserably at charming smile

    Also of possible relevance is that my web application is organised with each of my ciustomers having their own virtual folder which is pointing at a particular version of the product. So I have folders v1.1, v1.2 etc with virtual folders CompA..CompS pointing to v1.1 and CompT..CompZ pointing to v1.2. When I make a new release I create a new release folder and then repoint each company virtual folder as and when they are ready for the new release.

    What are the pros and cons for this? Am I wasting loads of memory by having an AppDomain for each company when they could all be using a single AppDomain?

    Thanks for your time,

    Jason

  10. A very nice post. keep them coming.

  11. Ramachandran says:

    I want to know the internals of how the ASP.NET request is processed from the begining ( the user typing the url) to the end ( Appdomain unload). Is there any material or links. This post is really useful and if there are many posts like this then that will be great :)

  12. Ken Egozi says:

    I’m having a Session clearing problem with a project I’m running.

    I’ll try yo investigate on you leads (web.config change by some service) but I think that it’s due to another thing. The website allows a lot of file uploads, through simple http requests, so my guess is that it is caused by inflation in the memory used by the AppDomain. Do you think that this can cause AppDomain restarts?

  13. Tess says:

    If it is due to memory usage, it would probably be caused by an out of memory exception, but then you wouldnt only see an appdomain restart, you would see the whole process restart (likely with a stopped unexpectedly eventlog entry).  Could it be that as part of the file upload you copy the results into the bin directory or similar?  I would try to log the restart reason and go from there.

  14. Tess says:

    Ramachandran,

    I am not sure if this is what you are looking for, but here is an article on the page lifecycle, from when the request comes in to the ASP.NET layer to the unload of the page object

    http://msdn2.microsoft.com/en-us/library/ms178472.aspx

    It doesn’t cover what happens before ASP.NET gets a hold of it, and it doesnt cover the appdomain unload since that is not really part of a request, but a part of the whole application, so again, I am not sure if it is what you are looking for, but it is pretty good…

  15. Tess says:

    JasonBSteele,

    I hadn’t seen the issue you describe before but a quick internal search revealed that you might be running into this issue which has to do with URL rewrites:

    http://support.microsoft.com/kb/911300

    The NullReferenceException mentioned in there might be what causes your tickets to seem expired.

    If it was my own app, and I could reproduce, i would probably log exceptions to see what I could find.

    I am not sure if you have any alternative ways of setting it up (without getting too deep into your particular architecture) but each appdomain has a certain overhead, and depending on how things are linked it might be quite large if you keep loading copies of the same dlls.

  16. Troy Hall says:

    In my case, I was using the Enterprise Library Logging Block to write to the web application root directory (locally, on my dev machine only!). Once I reconfigured to stop this logging, and reset the app domain manually (resaving web.config) the session reset went away. THANKS!!! This page gave me the clues to move on.

  17. Tess says:

    Awesome, glad it helped:)

  18. Rich S. says:

    I enjoyed the article, and was hopeful when I started tracking application restarts. I found that my application isn’t restarting when the session is unexpectedly abandoned.

    Can you recommend a course of debugging when it is only the session that seems to be intermittently and [seemingly] inconsistently restarting?

    TIA,

    Rich

  19. JasonBSteele says:

    Tess,

    I wouldn’t say that my application has the characteristics suggested in http://support.microsoft.com/kb/911300

    It doesn’t use URL re-writing, it simply uses several virtual folders (which often point to thesame physical folders).

    However, further investigation has revealed that the ticket expired event does not appear to coincide with the session loss.

    I followed your advice and swiched on the health monitoring logging and found that the session loss was caused by a "Application is shutting down. Reason: Configuration changed"

    However, nothing is touching the files you mentioned (no indexing, virus checking backups etc.). I have used FileMon and it also indicates that nothing is happening to the files.

    I know of one other person is having the same problem, see http://www.velocityreviews.com/forums/t294620-application-restarting-configuration-has-changed.html

    who has been trying to resolve this with MS support for quite some time now.

    Have you heard of this issue, or can you suggest a way of investigating it further?

    Thanks,

    JasonBSteele

  20. JDelmerico says:

    JasonBSteele & Tess,

    I can confirm the same behavior here. I am seeing app domain restarts no apparent cause under the label of “Application is shutting down. Reason: Configuration changed.” according to the ASP.NET 2.0 health monitoring API. I also ran Filemon and confirmed there were no writes affecting of config file around the time of the restart. The only file activity affecting the web.config file is the aspnet_wp itself, and only reads.

  21. Sudeep says:

    JDelmerico,

    In ASP.NET 2.0, any change to the contents of the application folder can cause an appdomain restarts. Check out Toddca’s site:

    https://blogs.msdn.com/toddca/archive/2006/07/17/668412.aspx

    Does filemon show any files from the application folder being *touched* ?

  22. Tony Bunce says:

    How can I get a dump file when my AppDomain restarts?  I am logging asp.net app restarts via the method described by ScottGu and am getting some weird results and want to dig deeper.

    _shutDownMessage=Overwhelming Change Notification in C:WINDOWSMicrosoft.NETFrameworkv2.0.50727Config

    nothing is changing in that directory (watched it with filemon) and even turned off NTFS last access time to see if that would help and it didn’t.

  23. Jethro says:

    I can confirm the same behavior here. I am seeing app domain restarts no apparent cause under the label of "Application is shutting down. Reason: Configuration changed." according to the ASP.NET 2.0 health monitoring API. I also ran Filemon and confirmed there were no writes affecting of config file around the time of the restart. The only file activity affecting the web.config file is the aspnet_wp itself, and only reads.

  24. Tess says:

    If you receive the Overwhelming Change Notifications and noone is touching the files I would recommend that you contact Microsoft Support.  It might be a problem in the framework that you are running into where ASP.NET reports an error during filechangesmonitoring as "overwhelming change notifications",  normally the way to troubleshoot is to set breakpoints on filechanges notification functions in the  webengine to see what error occurred.

  25. abbstl says:

    JasonBSteele –

    Did you ever figure out the cause of your issue?

  26. JohnWilley says:

    We’re having frequent recycles with event message "Reason: Configuration Changed".  Like several other folks, we’ve confirmed with filemon that nothing but w3wp.exe is touching the application folder.

    Wondering if anyone found the problem??

  27. Payal says:

    Great article..and helped me a lot..but i have one other issue related to session variable..My application is in asp.net 1.1..when i login in to the application i can see my details properly..but after some time in between i m able to see other person’s details instead of mine..and this problem is not happening frequently.. It is happening some times..can you please tell me reason and the solution. I have stored user information in one session variable..depending on that i have shown user details.

    This problem we are getting on live server only..not on other servers.

  28. Tess says:

    Hi Payal,

    Pretty much every time I have seen this it has been one of two things.

    1. The thing that is being put in session scope is comming from a cached or static object of some sort without proper locking mechanisms so while you are reading, some other thread/user has already written new data to it.  

    2. A router or load balancer is used and the users come from the same ISP getting the same IP address so the router gets confused and returns the session cookie to the wrong user

    The second scenario is a misconfiguration in a router, and is by far much less frequent than the first scenario.

  29. payal says:

    Hi Tess,

    thanks a lot.. i think i m getting this problem because of second issue because i have already checked first option but session scope is not comming from any cached or static object.

    and if that is the case, i should get the same issue on my other servers.

    Previously for this same site..we are getting above issue i.e. lost of session variable due to appdomain recycles.. that time ur article helped me lot..as we found on live server there are two anti-viruses are running and causing the problem.

  30. BorisL says:

    Hi Tess,

    We are getting the same problem as JasonBSteele:

    Event code: 4005

    Event message: Forms authentication failed for the request. Reason: The ticket supplied has expired.

    I have a question about this: I have FormsAuthentication enabled, so shouldn’t the ticket expiration (the cookie) be handled regularly by FormsAuthentication subsystem?

    We have this problem only on IIS5, it can not be reproduced on IIS6 :-(

    Any suggestions would be greatly appreciated…

    Boris

  31. Jethro says:

    Hi, there are several Apps in IIS 6.0 on Windows 2003, the config is:

    <authentication mode="Forms">

           <forms name="QuotationCollector" loginUrl="Authentication.asmx" protection="All" timeout="20" slidingExpiration="true">

    <credentials passwordFormat="SHA1">

    <user name="System" password="7C222FB2927D828AF22F592134E8932480637C0D"/>

    <user name="Feader1" password="7C222FB2927D828AF22F592134E8932480637C0D"/>

    </credentials>

    </forms>

         </authentication>

    the others are :

    authentication mode="Forms">

    <forms loginUrl="Login.aspx" name="TradingConsole" protection="All" timeout="15" slidingExpiration="true"></forms>

    </authentication>

    Now, only the first app log event every minute as:

    Event code: 4005

    Event message: Forms authentication failed for the request. Reason: The ticket supplied has expired.

  32. Tess says:

    I think the most common reason for this error is that the appdomain restarted, and since you are posting these questsions in connection to a post about appdomain recycling does that mean that you do see in perfmon that the appdomain has recycled?

    If it has, add logging to see why it recycled

  33. Boris says:

    Hi Tess

    I solved the "Forms authentication failed for the request. Reason: The ticket supplied has expired. " problem by providing permanent valudation and decription keys in machine.config. The problem was caused by generation of new random validation/decription keys when appdomain restarted…

    Here the article about how to create keys…

    http://support.microsoft.com/default.aspx?scid=kb;en-us;Q312906

    Boris

  34. Glenn Slayden says:

    Wow, after 8 hours of debugging the buck stopped here. You hit the nail on the head with this great article. In my case (ASP .NET 2.0, C#), a static generic List object was sometimes getting reinitialized across a System.Diagnostics.Process.Start() spawn of Perl.exe. As you point out, the AppDomain is being unloaded because Perl changes various files in a subdirectory.

    I haven’t tried this yet, but would a fix be to create a different application for those subdirectories (using IIS manager), thus excluding them from the main AppDomain and its notifications?

  35. Tess says:

    Hi Glenn,

    I’m happy that the post helped:)  The optimal setup would be if the files that were updated weren’t in the application path at all, but i suspect that they are supposed to be accessible from the web so in that case if you set the subdir up as an application, only that application would restart when the files are updated and a new request comes in to that application which would save the main app from restarting.

  36. Glenn Slayden says:

    Tess, thanks for your reply.

    Unfortunately, my idea does not work; creating an application for the subdirectory does not exclude it from the parent application’s FCN (file change notification).

    There are not many other options. It may be possible to create a directory junction (as discussed by Toddca) but from my reading it seems like they are somewhat delicate.

    What does seem to work–and is not delicate enough–is setting <httpRuntime waitChangeNotification="2147483647" maxWaitChangeNotification="2147483647" /> in the site’s web.config, but then we lose the whole auto-deployment feature, and it becomes difficult to re-JIT the site without a full-blown manual ‘iisreset’. With this option, to force the AppDomain to reinitialize I tried having a button on the site which causes it to call HttpRuntime.UnloadAppDomain() or AppDomain.Unload(AppDomain.CurrentDomain) on itself (even from a QueueUserWorkItem), but had little success. I can’t see a way to do it from the IIS Manager snap-in, either.

    Glenn

  37. Glenn Slayden says:

    Another thing to note is that the delays imposed by waitChangeNotification and maxWaitChangeNotification are in effect even for explicit calls to HttpRuntime.UnloadAppDomain(). In fact, the function blocks for the entire wait duration!

    This is not what I would expect. The result is that when using a large delay value, the function is not usable and there’s little chance of recycling your AppDomain.

    I confirmed this result with varying delay wait values ‘w’: the the call to UnloadAppDomain() blocked for ‘w’ seconds, after which the AppDomain was cheerfully unloaded.

    Glenn

  38. Colin Hale says:

    Hi,

    Back in Feb, there was a post from Payal.  He was experiencing a problem whereby users were getting other users session variables.

    I am having the same problem under .net1.1

    Changing the software to ‘cookieless’ – to display the sessionID in the url, we can see that the sessionIDs are changing to someone else’s.

    In your response, you give 2 possible scenarios

    I think I can rule the first one out, as no shared components are involved No reused or ‘cached’ components are being stored in the session, only a simple class structure defined within the asp.net project,  instantiated as an object and stored in the session within an ascx.vb file.

    As for scenario #2, what tool(s) would you recommend to determine the ip address of the incoming request?

    Thanks,

    Colin.

  39. Colin Hale says:

    Just for closure:

    We had to contact Microsoft.  They gave 3 or 4 scenarios, but it turned out that one of our aspx pages had the directive <%@ OutputCache %>.  This caused the page to be cached, hence the second user will get the first users page (and details taken from the first session).

    What was weird (and a bit of a red fish) was the 2nd user getting the 1st users session ID.

    After removing this directive – it all started to work correctly.

    Colin.

  40. Glenn Slayden says:

    And for closure on mine:

    It looks like the directory junction approach, while bizarre, will work for my application and I will eventually have to use it.

    For now though, I have been testing <httpRuntime waitChangeNotification="2147483647" maxWaitChangeNotification="2147483647" /> on the production site for a week and this works as well, with the caveat that this disables auto-deploy and you must do a manual iisreset to force updating or re-JIT.

    Glenn

  41. joe says:

    Hi There,

    I am having a similar problem, what other possible causes did MS give you.

  42. Colin Hale says:

    MS response was as follows:

    Everything we have checked so far results to be fine, so there is only one possibility left which can lead to this problem: caching.

    We are probably sending to the second client a page (or data) being cached by the first client.

    There are 3 way this could happen:

    1) using OutputCache on those pages: please, open the page directive section of all the pages in relation with the problem and ensure you are not using OutputCache.

    2) caching sensitive data using the Cache[] object. Please, make sure you aren’t

    3) having a proxy/firewall between server and clients which caches pages: please make sure this does not happen. Could you try to reproduce the problem using 2 different boxes as client, both directly connected to the server (no proxy between)?

    Hope that helps,

    Colin.

  43. mkraley says:

    I too am having mysterious "Reason: Configuration Changed" application restarts. Filemon confirms that there are no writes taking place either in the application tree or in the Framework folder tree. The first mention of anything relevant in Filemon is a CLOSE of web.config, which I suspect is just the beginning of the restart, not the cause.

    Has anyone made progress on deducing why this happens?

    …Mike

  44. Colin Hale says:

    Have you tried excluding the virt folder from any anti-virus s/w?

  45. says:

    In ASP.NET Case Study: Lost session variables and appdomain recycles beschreibt Tess was passiert wenn

  46. Minnie says:

    I have a question. How can we capture session timeout in IIS/ASP.NET?

  47. Salman Banani says:

    Great article, really helped.  My antivirus was casuing my appdomain to recycle.

    Thanks man.

  48. A local radio show has started giving out cookie points to their interviewees for personal development,

  49. Rob Kent says:

    Using ASP.NET 2, we’re seeing Appdomain recycles pretty frequently with a low load – just a few users on our test system. The reason given in the eventlog is ‘HostingEnvironment caused shutdown’, which looks to me like a memory issue.

    Looking at memory, we see big chunks being taken every time another user logs in and the wp process gets to 2.5G, at which point I guess it decides to recycle itself.

    I can’t find any reason for that amount of memory being used and on my development laptop, that doesn’t happen. We are using Ajax, and WCF to communicate with the data layer. Does anyone have any experience of that combination and whether there are known performance/memory issues.

    Cheers, Rob

  50. Tess says:

    Since you have a pretty well defined scenario where you see memory going up everytime someone logs in, an idea would be to

    attach windbg or visual studio for that matter,

    load up sos,

    do !dumpheap -stat just before the log in

    hit g to go

    !dumpheap -stat after the log in

    compare the outputs to see what types of objects have increased the most and investigate those.

    Alternatively you could hook up CLR Profiler and look there at what objects have been instantiated while you were logging in.

    Thanks

    Tess

  51. George says:

    We’ve been experiencing this same issue for quite some time now. Thanks to your blog entry here, I was able to confirm that it is indeed a configuration change which causes our worker process to recycle. Specifically, I see the "Application is shutting down. Reason: Configuration changed." message in the event log now.

    I figured it would be fairly simple to track down the culprit (indexing service is off, and virus scanner is explicitly set to ignore any directories that contain config files), but that has proven to not be the case. I left FileMon running overnight (filtering on *.config) and waited until the worker process recycled. Unfortunately, it looks as if nothing accessed (read or write) a config file immediately before the recycle occured. The sequence of events is as follows:

    Event code: 1002

    Event message: Application is shutting down. Reason: Configuration changed.

    Event time: 10/4/2007 5:01:27 PM

    Event time (UTC): 10/4/2007 9:01:27 PM

    Event ID: de5016e0e27944dc82f644b975c01d38

    Event sequence: 140208

    Event occurrence: 1

    Event detail code: 50004

    Process information:

       Process ID: 6052

       Process name: w3wp.exe

       Account name: NT AUTHORITYNETWORK SERVICE

    and from the FileMon log:

    5:00:24.164 PM w3wp.exe:6052 FASTIO_QUERY_OPEN D:AppRootmaindevelopmentsweb.config NOT FOUND Attributes: Error

    5:01:13.917 PM w3wp.exe:6052 FASTIO_QUERY_OPEN D:AppRootmainreportsweb.config NOT FOUND Attributes: Error

    5:01:13.917 PM w3wp.exe:6052 IRP_MJ_DIRECTORY_CONTROL D:AppRootmainreports NO SUCH FILE FileBothDirectoryInformation: web.config

    5:01:13.917 PM w3wp.exe:6052 FASTIO_QUERY_OPEN D:AppRootmainreportsweb.config NOT FOUND Attributes: Error

    5:01:27.448 PM w3wp.exe:6052 FASTIO_QUERY_OPEN C:WINDOWSMicrosoft.NETFrameworkv2.0.50727Configmachine.config SUCCESS Attributes: A

    5:01:27.448 PM w3wp.exe:6052 IRP_MJ_DIRECTORY_CONTROL C:WINDOWSMicrosoft.NETFrameworkv2.0.50727Config SUCCESS

    FileBothDirectoryInformation: machine.config

    5:01:27.448 PM w3wp.exe:6052 IRP_MJ_CREATE C:WINDOWSMicrosoft.NETFrameworkv2.0.50727Configmachine.config SUCCESS Options: Open  Access:

    00020000

    5:01:27.448 PM w3wp.exe:6052 IRP_MJ_QUERY_SECURITY C:WINDOWSMicrosoft.NETFrameworkv2.0.50727Configmachine.config BUFFER OVERFLOW

    Repeat 1000x as the restart finishes. The curious thing about this is twofold:

    1) FileMon reports the same PID for the worker process before, during, and after the restart. A new process should be spun up after the restart, correct?

    2) There are multiple worker processes running on this machine, but only one ever restarts. This all but rules out global web.confg/machine.config changes.

    Is there anything else other than .config files that ASP.NET considers to be configuration files? Any ideas would be greatly appreciated.

    Thanks for your time,

    George

  52. Tess says:

    Hi George,

    Not sure why the config changes are being reported, but to answer your questions

    1) The workerprocess doesnt restart only the application domain inside it so the PID should stay the same.  It will only change if the whole process restarts.

    2) No, only .config files are considered configuration files.

    I do seem to recall some issues reported in the early stages of 2.0 with false config restarts because of exceptions under certain conditions but i dont recall the exact details.  You might just want to check that you are on the latest rollups/service packs for the framework if you haven’t checked already.

  53. George says:

    Hi Tess,

    Thanks for the clarification.

    It looks like the framework is indeed up to date. The only other thing I can think of is that we recently began writing data to a directory inside of the application’s directory tree, although Filemon confirms there are no writes in the moments leading up to the recycle. Maybe this is tangentially related to the overwhelming changes (http://support.microsoft.com/?id=913297) issue?

    I’ll keep after it and post back here if I discover anything new.

    Thanks again,

    George

  54. Jethro says:

    the perfmon counter ASP.NET v…/Application Restarts always ==0 for 2 hours, BUT the eventLog is full of this event, see Event occurrence: 36401 !   Help !!!

    Event code: 4005

    Event message: Forms authentication failed for the request. Reason: The ticket supplied has expired.

    Event time: 12/7/2007 11:17:06 AM

    Event time (UTC): 12/7/2007 4:17:06 AM

    Event ID: 57a0de50fe674a3fb5ab61c36938c1a7

    Event sequence: 85310

    Event occurrence: 36401

    Event detail code: 50202

  55. salsam says:

    I have also facing same problem.  during normal execution of the site my session information getting lost.

    Event log is full of such errors.

    Event code: 4005

    Event message: Forms authentication failed for the request. Reason: The ticket supplied has expired.

    Event time: 14/03/2008 13:19:26

    Event time (UTC): 14/03/2008 13:19:26

    Event ID: d9f0333c2eed46e0b0207da69d2ea70e

    Event sequence: 154

    Event occurrence: 5

    Event detail code: 50202

    If some on find solution of this problem please do post here.

  56. Tess says:

    Have you gone through and verified if your appdomain is recycling (in performance monitor) that is the most common case for lost session variables.  

    Did you follow hte article and log application restarts in the eventlog?  The solution will depend on why the application/appdomain is restarting…

  57. salsam says:

    If you mean, copying ‘Application Lifetime Events ‘ in web.config file. then yes i have copied it to webconfig but didnt got any of known reason as yet, like it is being happened becos of ‘Configuration changed.’

    do i have to restart webserver in order to make below mention changes come in to effect.

    <add name="Application Lifetime Events Default" eventName="Application Lifetime Events"

                       provider="EventLogProvider" profile="Default" minInstances="1"

                       maxLimit="Infinite" minInterval="00:01:00" custom="" />

  58. Tess says:

    No you shouldnt have to,  the best thing is probably to contact support and have someone look at it in more detail

  59. salsam says:

    <add name="Application Lifetime Events Default" eventName="Application Lifetime Events"

                       provider="EventLogProvider" profile="Default" minInstances="1"

                       maxLimit="Infinite" minInterval="00:01:00" custom="" />

    Even after adding this , still i m getting this in  event log.

    Event code: 4005

    Event message: Forms authentication failed for the request. Reason: The ticket supplied has expired.

    Event time: 14/03/2008 13:19:26

    Event time (UTC): 14/03/2008 13:19:26

    Event ID: d9f0333c2eed46e0b0207da69d2ea70e

    Event sequence: 154

    Event occurrence: 5

    Event detail code: 50202

  60. Salsam says:

    hi Tess,

    it is now confirmed that in my case appdomain is not recycling. beoc since one day i have received even single event of appdomain recycle. so problem is somewhere else. i have also update machine.cofig file with custom generated keys but still having same errors in evenlog.

    please suggest some solution

    thanks

  61. salsam says:

    hi Tess,

    it is now confirmed that in my case appdomain is not recycling. beoc since one day i have received even single event of appdomain recycle. so problem is somewhere else. i have also update machine.cofig file with custom generated keys but still having same errors in evenlog.

    please suggest some solution

    thanks

  62. Tess says:

    I would have to look at it in a lot more detail to see what is going on and unfortunately I can’t really do that in this blog setting so the best is probably to contact support and have someone look at it for you.  sorry:(

  63. The purpose of my presentation was to show some common pitfalls and of course to show off windbg and

  64. Salsam says:

    Hi guys,

    I was working on the issue where event viewer showing the log "Forms authentication failed for the request. Reason: The ticket supplied has expired.

    ".

    Finally able to get to the roots of this message. here what it was happening to me.

    I was working on web application which is using Form Authentication and Sessions to store the user specific profiles.

    Here is my setting in web.config file.

     <sessionState timeout="60" mode="InProc" cookieless="false"/>

    <authentication mode="Forms">

    <forms loginUrl="Pages/Login.aspx" defaultUrl=".PagesDefault.aspx" timeout ="60" slidingExpiration ="true">

    </forms>

    </authentication>

    Now problem was before i didn’t had form authentication timeout, which had default of 30 mins. So when  request is made after being idle for 30 mins, Form authentication was throwing the user back to login page. beacuse after 30 mins the ticket supplied with the request  gone expired. and this also logged warrning that user was thrown out beacuse of expire ticket passed.

    other problem i was having that Form Authentication was unable to redirect the user back to login page and end up with page no found error. reason for that was, missing <authorization> tags.

    ForAuthentication requires deny  anonymous access in the Authorization tags in order to redirect the user back to login page in case if authorization failed or timeout.

    So in order to fix these issues i have stick in the Authorization tags. and along with that, i have also set the Form Authentication timeout same as session timeout.

    But this still give the warraning in Event viewer beacuse if session and Form Autentication Cookie is expiring at same time. and user makes any request after the timout first things which triggered is Form Authentication and it will throw the user to login page and log the event in Event viewer.

    To do  quick fix i have increased Form Autentication timeout to 240 mins by leaving the session timout 60 just to make session timout before Form Authentication timeout. So when session timeout after 60 mins Code is already there to take the user back to login page so user can re loggin.

    If any one has batter solution to this problem, please do share it

    thanks

  65. Adam says:

    Hi

    I just thought I would share another scenario I came across, not that its because of application pool recycling but because it looked like it at first.

    I had an application that kept losing its session state and the actual reason was I had a maximum number of worker processes allowed set to 5.

    Because the application used inproc session state the session state was lost each time it jumped a worker process. The setting is in the properties of the Application Pool.

    Thanks

  66. Tess says:

    Absolutely,  the same is of course true for web farms.

  67. Hi Tess,

    I was wondering why the waitChangeNotification and maxWaitChangeNotification settings support has been dropped in IIS 7. It was the only mechanism available for hot swapping applications on the fly.

  68. Tess says:

    I honestly had no clue that that was the case, and it surprises me, because that is a framework setting, not an IIS setting.

  69. Oh, and obviously, this occurs in the Integrated Pipeline mode only.

  70. Tess says:

    I would have to look at this in more detail and unfortunately I do not have an IIS 7 machine set up at the moment, but the bullet item says:

    __________

    25) IIS 7.0 always restarts ASP.NET applications when changes are made to the application’s root web.config file.  Because of this, waitChangeNotification and maxWaitChangeNotification attributes have no effect.

    IIS 7.0 monitors changes to the web.config files as well, and cause the ASP.NET application corresponding to this file to be restarted without regard to the ASP.NET change notification settings including the waitChangeNotification and maxWaitChangeNotification attributes in the system.web/httpRuntime configuration sections.

    _____________

    My comments: Web.config changes have always been monitored and do restart the application domain.  Off the bat I can’t see how this disables waitChangeNotifications other than for changes in web.config.  

    One of the biggest reasons you would use them is to be able to update multiple dlls at a time without restarts, and from this comment I am not sure that that would change.  The only thing that this comment seems to suggest is that waitChangeNotifications are not honored for web.config changes.

    I might be wrong though as I haven’t tested it personally.

  71. Issues caused by frequent application restarts are pretty common. If you are interested in what causes

  72. simone_b says:

    Yes, it happens to ignore the settings only when changes to the web.config file occur, but it represents a major issue since replacing the web.config file is exactly what happens when doing hot swap of applications.

  73. Tess says:

    Still that would only mean one appdomain recycle, which is what you would see anyways with the maxWaitChangeNotification, so I am not sure I understand the issue…

  74. simone_b says:

    This is the issue, supposing that I don’t set any waitChangeNotification or maxWaitChangeNotification:

    I trigger a batch update, using robocopy for example, which takes all files from the source directory and copies them over the files in the old directory, where the application is running.

    1. some – but not all – files from the source bin folder are copied to the destination – the application restarts immediately

    2. the web.config file is copied – the application restarts immediately

    3. the rest of the content of the bin folder is copied, the application restarts again.

    This is the issue, supposing that I set waitChangeNotification = 10 and maxWaitChangeNotification to a big value (meaningless in this example):

    1. some – but not all – files from the source bin folder are copied to the destination – this takes much less than 10 seconds – the application doesn’t restart

    2. the web.config file is copied – the application restarts immediately

    3. the rest of the content of the bin folder is copied, the application waits 10 seconds from the last copy then restarts.

    Here we have 2 restarts, one of them probably error prone, since not all assemblies were copied and might be incompatible to each other.

  75. Tess says:

    ok, then you would either have to do the robocopy first, and then copy the web.config   or do the update when the site is not live…

  76. simone_b says:

    Yes, that’s the workaround I came up with, but that’s not as optimal as it was before, and mostly I can’t see the reason why it should have been changed in IIS7, it worked so well in the previous versions..

  77. I started this blog 2.5 years ago today, mostly because I felt that the same types of issues came up

  78. I just read a post on If broken it is, fix it you should , where he&#39;s debugging a performance problem

  79. Mike Driest says:

    Hello Tess,

    On Thursday, February 01, 2007 7:27 AM you wrote:

    ————————————————————————-

    Hi Payal,

    Pretty much every time I have seen this it has been one of two things.

    1. The thing that is being put in session scope is comming from a cached or static object of some sort without proper locking mechanisms so while you are reading, some other thread/user has already written new data to it.  

    2. A router or load balancer is used and the users come from the same ISP getting the same IP address so the router gets confused and returns the session cookie to the wrong user

    The second scenario is a misconfiguration in a router, and is by far much less frequent than the first scenario.

    ————————————————————————-

    We have a similar issue with one of our Internet facing ASP.NET 2.0 Applications with only 1 customer/computer where we’re seeing 3 Public IP Addresses hit our web servers (2 node with NLB) in a matter of 3.5 minutes.  These 3 Public IP Addresses from the Client are in the same Subnet and are literly .101, .102 and .103.  Our ASP.NET App uses .NET Forms Authentication and I think it’s possible that the session is being responded to via the Wrong Client Public IP Address.  The user is logged out shortly after they login to the application and try to execute a Search.

    Our code barely uses caching and other customers/users don’t have this issue from other corporate networks, computers or ISPs.  We’ve used the user’s credentials on PCs on the Public Internet and our Corporate computers successfully without errors or being logged out.

    Would you agree that the issue lies within the client/customer network infrastructure (firewalls, routers, etc)?

    Thank you,

    Mike Driest

  80. I got an email with some questions around application domains, application pools and unhandled exceptions

  81. Craig Hubbard says:

    Thanks for this post, the recycling issue has become quite an issue for our DotNetNuke implemenations of recent times (Sept 2008) and this post will help me to investigate further – great stuff!

  82. I have handled a lot of session related issues till now; I felt the need to key in a few checkpoints

  83. I stumbled across a very interesting find yesterday morning and Todd Carter confirmed it by sending a

  84. You learn something new every day… I was working on an issue with appdomain restarts caused by page re-compilations

  85. Application restart is a common issue in ASP.NET web application. Here is a very good blog entry providing

  86. Måns Tånneryd says:

    Nice post! But, I have a problem with app domain restarts that does not seem to be covered in your post. My web application runs smoothly on a win2003 server in iis 6 but when I deployed it on a win2008 server with iis 7 it started hickupping. After about 60 seconds the appdomain restarts due to "_shutDownMessage=IIS configuration change". No configuration files have been changed! What is happening though is that the web application is downloading files using ftp on a background thread and placing them in a directory in the app_data folder. I seems unreasonable to me that adding files in the app_data folder should cause the application to restart. Calling an "IIS configuration change" doesn’t make any sense to me either!

    Any ideas?

    /Måns

  87. Tess says:

    Hi Måns,

    Just adding or removing files from there should not restart the app, but if something creates or deletes subfolders, even in the app_data folder then you will see app restarts with the iis config change message

  88. Nick Crowther says:

    Just fixed my session loss problem (deleting folders in App_Data). Thanks Tess (& Måns :), useful blog.

  89. Siddhartha Bhuyan says:

    Nice article Tess. Got a really good understanding about application domain recycling.

    I have been working on a content management website since the past few weeks and my part involves the file handling functionalities.

    Our website actually has to create files, folders etc. in a 3-level hierarchy and also manipulate them based on what the user does.

    While everything was working fine, I kept getting puzzled as to why, everytime a directory was renamed (moved) successfully, I used to lose everything in the session… Debugging only revealed that the session was getting invalidated.

    Only after I read this and Scott Forsyth’s article, did the problem become clear to me.

    But I have to admit that this is a sticky problem. And there is no easy way to work around this one.

    Keep writing!

  90. Surat Jewpojaroen says:

    Thanks for sharing knowledge.

    It’s really good article.

  91. Codru says:

    Hi!

    I have a question, maybe you could help me. For example, if I try to submit a job (request to the server) after a recycle, my job is lost – no trace of it, no errors are logged, nothing but the information in event viewer that the application has recycled. I understand why this happens, but I would like to know if there’s by any chance a way of showing a message to the user that something happened and he will be logged out when submitting the job? Or maybe after he logged back in?

    I cannot see a way of doing this, since all the dlls need to be reloaded, but I thought it’s worth a shot to ask :)

    Thanks for the article, it’s really good and revealing :)

  92. Tess says:

    i dont really see a way that you could do that… unless perhaps you keep some session variables and detect that they are lost for example

  93. Josh says:

    Great article Tess!  I have been troubleshooting recycle events for a few days.  A combination of virus scanning and mis-used app pools in IIS6 were the trouble.  Thanks again.

  94. csaxena says:

    Great blog Tess.  I have learnt so much from it.  I have a few questions.  Is there any disadvantage of setting the value for maxWaitChangeNotification and waitChangeNotification to be really high (like 10 hours)?  We are seeing a lot of Application restarts in our DotNetNuke environment and we are not making any changes.  Is it because the different user controls on different pages are getting compiled and causing the application to be reloaded.  

    What is the disadvantage (other than the memory footprint) of increasing numRecompilesBeforeAppRestart to a really high number like 500 or 1000?  

    Thanks !

  95. Tess says:

    csaxena,

    There are basically two disadvantages.  

    1. Setting numRecompilesBeforeAppRestart to something really high, as you pointed out will mean that you store more copies of stale dlls in the process leading to higher memory usage.  That is pretty much the only drawback you will see from that…

    2. setting maxWaitChangeNotification and waitChangeNotification to something really high like 10 hrs means that changes made during that time that require appdomain restarts wont take effect.  For example if you modify the web.config, the old settings will be "cached" in that case for 10 hrs and new ones wont take effect until the appdomain is restarted.   Same with dll updates.  As long as you are aware of that and restart the appdomain manually (by restarting the process) when changes that you need to see go through immediately are made, you’ll be ok.  In general I really wouldn’t recommend setting this higher than 5-10 minutes though…

  96. frankosaurus says:

    Great article Tess…

    I have an ASP.NET application (in IIS7) where content editors make regular updates to .resx files.

    The problem is that this eventually causes the app domain to recycle.

    With changes to .aspx pages, I can set compilationMode="never" in web.config and I avoid the problem of excess recompiles.

    Is there a solution for .resx files??

    I would prefer not to set numRecompilesBeforeAppRestart to a high value (or 0) because I don’t want to use up RAM — I need RAM for ASP.NET cache.

  97. Tess says:

    unfortunately i dont think there is, but even if there was, that would mean that the updates wouldnt go through until you manually recycle.  So given that, perhaps it is better to have the content editors edit to a staging location and copy them all in one go at some point.

  98. srinivasan says:

    ”Sec 1.           a.dll and b.dll are update to v 1.00.12    – appdomain unload started (any pending requests will finish before it is completely unloaded)

    Sec 2.           Request1 comes in and loads a new appdomain with 2 out of 7 of the dlls updated

    Sec 3.           c.dll is updated                                    – appdomain unload started         (any pending requests will finish before it is completely unloaded)

    Sec 4.           d.dll is updated

    Sec 5.           Request2 comes in and loads a new appdomain, now with 4 out of 7 dlls updated

    Sec 6.           e.dll and f.dll is updated                       – appdomain unload started         (any pending requests will finish before it is completely unloaded)

    Sec 7.           f.dll is updated

    Sec 8.           Request3 comes in and loads a new appdomain with all 7 dlls updated

    So, many bad things happened here…

    First off you had 3 application domain restarts while you probably thought you would only have one, because asp.net has no way of knowing when you are done.”

    ———————————–

    In the above scenario, don’t you feel there are more than 3 appdomain restarts, but you’ve stated there are 3. Am I getting this wrong ?

  99. Bob Dickow says:

    Then how about the web.config <system.web><hostingEnvironment idleTimeout="Infinite" approach? The Microsoft docs suggest that this can be set in the application level web.config and will prevent application shutdown. Problem is, I tried this on a Server2003/IIS 6.0 hosting service setup and it had no effect. The result is that every few minutes the website page requests take 20 seconds or so for a page to load up as the whole site recompiles or whatever it needs to do. The config setting seems to have no effect. Would this be something that would be overridden higher up the chain?

  100. Deepti says:

    Hi Tess

    Came across your blog by chance and found it amazing! Was hoping you could help us with a weird problem we're encountering on our production server which has IIS6 installed.

    What we deal with is a simple WCF application which is hosted on default app pool in IIS. This is the only application hosted btw.

    Once in a while the server requests just hang up for a while, a blank period of total inactivity and at the end of the lull period activity suddenly shoots up and request completes successfully. But why this delay? There's nothing in the logs in event viewer or in the logs of the application which suggests anything. The time in between is just LOST. Nothing in the dump suggested deadlocks. Can you please help me figure out the root cause? Thanks a bunch!

  101. Abhijeet Dighe says:

    Hi,

    I am updating style.css dynamically which is in App_Themes, whenever style.css is updated user gets logged out frequently.

    I think application restarts on style.css changes since it is in App_Themes folder.

    Is there any workaround for this issue?

  102. Rahul says:

    Very Bad to the user.It can't give clear view to the user.

  103. HydTechie says:

    I have found a scenario where session is lost – In a asp.net page, for a amount text box field has invalid characters, and followed by a session variable retrieval for other purpose.After posting the invalid number parsing through Convert.ToInt32 or double raises a first chance exception, but error does not show at that line, Instead of that, Session being null because of unhandled exception, shows error at session retrieval, thus deceiving the debugging…

    HINT: Test your system to fail it- DESTRUCTIVE.. enter enough junk in unrelated scenarios

    for ex: after search results shown enter junk in search criteria and goto details of search result…

    , you would be able to reproduce this machine on your local code base too…:)

    Hope it Helps,

    hydtechie

  104. Klas says:

    Thank you, Tess!

    The explanation of when an application domain is unloaded helped me solve a performance problem for a customer.

  105. Danny says:

    Thank you very much Tessa Fernandez. You've helped me solve an issue of mine. I've blogged about it here: dannyvanderkraan.wordpress.com/…/losing-session-variables-with-asp-net

    The world of web development can be a treacherous place! Pphheeww…

  106. Tokenian says:

    so great, the bin directory change will restart the domain,how awful

  107. Thorsten says:

    Hi Tess,

    thanks for this great article, help me a lot.

    The next step for me is to understand where the FileChangesMonitor gets its configuration from? How does FileChangesMonitor know which files and directories to monitor?

    Any clues?

    Have a nice day!

  108. Fred says:

    I'm using Microsoft routing service with Net Tcp binding. This routing service has a heavy load, 200 – 300 calls per second. The problem is, the service stop processing request from time to time. But I can see the application pool is still "started" state, and the process is still running. But just doesn't process anything. I have to manually restart application pool.

    Any idea for that?

  109. srinivas says:

    Great Post … thank you so much for sharing this article.