Identifying why Application Pool is recycling
Starting with IIS 7, application recycle for managed Web Applications does not recycle w3wp.exe (IIS Worker Process), instead it simply reload (unload and load a new one) the Application Domain. I mentioned it on another post how to identify the cause of an application pool recycle, but now I will tell how to capture a dump file when it happens.
Capturing a dump when an Application Pool recycles is as easy as capturing a dump file at System.AppDomain.Unload(System.AppDomain) via DebugDiag crash rule. However, when doing this on a machine with .NET 4.6, I was unable to capture the dump. Tinkering a bit I found how to resolve the issue via workaround which I explain later on this post.
After you capturing the dump, it is easy to find the cause if you use NetExt: Getting started with NetExt
Open dump file with WinDbg, load NetExt, index heap with !windex
The run !wruntime to know the cause:
To identify the file(s) changed, run this command: !wfrom -type System.Web.FileMonitor where ($enumname(_lastAction) == "Modified") $a("==================\nPath",DirectoryMonitor.Directory+"\\"+_fileNameLong),$a("Change Time:",$tickstodatetime(_utcLastCompletion.dateData)),$a("Action",$enumname(_lastAction))
Setting a managed Breakpoint for Application pool to catch managed recycling
If you try to add a DebugDiag crash rule when a managed breakpoint is hit, you may fail if .NET 4.6+ is installed. I tested it with DebugDiag 2 Update 1 and it still fails. This is a step-by-step instruction that I sent to a customer having this issue. The instructions also work for people not having the issue, just skip the steps related to .NET 4.6.
*** Update: DebugDiag2 Update 2 resolved the issue that required steps 13 to 16. If you have DebugDiag2 Update 2 skip those steps.
- Download DebugDiag 2 Update 1 and install on the server showing the problem: http://www.microsoft.com/en-us/download/details.aspx?id=42933
- Download DebugDiag 2 Update 2 and install on the server showing the problem: https://www.microsoft.com/en-us/download/details.aspx?id=49924
- Run DebugDiag 2 Collection
- Choose Crash on Rule Type and click Next
4. Then choose A specific IIS web application pool and click Next
5. In the next window, choose the Application Pool you are having problem with (you will see different options in your environment) and click Next
6. In “Action type for unconfigured…” choose Log Stack Trace. In “Action Limit”, write 100. In “Maximum number of userdumps…” choose 1 as below (do not click Next yet):
7. Click on the button Breakpoints. Click on Add Breakpoint…
8. Type mscorlib.dll!System.AppDomain.Unload on Breakpoint Expression. Make sure This is a managed (.NET) breakpoint is selected.. Action Type is to be set to “Full Userdump” and action limit to be set to 1 and click Ok
9. If things are correct you should see the screen below. Then choose “Save & Close”:
10. Click Next
11. By default, the dump will be written in C:\Program Files\DebugDiag\... If you want to save in a different place, change the appropriate filed (it will create the folder if it does not exist:
11. Click Finish (and Yes for any window that pops up):
12. Choose not to activate the rule at this time
* For steps 13 to 16 it is better to have DebugDiag2 Update 2 installed than to make these extra steps
13. (ONLY IF .NET 4.6 is installed and DebugDiag2 Update 1 is being used) Open Notepad as Administrator
14. (ONLY IF .NET 4.6 is installed and DebugDiag2 Update 1 is being used) Open File C:\Program Files\DebugDiag\Scripts\CrashRule_WebAppPool_DefaultAppPool.vbs
15. (ONLY IF .NET 4.6 is installed and DebugDiag2 Update 1 is being used) Locate Function AddBreakpoint and replace until EndFunction by this code (in fact there is only one line added to load sos).
16. (ONLY IF .NET 4.6 is installed and DebugDiag2 Update 1) Save the file
17. Right-click on the rule and choose activate
I talked to my dear DebugDiag colleague Wade Mascia and he suggested a cleaner approach: to add a custom rule for clr.dll module load. The rule would be simply load sos. This option also makes the export and import of the rule possible. Anyway, this issue should be resolved by next DebugDiag release.