How to Identify Why an Application Pool is Recycling in IIS 7+ and DebugDiag 2 Fails to Add Managed Breakpoint in .NET 4.6+ in one post


 

Identifying why Application Pool is recycling

Starting with IIS 7, application recycle for managed Web Applications does not recycle w3wp.exe (IIS Worker Process), instead it simply reload (unload and load a new one) the Application Domain. I mentioned it on another post how to identify the cause of an application pool recycle, but now I will tell how to capture a dump file when it happens.

Capturing a dump when an Application Pool recycles is as easy as capturing a dump file at System.AppDomain.Unload(System.AppDomain) via DebugDiag crash rule. However, when doing this on a machine with .NET 4.6, I was unable to capture the dump. Tinkering a bit I found how to resolve the issue via workaround which I explain later on this post.

After you capturing the dump, it is easy to find the cause if you use NetExt: Getting started with NetExt

Open dump file with WinDbg, load NetExt, index heap with !windex

image

The run !wruntime to know the cause:

0:032>; !wruntime
Runtime Settings per Application Pool
 
=========================================================================
Address         : 0000000DB8DDB100
First Request   : 10/29/2015 11:34:50 PM
Runing Time     : 00:00:07
App Pool User   : IIS APPPOOL\DefaultAppPool
Trust Level     : Full
App Domnain Id  : /LM/W3SVC/1/ROOT-4-130906352796685448
Debug Enabled   : False
Active Requests : 0n0
Path            : C:\inetpub\wwwroot\ (local disk)
Temp Folder     : C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files
Compiling Folder: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\e22c2559\92c7e946
Shutdown Reason : ConfigurationChange at 10/29/2015 11:34:57 PM
 
IIS configuration change
HostingEnvironment initiated shutdown
HostingEnvironment caused shutdownn   at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
   at System.Environment.get_StackTrace()
   at System.Web.Hosting.HostingEnvironment.InitiateShutdownInternal()
   at System.Web.Hosting.HostingEnvironment.InitiateShutdownWithoutDemand()
   at System.Web.Hosting.PipelineRuntime.StopProcessing()
 

To identify the file(s) changed, run this command: !wfrom -type System.Web.FileMonitor where ($enumname(_lastAction) == "Modified") $a("==================\nPath",DirectoryMonitor.Directory+"\\"+_fileNameLong),$a("Change Time:",$tickstodatetime(_utcLastCompletion.dateData)),$a("Action",$enumname(_lastAction))

0:032>; !wfrom -type System.Web.FileMonitor where ($enumname(_lastAction) == "Modified") $a("==================\nPath",DirectoryMonitor.Directory+"\\"+_fileNameLong),$a("Change Time:",$tickstodatetime(_utcLastCompletion.dateData)),$a("Action",$enumname(_lastAction))
==================
Path: C:\inetpub\wwwroot\web.config
Change Time:: 10/29/2015 11:34:57 PM
Action: Modified
==================
 
1 Object(s) listed
45 Object(s) skipped by filter

 

Setting a managed Breakpoint for Application pool to catch managed recycling

If you try to add a DebugDiag crash rule when a managed breakpoint is hit, you may fail if .NET 4.6+ is installed. I tested it with DebugDiag 2 Update 1 and it still fails. This is a step-by-step instruction that I sent to a customer having this issue. The instructions also work for people not having the issue, just skip the steps related to .NET 4.6.

*** Update: DebugDiag2 Update 2 resolved the issue that required steps 13 to 16. If you have DebugDiag2 Update 2 skip those steps.

 

  1. Download DebugDiag 2 Update 1 and install on the server showing the problem: http://www.microsoft.com/en-us/download/details.aspx?id=42933
  2. Download DebugDiag 2 Update 2 and install on the server showing the problem: https://www.microsoft.com/en-us/download/details.aspx?id=49924
  3. Run DebugDiag 2 Collection
  4. Choose Crash on Rule Type and click Next

clip_image001

 

4. Then choose A specific IIS web application pool and click Next

clip_image002

5. In the next window, choose the Application Pool you are having problem with (you will see different options in your environment) and click Next

clip_image003

6. In “Action type for unconfigured…” choose Log Stack Trace. In “Action Limit”, write 100. In “Maximum number of userdumps…” choose 1 as below (do not click Next yet):

clip_image004

7. Click on the button Breakpoints. Click on Add Breakpoint…

clip_image005

8. Type mscorlib.dll!System.AppDomain.Unload on Breakpoint Expression. Make sure This is a managed (.NET) breakpoint is selected.. Action Type is to be set to “Full Userdump” and action limit to be set to 1 and click Ok

clip_image001[5]

9. If things are correct you should see the screen below. Then choose “Save & Close”:

clip_image007

10. Click Next

11. By default, the dump will be written in C:\Program Files\DebugDiag\... If you want to save in a different place, change the appropriate filed (it will create the folder if it does not exist:

clip_image008

11. Click Finish (and Yes for any window that pops up):

12. Choose not to activate the rule at this time

clip_image009

 * For steps 13 to 16 it is better to have DebugDiag2 Update 2 installed than to make these extra steps

13. (ONLY IF .NET 4.6 is installed and DebugDiag2 Update 1 is being used) Open Notepad as Administrator

14. (ONLY IF .NET 4.6 is installed and DebugDiag2 Update 1 is being used) Open File C:\Program Files\DebugDiag\Scripts\CrashRule_WebAppPool_DefaultAppPool.vbs

15. (ONLY IF .NET 4.6 is installed and DebugDiag2 Update 1 is being used) Locate Function AddBreakpoint and replace until EndFunction by this code (in fact there is only one line added to load sos).

Function AddBreakpoint(BreakpointExpression, IsManaged)
    Dim LogString
    Dim bpID  
 
    LogString = "Attempting to set "
    If Not IsManaged Then LogString = LogString & "un"
    LogString = LogString & "managed breakpoint at " & BreakpointExpression                 
    WriteToLog LogString           
    ' This is the ONLY line added
    WriteToLog Debugger.Execute(".loadby sos clr")
    ' End of modification
 
    bpID = Debugger.AddCodeBreakpoint(BreakpointExpression, IsManaged)              
 
    If IsManaged Then
        If bpID < -1 Then
            RecreateDeferredManagedBreakpointExpressionsCollectionIfNecessary
            If IsEmptyOrNothing(g_DeferredManagedBreakpointExpressions) Then Exit Function
 
            If g_DeferredManagedBreakpointExpressions.Exists(BreakpointExpression) Then
                g_DeferredManagedBreakpointExpressions(BreakpointExpression) = bpID
            Else
                g_DeferredManagedBreakpointExpressions.Add BreakpointExpression, bpID
            End If
        End If     
    End If
 
               DbgState("BP_" & BreakpointExpression & "_ID") = bpID
 
    WriteToLog "bpID = " & bpID
 
               AddBreakpoint = bpID
End Function

16. (ONLY IF .NET 4.6 is installed and DebugDiag2 Update 1) Save the file

17. Right-click on the rule and choose activate

clip_image010

Other Approach

I talked to my dear DebugDiag colleague Wade Mascia and he suggested a cleaner approach: to add a custom rule for clr.dll module load. The rule would be simply load sos. This option also makes the export and import of the rule possible. Anyway, this issue should be resolved by next DebugDiag release.


Comments (0)

Skip to main content