Identifying Memory Leak With Process Explorer And Windbg


 

Alik Levin    Your ASP.NET web application behaves strange. It gets stuck, it stop serving incoming requests, or gets recycled. You observe strange application pool recycles events in event log.
Chances you are dealing with memory leak in your ASP.NET web application. Lately I've been involved with few of such situations. Here is the method and the resources that helped me to extremely quickly identify the root cause of the memory leaks. 

Related Books 
_________________________________

 

I am super grateful to the resources published by Tess and Rico that discuss the process of memory leak identification in more detail:

Customer Case Study

The customer complained his application gets less responsive as more subsequent request come in. The customer reported the high CPU utilization. At some point the application gets stuck and the only way to get it back is..... iisreset. The customer felt there is a memory leak in the application.

Analysis - Summary of steps

  • Step 1 - Use Process Explorer To Confirm The Memory Leak
  • Step 2 - Take Memory Dump Using adplus.vbs
  • Step 3 - Use Windbg to identify the leaking Type
  • Step 4 - Use Windbg to identify to identify who's holding on the Type
  • Step 5 - Find the culprit in the source code

Step 1 - Use Process Explorer To Confirm The Memory Leak

I am using Process Explorer to quickly identify whether we are dealing with memory leak. In our case it showed we are. This is the memory consumption that Process Explorer reveals in very convincing way - you can clearly see how the memory consumption climbs up until the application hangs/restarts:

Porcess Explorer Memory Leak

Next question is who's consuming the memory? Windbg to the rescue.

Step 2 - Take Memory Dump Using adplus.vbs

Use Windows Debugging Tools package to take a memory dump of the leaking managed process. Download the tools and extract on the server where the application runs. Open command prompt and change the directory to the folder that contains the tools, then type the following command:

adplus.vbs -quiet -hang -p <<PROCESSID>>

where <<PROCESSID>> is the process of w3wp.exe PID (process ID) that runs the leaking application. The result should be another folder inside of tools folder with very lengthy name. Open that folder- you should see a big size file with DMP extension. This is the memory dump you have just taken.

Step 3 - Use Windbg to identify the leaking Type

To analyze the dump file you will going to use Windbg.exe. The tool is located in the Debugging Tools for Windows folder, same folder that contains adplus.vbs that was used in the previous exercise. Open the tool, Windbg.exe. Press Ctrl+D, locate your DMP file and open it.

Now you need to load SOS.DLL extension to analyze managed memory dump. Type the following command according ro what .Net framework you are analyzing:

.load clr10\sos - for Net 1.X

.loadby sos mscorwks.dll - for Net 2.0 and up.

Type the following command to identify the Type that consumes most of the memory:

!dumpheap -stat

The result should be similar to the following:

.....
654088b4 92811 13736028 System.Data.DataColumn
000dec48 44 13900264 Free
654359c8 31704 32845344 System.Data.RBTree`1+Node[[System.Int32, mscorlib]][]
7912d8f8 320769 47859900 System.Object[]
790fd8c4 827939 186834312 System.String

Total 3351270 objects

Most consuming is that System.String type that is responsible for almost 190 MB of allocated memory.

Step 4 - Use Windbg to identify to identify who's holding on the Type

Next question your should ask is who's holding on it so that it's not collected. For that purpose type the following command in Windbg but be prepared to hit Ctrl+Break to stop it as you do not want to list all 827939 strings, you are looking for a few to get its addresses:

!dumpheap -type System.String

The output should be similar to this:

057f9364 790fd8c4 4116
057fa4b8 790fd8c4 4116
057fb52c 790fd8c4 4116
057fc68c 790fd8c4 4116
057fd700 790fd8c4 4116
0583fef8 790fd8c4 4116

Next you want to start trying to identify who's holding on the objects. That can be done using the following command (I replaced the real Namespaces with XXXXXXXX to keep customer's privacy):

0:000> !gcroot 0583fef8
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.

Scan Thread 27 OSTHread e40
DOMAIN(00102980):HANDLE(WeakLn):945e00:Root:19312e44(System.Web.Hosting.ISAPIAsyncCompletionCallback)->
190c2950(System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6)->
190c3138(System.Web.HttpContext)->
190c340c(System.Web.AspNetSynchronizationContext)->
0cbb80d0(ASP.global_asax)->
01b3a5b4(System.Web.HttpApplicationState)->
01b3a610(System.Collections.ArrayList)->
0a4e4c10(System.Object[])->
05846388(System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry)->
058331c8(XXXXXXXX.Web.UI.WebControls.XXXXXXXX.XXXXXXXX)->
058333c0(XXXXXXXX.Web.UI.WebControls.XXXXXXXX.Statuses)->
058333cc(System.Object[])->
0583fe20(XXXXXXXX.Web.UI.WebControls.XXXXXXXX.Status)->
0583fee0(System.Object[])->
05840f0c(XXXXXXXX.Web.UI.WebControls.XXXXXXXX.Style)->
0583fef8(System.String)

Looks like the code using endlessly Application object sticking strings into it. Once there it's never collected until the end of life of the Application which caused only by recycle.

Step 5 - Find the culprit in the source code

Next is to look in the source code all the usages of assigning strings to Application object. After quick look up we identified the following code that is called recursively:

HttpApplicationState app = HttpContext.Current.Application;           
for (int i=0; i < m_XXXXXX.Count; i++)
{
    string sProcessId = m_XXXXXX[i].SelectSingleNode("@XXXXXX").Value;
    app.Set (GetPrefix() + XXXXX, new XXXXXX(m_XXXXXX[i].OuterXml));
}

Conclusion

I have been using the following approach several times lately and it has proven to be very effective and efficient. It's simple too.

Happy memory leak analysis.

Related Materials

 

This post is made with PracticeThis.com plugin for Windows Live Writer

Comments (2)

  1. gOODiDEA.NET says:

    Web Pimp My Code #1: Element.Storage Things you may not know about jQuery YUI 2.7.0 Released Windows

  2. gOODiDEA says:

    WebPimpMyCode#1:Element.StorageThingsyoumaynotknowaboutjQueryYUI2.7.0Rele…

Skip to main content