Using DebugDiag's LeakTrack with ProcDump's Reflected Process Dumps

  • Article

DebugDiag and Procdump are two of my favorite tools.  They're both incredibly useful and I rely on them heavily to gather MemoryDumps when debugging production issues.  They have a myriad of overlapping features, but they each have (at least) one feature that makes them stand apart from each other:

  • DebugDiag has the LeakTrack feature.  This lets users inject a module that tracks callstacks for allocations and stores them to heuristically identify the cause of native memory leaks in an application.  I use the feature whenever I have a native memory leak and find it indispensable.
  • ProcDump has the ability to reflect a process before creating a memory dump.  This substantially reduces the amount of time the process is interrupted when a memory dump is taken.  When dealing with production issues, this is an extremely efficacious feature.

The catch is, DebugDiag currently can't create reflected process dumps, and ProcDump doesn't have a way to inject the LeakTrack dll to track allocations.  We can get around this by working with both tools.  Assuming both tools are installed on the server with a leaky application, the process is as follows:

  1. Identify the application that has a native memory leak: https://blogs.microsoft.co.il/blogs/sasha/archive/2008/07/13/is-it-a-managed-or-a-native-memory-leak.aspx
  2. Use DebugDiag to inject the LeakTrack dll and begin tracking allocations
  3. Set up ProcDump to create a dump of the reflected process when a memory threshold is breached

Step 1 is easy and documented elsewhere.  Step 2 is easy if we have direct GUI access to the machine.  We can simply go to the Processes tab in DebugDiag, right click the process, and chose "Start Monitory for Leaks."  For the sake of argument, let's say we want to script the whole process.  We can do that by scripting DebugDiag and ProcDump to do the individual tasks we've set out for them.  Once we have the PID of the troubled process, we can use a script to inject the LeakTrack dll into the process.  The following vbs script will, when used by DebugDiag, start the native leak tracking process:

Sub Debugger_OnInitialBreakpoint()
   Debugger.Write "Injecting LeakTrack" & vbLf
   Debugger.InjectLeakTrack
   Debugger.Write "Detaching2" & vbLf
   Debugger.Detach
End Sub

The above code can be saved into a vbs file.  For this post we'll assume it was saved to c:\scripts\InjectLeakTrack.vbs.  We'll also assume our PID was 1234

With the PID known and the script created, we can launch DebugDiag from a command line as such (if you installed DebugDiag elsewhere, you'll have to adjust the install location): C:\PROGRA~1\DEBUGD~1\DbgHost.exe -script "c:\scripts\InjectLeakTrack.vbs" -attach 1234

This will cause DbgHost, the debugger process DebugDiag uses, to attach to the process with PID 1234 and run the script we provided.  The script simply injects the LeakTrack dll and detaches the debugger.  Now that LeakTrack is injected, we can launch ProcDump and tell it to reflect the process and create a memory dump when we've crossed a memory threshold.  Let's say we know our application has leaked if it ever crosses 600MB.  We can kick off ProcDump as follows: procdump 1234 C:\Dumps\Over600.dmp -accepteula -r -m 600 -ma

ProcDump will then monitor the process and create a reflected process dump when the committed bytes of the process breach 600 MB.  This memory dump can be analyzed by DebugDiag on the Advanced Analysis tab, and the offending callstacks can be identified.

 

-Jarrod