Post Mortem Debugging Overview

Throughout my career in software development I have come across issues that are tough to capture or only seem to occur in production. Each time I encountered these situations I would say to myself, there has to be something better.

Post Mortem debugging allows you, through the use of several tools, to take a memory dump when an issue occurs and later analyze it offline. Assuming that there are other developers out there with the same issue I have decided to write a series of posts outlining how to do just this.

I will break this series into several steps:

  1. Post Mortem Debugging Overview (this posting)
  2. How to collect a memory dump using ADPLUS and DebugDiag
  3. How to setup WinDBG
  4. Basic analysis of an unmanaged memory dump (C++)
  5. Basic analysis of a managed memory dump (.NET)

A memory dump is a snapshot of a portion of memory at an instance in time.

 

When capturing a memory dump there are options used to identify the amount of memory to be captured. The dumps for user mode typically are referred to either mini or full memory dumps. Ironically a mini memory dump with specific flags set can actually result in a larger file being generated.

An important thing to understand is that memory dumps are point in time. Think of these as being similar to a photograph. The photograph does not identify the circumstances that led up to it and as such it can be misinterpreted.

Creating a memory dump is typically done using ADPLUS or DebugDiag. After the memory dump is captured WinDBG comes into play for the final analysis.

Symbols are an important concept to understand when analyzing a memory dump. In simple terms a symbol is a name assigned to a location in memory. In the case of private symbols they include function names, types, parameters, variables and line numbers. Public symbols do not include the line numbers and local variables in an effort to protect intellectual property.

Symbol files have a dbg and pdb extension, with that latter being the newer format. The debugger determines the proper symbol file by analyzing the PE header which contains in the case of a PDB file the PDB signature and age. Every time code is compiled it will generate a new signature and age stamp making any older pdb’s obsolete, even if nothing is changed. This means that the symbol files in use should be the ones that were created from the build being debugged.

In the next posting I will outline how to collect a memory dump.