.NET Debugging Quick Start

Many a time I am asked how to get a head start with debugging .NET issues – hangs / crashes and most commonly, memory issues. So I assembled this list of resources which helps you get started. In addition, if you are a Premier Support subscriber, you can avail of our most excellent .NET Debugging WorkshopPLUS conducted by my team (Premier Field Engineering).

I hope you find the links useful and if you do I would appreciate some comments and / or you rating this post!

WARNING: Stack unwind information not available. Following frames may be wrong.

I’m sure if you have ever used the WinDbg family of debuggers sometime, you must have seen the above message. What does it really mean? How does it affect you?

Quick Background on stack operation

In x86, the way the stack is built up, the entry point of the function (a.k.a. prolog) generated by the compiler contains some key instructions including one which saves the previous value of the EBP register for that thread on the stack. The next instruction sets the EBP register to point to the current stack pointer (ESP).

What am I talking about? If you don’t know what EBP and ESP are, I recommend you take a quick look at the links in the reference section at the end.

Frame Pointer Omission

So in some cases the compiler may choose to omit the setting of the EBP register and may instead directly use ESP to reference locals and parameters. In such cases the EBP register (a.k.a. ‘frame pointer’) is deemed as ‘omitted’ and the generated code is called FPO (Frame Pointer Omission) code.

In such cases the debugger will typically complain that it cannot unwind the stack based on the EBP (unless it has symbol files which match the module whose function has FPO enabled). Under those cases it will emit the warning which is the title of this post.

Recommendation

  • For builds which have /Oy enabled, it is necessary to have symbol files to successfully reconstruct the stack.
  • In the real world the most common reason to encounter the above message is faulty symbol paths. Check your symbol path.
  • In cases where you can tolerate the overhead of setting up the frame pointer, leave FPO off (/Oy-). See the last reference in my post for something which Windows team has supposedly done regarding this.

References

YADCU – Yet another dump capture utility

The plethora of dump capture tools is amazing and sometimes confusing. But here is one from Mark Russinovich which looks interesting: ProcDump. Some unique capabilities I can see in this tool are things like CPU threshold based triggers, the ability to clone a process so that it is suspended for minimum time when dump is captured, ability to launch another image on the event trigger, etc. Take a look at it, it should add to your debugging toolbox!

Quick Tip: vfbasics!_AVRF_EXCEPTION_LOG_ENTRY symbol not resolved?

I was debugging some issues with the help of Application Verifier and WinDbg. Since I was onsite I did not have any access to Microsoft’s private symbol servers, so I was using the public symbol server (http://msdl.microsoft.com/download/symbols). On executing the !avrf extension command in WinDbg I was presented with the following error message in WinDbg:

***    Your debugger is not using the correct symbols                 ***
***                                                                   ***
***    In order for this command to work properly, your symbol path   ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: vfbasics!_AVRF_EXCEPTION_LOG_ENTRY                ***

It turns out that my WinDbg symbol path was as follows, and due to it pointing just to the public symbol server it was loading public symbols for vfbasics.dll:

0:001> .sympath
Symbol search path is: SRV*c:localsymbols*
http://msdl.microsoft.com/download/symbols
Expanded Symbol search path is: srv*c:localsymbols*http://msdl.microsoft.com/download/symbols
0:001> lml
start             end                 module name
00000000`76dc0000 00000000`76f68000   ntdll      (pdb symbols)          c:localsymbolsntdll.pdbFDAD9EE7D6E44F4F9672ECB401A802192ntdll.pdb
000007fe`f0e50000 000007fe`f0ebe000   verifier   (pdb symbols)          c:localsymbolsverifier.pdb43FCE2D63C4544F9B1C67110EB3406951verifier.pdb
000007fe`f1660000 000007fe`f1693000   vrfcore    (pdb symbols)          c:localsymbolsvrfcore.pdb751D23CCD6504794AF2F18C1E547FE371vrfcore.pdb
000007fe`f28e0000 000007fe`f292a000   vfbasics   (pdb symbols)          c:localsymbolsvfbasics.pdb1ABCDFEFF9F4602A7F055801457A7D61vfbasics.pdb

To resolve the issue, I explicitly pre-pended the path to private symbols for vfbasics (which is c:windowssystem32 or in general %WINDIR%System32:

.sympath c:windowssystem32*SRV*c:localsymbols*http://msdl.microsoft.com/download/symbols

.reload

ld vfbasics

0:001> lml
start             end                 module name
00000000`76dc0000 00000000`76f68000   ntdll      (export symbols)       C:WindowsSYSTEM32ntdll.dll
000007fe`f28e0000 000007fe`f292a000   vfbasics   (private pdb symbols)  C:WindowsSYSTEM32vfbasics.pdb

Then !avrf works just fine!

If you liked this post, please do rate it and try to leave some comments if you can!

Debugging Toolbox

This one is a quickie for an easy reference to most commonly used debugging tools and links. I hope you find it useful, and kindly indicate your feedback on this page by using the comments section or by rating the post!

Debugging Toolbox

 

Tool

Key Usage Scenarios

Download location

WinDbg

Interactive production debugging

Dump analysis

Local kernel debugging

 

http://www.microsoft.com/whdc/devtools/debugging/default.mspx

CDB

Scriptable, low overhead capture of dumps

 

Low overhead interactive debugging

ADPlus

Scripted automation for capturing dumps

DebugDiag

Can setup rules for capturing dumps in an unattended, logged-off scenario

 

Automated analysis of basic crash, hang and leak scenarios

http://www.iis.net

SysInternals Process Monitor

Interactive process and thread level details

 

Handle usage

http://www.sysinternals.net

TLIST

Listing services

Listing process tree

Listing which processes have loaded a module

http://www.microsoft.com/whdc/devtools/debugging/default.mspx

Performance Monitor

Interactive display of performance data on the system (System Monitor)

 

Logged-off unattended capturing of performance logs

Shipped with the OS

Application Verifier

Enable various checks to trap deep rooted issues earlier such as

 

Orphaned critical sections

Heap corruption

Unsafe API usage

Simulate low memory conditions

 

http://msdn.microsoft.com/en-us/library/aa480483.aspx

http://www.microsoft.com/downloads/details.aspx?FamilyID=C4A25AB9-649D-4A1B-B4A7-C9D8B095DF18&displaylang=en

 

PREFast

Static source code analysis to detect potential buffer / stack overruns etc.

http://www.microsoft.com/whdc/devtools/tools/prefast.mspx

XPerf / XPerfInfo

Profiler / tracing of user mode applications

http://msdn.microsoft.com/en-us/library/cc305187.aspx

http://msdn.microsoft.com/en-us/performance/cc825801.aspx

http://blogs.msdn.com/seema/archive/2008/10/08/xperf-a-cpu-sampler-for-silverlight.aspx

 

UserDump

Automated capture of dumps

http://www.microsoft.com/downloads/details.aspx?FamilyID=e089ca41-6a87-40c8-bf69-28ac08570b7e&DisplayLang=en

 

LeakDiag

Leak diagnostics

ftp://ftp.microsoft.com/PSS/Tools/Developer%20Support%20Tools/LeakDiag

 

Native code debugging sites and blogs

http://www.advancedwindowsdebugging.com

http://blogs.msdn.com/ntdebugging

http://www.osronline.com

http://www.alex-ionscu.com

http://www.dumpanalysis.org

http://www.nynaeve.net/

http://blogs.thinktecture.com/ingo/archive/2006/08/05/414674.aspx

 

.NET articles:

http://msdn2.microsoft.com/en-us/library/ms954594.aspx

http://msdn.microsoft.com/msdnmag/issues/03/06/Bugslayer

http://msdn.microsoft.com/msdnmag/issues/05/03/Bugslayer

http://support.microsoft.com/kb/892277

http://msdn.microsoft.com/msdnmag/issues/05/07/Debugging/

http://msdn.microsoft.com/msdnmag/issues/06/11/CLRInsideOut/default.aspx

 

.NET debugging blogs:

http://blogs.msdn.com/yunjin/

http://blogs.msdn.com/tess/

http://blogs.msdn.com/jmstall/

http://blogs.msdn.com/mvstanton/ 

http://blogs.msdn.com/cbrumme/

http://blogs.msdn.com/maoni/

http://blogs.msdn.com/toddca/

http://blogs.msdn.com/suzcook/

http://www.debugtricks.com

Symbols and symbol files

Some of us are unaware of the relevance of proper symbols files. The symbol file is a deliverable just like the binary. Though it is not required for running the application, it is very important to debug issues which arise in production.


 


What are symbols and symbol files?


Symbols contain the mapping between the compiler-generated machine code and your source code. They help the debugger to “understand” the addresses of functions, parameters and variables and map them back to source code references. Without a proper symbol file, the debugger may present disassembly output which looks like this:


 


            call YourApp+0x11c0d (00411c0d)


 


With proper symbol resolution, the debugger can correctly interpret this machine code as the equivalent function name:


 


            call YourApp!CSampleDlg::OnBtnClicked


 


In the Visual Studio family of products, we use the PDB (Program Database) format for encoding the symbolic information into a file. Given below are the steps to generate the symbol files for various product versions. Please ensure that the PDB files generated are of the same code base as the service EXE/DLLs that are actually used in the testing. Everytime your code is rebuilt, the PDB is re-generated and has to be maintained in sync with your code base and binaries.


 


Private vs. Public symbols


Public symbol files are what you get from Microsoft for the Windows OS and other products. You can obtain public symbols for Microsoft products either by referencing the public symbol server at http://msdl.microsoft.com/download/symbols, or by downloading appropriate symbol packages from http://www.microsoft.com/whdc/devtools/debugging/default.mspx. Public symbols normally just contain:




  • Mapping from addresses to function names



  • Frame pointer omission (FPO) records


Private symbol files are the default kind of files generated by the Visual Studio linkers. These files contain the following details in addition to what the public symbols contain:




  • Line numbers and source file information



  • Parameter names and data types



  • Local variable names and data types


Normally you would want private symbol files for any detailed debugging of your own application. However, it may not be a very good idea to share your private PDB files with your end customers, since some of the information in the private PDB can be viewed as intellectual property.


 


To generate stripped / public PDB files with Visual C++, use the information in http://msdn2.microsoft.com/en-US/library/y87kw2fd.aspx for steps.


 


How to generate symbol files for Visual C++.NET 7/7.1/8.0


In these versions of VC++, a symbol file should automatically be created if you are compiling a debug build. If you are compiling a release build, then you will need to check the steps below to ensure that linker will create symbols. Please perform the symbol generation steps for each and every module in your application. If it loads other DLLs then these steps need to be done for those DLLs also.




  • Open the source code for your project.



  • Open the properties page for the project from Solution Explorer



  • View the Configuration Properties->Linker->Debugging node



  • Make sure “Generate Debug Info” is set to YES



  • Also check that the “Generate Program Database File” is set to a non-blank path



  • Rebuild the project to generate the PDB file


How to generate symbol files for Visual C#.NET 7/7.1/8.0


In VC#, a symbol file should automatically be created if you are compiling a debug build. If you are compiling a release build, the steps are given below.




  • Open the source code for your project.



  • Open the properties page for the project from Solution Explorer



  • View Configuration Properties->Build node



  • Make sure “Generate Debugging Information” is set to True.



  • Rebuild the project to generate the PDB file


How to generate symbol files for Visual Basic.NET 7/7.1/8.0


In VB.NET, a symbol file should automatically be created if you are compiling a debug build. If you are compiling a release build, the steps are given below.




  • Open the source code for your project.



  • Open the properties page for the project from Solution Explorer



  • View Configuration Properties->Build node



  • Make sure “Generate Debugging Information” is checked.



  • Rebuild the project to generate the PDB file


How to generate symbol files for Visual C++ 6.0


Important note: if you are still using VC++ 6.0, be aware that Microsoft Support will not be in a position to support you on issues concerning the product. That is because the end of support lifecycle has been reached for this version of VC++. For more information, review the details at http://support.microsoft.com/lifecycle and plan to upgrade to higher versions of VC++.NET. The information below is provided for reference only, and should not be interpreted as a sign of support!


 


In VC++ 6.0, a symbol file should automatically be created if you are compiling a debug build. If you are compiling a release build, then you will need to follow these steps to create symbols. Please perform the symbol generation steps for each and every module in your application. If it loads other DLLs then these steps need to be done for those DLLs also.




  • Open the source code for your project.



  • Select Settings from the Project menu.



  • In the Link tab, select General from the Category drop-down.



  • Check Generate Debug Info



  • In the C/C++ tab, select General from the Category drop-down.



  • Select Program Database from the Debug Info drop-down.



  • Link with /RELEASE to get a non-zero checksum.



  • Go to the Category drop-down & select Listing Files.



  • In the Listing File drop-down, select ‘Assembly, Machine Code, and Source’.



  • Rebuild your project.


How to generate symbol files for Visual Basic 6.0


In VB6, symbols are not created by default. To create symbols for your VB6 applications, follow these steps:




  • Open the source code for your project.



  • Select “<projectname> Properties” from the Project menu.



  • In the Compile tab, check Create Symbolic Debug Info and select No Optimizations



  • Recompile. This will create a .PDB file (the symbols) in the same directory as your .DLL / .EXE / .OCX file.



  • Note: This does *not* break binary compatibility.