I already touched this topic a while ago, but since it’s an important part of the debugging process (and your debugging techniques may vary a lot, depending if you have or not good symbols for your dump) I though would be a good idea to give some more details. And just to jump start on the topic, here’s something I learnt a while ago after wasting a few hours typing commands and looking at inconsistent results… debugging with the wrong symbols could be much worse than debugging with no symbols at all.
What are symbols?
You can think of symbol files basically as small databases, files which contain source line information, data types, variables, functions and everything else needed to provide names for all lines of code, instead of hexadecimal addresses. They usually have .pdb (or .dbg) extension, and are matched with the actual executable code using an internal timestamp, so it’s very important to generate symbols every time you build the application, also for release builds. If you’ll ever have a problem with your live application and you’ll need to debug it, and if you’ll not have the matching symbols (matching means the symbols obtained from the same exact build of the dlls you put in production) you could be in troubles… Building once again the application to create the symbol files, even without changing your code does not really help because the timestamp will not match, and WinDbg will complain about missing symbols anyway…
Note that in Visual Studio if you set "Release" in the "Standard" toolbar, the "Generate debugging information" checkbox will automatically be cleared, so remember to go to the Project properties and flag it again before rebuilding the project.
Since in ASP.NET 2.0 we have a new default page architecture, there is no real need to create the .pdb files unless you’re using the "Web Application Project" template which comes with the Service Pack 1 for Visual Studio 2005, in which case you can find it from the project properties, "Compile" tab.
The same applies for Visual Studio 2008.
How can we use symbols?
When you open a dump within WinDbg and you type in a command for example to inspect the call stack, the debugger will start looking for matching symbols to show you an output as detailed as possible: but how does it decide where to look for those symbols? It will use the path(s) specified in the "Symbol Search Path" dialog you can find under the File menu (CTRL+S is the keyboard shortcut.
Here you can specify the symbol servers where WinDbg can download the symbols from, and of course you can use has many server more than one server at a time; WinDbg will simply access those servers in the same order you put in the search path, and it goes through the end of the list until it finds a match.
Microsoft has a public symbol server accessible through the Internet which stores public .pdb files for almost all of our products: http://msdl.microsoft.com/download/symbols.
I work a lot with WinDbg and memory dumps in my daily job, also with my laptop when not connected to the Internet or corporate network, so I need to be able to debug while offline and in any case I don’t want to waste time waiting for the tool to download the same symbols over and over again, through one dump to the other… for this reason it’s also possible to create a local symbol cache (also referred as downstream store) on your hard disk. When looking for symbols, WinDbg will first of all check your local cache and if the match is found there no additional check is done against the other symbol servers, while if the match is not found WinDbg goes on as usual until it finds the right match; in this case it downloads the .pdb and stores it in your local symbol cache, so that next time it will be readily available for you.
The symbol path is a string composed of multiple directory paths, separated by semicolons. For each directory in the symbol path, the debugger will look in three directories: for instance, if the symbol path includes the directory c:\MyDir, and the debugger is looking for symbol information for a dll, the debugger will first look in c:\MyDir\symbols\dll, then in c:\MyDir\dll, and finally in c:\MyDir. It will repeat this for each directory in the symbol path. Finally, it will look in the current directory, and then the current directory with \dll appended to it. (The debugger will append dll, exe, or sys, depending on what binaries it is debugging.)
Here is a sample symbol path:
The above is actually the symbol path (a bit simplified) I use on my machines: as you can see I have a local cache in C:\Symbols; if I’ve never downloaded a particular symbol before, WinDbg does to an internal share were we have full symbols, and if still unsuccessful I finally give a try to the public Microsoft symbol server on the Internet. If you include two asterisks in a row where a downstream store would normally be specified, then the default downstream store is used. This store will be located in the sym subdirectory of the home directory. The home directory defaults to the debugger installation directory; this can be changed by using the !homedir extension. If the DownstreamStore parameter is omitted and no extra asterisk is included (i,e. if you use srv with exactly one asterisk or symsrv with exactly two asterisks) then no downstream store will be created and the debugger will load all symbol files directly from the server, without caching them locally. Note that If you are accessing symbols from an HTTP or HTTPS site, or if the symbol store uses compressed files, a downstream store is always used. If no downstream store is specified, one will be created in the sym subdirectory of the home directory.
The symbol server does not have to be the only entry in the symbol path. If the symbol path consists of multiple entries, the debugger checks each entry for the needed symbols; moreover the symbol path can contain several directories or symbol servers, separated by semicolons. This allows you to locate symbols from multiple locations (or even multiple symbol servers). If a binary has a mismatched symbol file, the debugger cannot locate it using the symbol server because it checks only for the exact parameters. However, the debugger may find a mismatched symbol file with the correct name, using the traditional symbol path, and successfully load it; in this case it’s important to know if our symbols matches (see the next topic).
You can set the symbol path in advance once for all within WinDbg: open an empty instance, press CTRL+S, type in the path, clock "Ok" on the dialog and close WinDbg, accepting to save the workspace if prompted to do so (next time you’ll open WinDbg the value will still be there). Or you can use the .sympath command within WinDbg with a dump open.
Another option is to set the system wide variable _NT_SYMBOL_PATH (the syntax is still the same), used by debuggers like WinDbg or also from adplus directly when it captures the dump.
The same principle applies to the Visual Studio debugger (also have a look at the article http://support.microsoft.com/kb/311503/en-us):
How can I check if my symbols matches?
Looking at a call stack sometimes it’s clear you’re having a problem with unmatched symbols because WinDbg tells you something like:
Unfortunately could happen to not be so lucky, and you’ll find yourself wondering if the stack you are looking at is genuine or there are some small (or maybe even not so small) inconsistencies which may lead you down to a completely wrong path. In such cases, you can first of all use the lm command to find which .pdb files have been loaded:
As you can see in the example above, two symbols were loaded (for kernel32.dll and ntdll.dll), while shell32.dll and user32.dll were not part of the stack analyzed, so WinDbg has not loaded yet (deferred) their symbols. A bad match will look like the following:
Notice the "M" highlighted in red (could also be a "#" pound sign)? That stands or "mismatch", and indicates there is a problem with that particular module (search for "Symbol Status Abbreviations" in WinDbg help for further details). Alternatively you can use the !sym noisy and .reload command to reload symbols verbosely to have a detailed output. Look for "Symbols files and paths – Overivew" in WinDbg help for more details.
Trick: you have the right symbol, but WinDbg does not matches it anyway…
I’m not sure why this happens, and especially why I had this problem only with ntdll.dll (and its .pdb): I was not able to get the proper stack even with a matching symbol (and I checked more than once to be really sure)… until I got the idea to delete the ntdll.pdb folder in my local cache (if you have a dump open you must first unload the symbol from WinDbg or the file will be locked: use the .reload /u <module_name> command), then run a .reload /f <module_name> (/f forces immediate symbol load) and let WinDbg to download it again… this usually does the trick and I finally get the correct stack.
Debugging without symbols?
It’s not impossible, but it’s harder than debugging with matching symbols; the main difference is that you’ll not be able to see method names, variable names etc… and generally speaking the stack will be less easily readable. To give you a quick example, here is an excerpt of the stack of a very simple application I wrote for test (it has just a button which sets the text of a label to the DateTime.Current.ToString()):
with matching symbols:
The difference is quite obvious… The WinDbg help file also gives a few hints:
- To figure out what the addresses mean, you’ll need a computer which matches the one with the error. It should have the same platform (x86, Intel Itanium, or x64) and be loaded with the same version of Windows
- When you have the computer configured, copy the user-mode symbols and the binaries you want to debug onto the new machine
- Start CDB or WinDbg on the symbol-less machine
- If you don’t know which application failed on the symbol-less machine, issue an | (Process Status) command. If that doesn’t give you a name, break into KD on the symbol-less machine and do a !process 0 0, looking for the process ID given by the CDB command
- When you have the two debuggers set up — one with symbols which hasn’t hit the error, and one which has hit the error but is without symbols — issue a k (Display Stack Backtrace) command on the symbol-less machine
- On the machine with symbols, issue a u (Unassemble) command for each address given on the symbol-less stack. This will give you the stack trace for the error on the symbol-less machine
- By looking at a stack trace you can see the module and function names involved in the call
I got symbols from my customer: what should I do now?
The easiest thing you could do is use symstore.exe (you’ll find it in WinDbg/adplus installation folder) with a command like the following:
symstore add /f c:\temp\SymbolsTest\Bin\*.pdb /s c:\symbols /t "Symbols Test"
Let’s have a quick look at the syntax:
- The "add" keyword is quite self explanatory
- /f tells symstore which is the file you want to add; note you can use wildcards, so you can add multiple files at once
- /s is the path to your symbols store (this will most likely be your local cache, or your shared symbol server)
- /t a required description for the symbol to store
Symstore will create a structure similar to the following:
Also note the "0000Admin" folder created by symstore, which contains one file for each transaction (every "add" or "delete" operation is recorded as a transaction), as well as the logs server.txt and history.txt; the former contains a list of all transactions currently on the server, while the latter contains a chronological history of all transactions run on the machine. For further information you can see the "Using SymStore" topic in WinDbg help (debugger.chm).
While it is possible to debug without symbols (this is true especially for managed code), remember that your life could be much easier if you (and your customers) will take care of your symbols and will generate them every time the application will be rebuilt, also in release mode.
I deliberately simplified the argument (I just wanted to share what I learnt in my daily job and give some quick tips to get started), much more could be said and if you’re interested I encourage you to read the "Symbols" section in the WinDbg help, or search the Internet where you’ll find good blog posts/articles on this subject