I am going to start of talking about the debugging tools that I use to generate memory dumps when customers have hang or crash. I am mainly going to talk about debugging 32bit apps for now. The debugging tools can be downloaded from the following link. One cool thing about these debugging tools is you can do xcopy deployment. Sometimes customers do not want to install anything on their production boxes. In such cases, you can install the debugging tools on a different machine and then copy the folder over to the target machine and the debugging tools will work just fine.
The debugging tools come with tools such as windbg (I can’t imagine doing my job without it), adplus, cdb, gflags and a bunch of other tools.
I will talk about how to use them later as we discuss hangs and crashes.
.One of the most important things before getting dumps is that you need to generate symbol files (pdb) for your application and any other dlls that you may have. Without symbol files the dump will not be of much use. When we try to look at call stacks, it will only show raw addresses and it will be almost impossible to get meaningful information from dumps. If you can get a debug version of the app the fail, that will give us more information because there will be less optimization in the code and it will be easier to review dumps. In VS.net, when you build a debug version of the app, the pdb file will be generated in the same folder as the exe or dll. For VS.net, in release mode, you need to make sure that Under Project->Properties, Configuration Properties->Build, “Generate Debug Information” should be set to true For VC 6.0, you need to make sure that the following options are set.
a) Go to Project->Settings->C/C++ tab and under “Debug Info” make sure that Program Database is selected.
b) Then go to the Link tab and under “Category” choose “Debug” and under “Debug Info” frame make sure that “Debug Info” is selected with “Microsoft Format” and uncheck “Separate Types”.
c) Now build the application and this will generate the corresponding symbol file.
You can follow the above steps in release mode as well and you can generate the symbol files.
For VB 6.0 applications go to Project->Properties and “Compile” tab. In compile tab, choose “Compile to Native Code”, select “No Optimization” and check “Create Symbolic Debug Info”. When you build your app now, it will generate the symbol file.
The other important thing is, once a symbol file is generated from a build, you would need to deploy that version of the application on the target machine. Even if you do not change the code and if you rebuild your application, the symbol files will not match up and the dump will not be useful. So whenever you compile your app and get a new symbol file, the application should be redeployed. This step is very important because, dumps will be very huge and if you transfer it from other customer sites, the last thing you want is a symbol mismatch.
You can use the “SymChk” tool that is part of the “Debugging tools” to verify that you have the correct symbols. The help that comes as part of the debugging tools has examples listed.
I am mainly going to be talking about user mode crashes. Application crash typically happens when we try to read from memory location that has already been freed or if we try to write to a memory location whose address is invalid. When a crash occurs if dr Watson is installed as the default post mortem debugger, it will generate a log file and also a dump file. Sometime we see the following error message as well.
"The instruction at the (address ), refernce memory at (address).The memory couldn't be written."
Some times the application might just disappear.
In these situations that data that we collect is typically user mode memory dumps of the process when the crash occurred and then we analyze it. In order to generate crash dumps, here the steps that you need to follow after having generated symbol files and deploying that version of the app on to target machines
1)Start your application.
2) Go to the folder where you installed the debugging tools through command prompt
3) Then type the following command
adplus -crash -pn YourApp.exe ( When you run adplus the first time, you might be prompted to make cscript your default script interpreter, please say yes to it, Also ignore the warning about NT symbol path)
4)Re-create the application crash and once the crash occurs there will be a folder generated in the folder where you installed debugging tools and it will look like
In this folder, you will mostly find files with extension .dmp. and bunch of other folders and files.. In the case of crash, you will typically find a file with a long name and it will have “2nc_Chance_AV” as part of it. This is the dump file when the Second Chance Exceptions was thrown. Whenever an exception occurs, you get a first chance exception and if that is not handled, then you will get a second chance exception and if there is no handler, it will crash the application. Since crashes are unexpected, in most cases we will get a second chance exception.
So when you call Microsoft support to troubleshoot crashes, this is the data that we typically collect. So if you already have it when you call support, the time it takes to go through the process of getting crash dumps and symbol files will be eliminated and will greatly increase the speed at which we can analyze the problem.
I shall stop here for today. I will talk about the data required for Hangs at a later time. If any of you want to add to this or if you have any questions please feel free to post your comments.
This posting is provided "AS IS" with no warranties, and confers no rights. OR if you wish to include a script sample in your post please add "Use of included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm