How to debug crashes and hangs


At my job on the C# IDE QA team I’ve learned some useful things about debugging in Visual Studio, which I’d like to summarize in this post. Although the screenshots were made using Visual Studio 2008 SP1, this pretty much applies to other versions of VS as well.


Rich debugging support


When you develop your C# application and hit F5, the target process (your program) gets started, and then the Visual Studio process attaches the debugger to the process where your code is running. This way, you can break into the debugger and VS will provide you with all sorts of rich debugging support – current statement highlighting, call stack, watches, locals, immediate window, Edit-and-Continue and so on.


More importantly, if your application throws an exception or crashes, the debugger will intercept that and provide your with all the information about the exception.


As a side note, in Visual Studio there is a way to run your code without attaching the debugger – the shortcut is Ctrl+F5. Try throwing an exception in your code when using F5 and Ctrl+F5 to feel the difference.


throw null;


By the way, my favorite way to artificially throw exceptions is throw null; I just love the fact that it throws a NullReferenceException because it can’t find the exception object and nevertheless does exactly what I want it to do :)


Crashes and the Watson dialog


What if a program crashes or throws an exception, which you don’t have source code for? Moreover, you didn’t start the program using F5, but the operating system launched the process. I remember that before coming to Microsoft, the only thing I could do about some application crashing was to express my disappointment about the fact (usually in Russian). Now I don’t feel helpless anymore, because I’ve learned a couple of tricks. As an example for this we’ll crash Visual Studio itself and then debug the crash.


How to crash Visual Studio?


Viacheslav Ivanov reported an interesting crashing bug in our language service recently. Save all your work and then paste this code in a C# Console Application and change ‘object’ to ‘int’:

using System;

static class Program
{
static void Main()
{
ITest<object> test;
test.Test((object /* and now change the argument type to “int” */ i) => { });
}
}

public interface ITest<T> { }

public static class Extensions
{
public static void Test<T, I>(this ITest<T> test, Action<ITest<I>> action) { }
}


What you will see is the Watson dialog:


image


Given this chance, I’d highly recommend everyone to click “Send Error Report” if you ever see this dialog for Visual Studio. Many people “Don’t Send” and frankly I don’t understand why not. There is no personal information being sent, and we don’t want your personal information anyway, honest. What we want is a call-stack and minidump, if possible, so if you want us to fix the product to make it more stable in the future, you will greatly help us if you send us the Error Report. By the way, we usually fix most (if not all) crashes that come in through this dialog, so the chances that we’ll fix the crash you report using the dialog are actually pretty high. For example, we’ve already fixed the bug mentioned above and it works just fine in current builds.


Attaching a debugger


So what can you do if an application crashes or hangs? You can attach the debugger to a running process, even if it has already crashed. The code is still being executed (the main thread of the crashed application is usually pumping messages for the error dialog). You can either choose “Debug” on the Watson dialog, or (what I usually do) is start a new instance of Visual Studio myself and attach to the process manually, without dismissing the Watson dialog.


Note: if the debuggee process crashes and you attach the debugger after the fact, you’ll have to manually break into the debugger by pushing the “Pause” button. If the debugger was already attached at the moment of the crash, then it will offer you to break or continue.


You can attach the Visual Studio debugger to a running process by choosing Tools | Attach To Process (Ctrl+Alt+P):


image


You will see the Attach to process dialog:


image Interesting things to note here are the Attach to: selection. Since Visual Studio is a mixed-mode managed/native application, to get call stacks for both managed and native parts, you’d want to attach to both of them (by clicking Select…):


image


If you select both Managed and Native, you’ll get richer debugging information about your callstacks – this is recommended.



Note: if you want to enable mixed-mode debugging (managed+native) for the application that you have source code for, go to project properties of the startup project, and on the Debug tag select “Enable unmanaged code debugging”. Then the debugger will automatically attach using mixed-mode.


Finally, select the process from the list which you’d like to attach to (in our example, it will be devenv.exe) and click Attach. Notice that the process of the debugger itself is not shown in the list, that’s why you don’t see two devenv.exe in the list.


Remote Debugging


What many people don’t know is that VS can be used to debug a process running on another machine on the network. To do this, you just need to start the Visual Studio Remote Debugging Monitor on the same machine with the process you want to debug:


image


The remote debugging monitor will listen to debugger connections on the other machine and you’ll be able to attach the debugger using the Transport and Qualifier fields on the Attach to process dialog:


image


You can find more detailed information about Remote Debugging in MSDN and on the internet.


Set Enable Just My Code to false


One very important option in Visual Studio is “Enable Just My Code”, which is set to true by default. To be able to see more information on the callstacks instead of just “Non-user code”, you need to go to Tools | Options and disable this option:


image


I usually do this right after I install Visual Studio, so that I can always debug into “not my code”.


Other interesting options on this page are:



  • Enable .NET Framework source stepping – in case you’d like to step into the .NET framework source code

  • Enable Source Server support

  • Step over properties and operators (Managed only) – won’t step in to property getters, operators, etc. – this is new in VS 2008 SP1

  • Require source files to exactly match the original version – in case you don’t have the source files for the exact .pdb symbol file, but still have a close version of the source code

Break on first chance exceptions


One very useful option in the debugger is the ability to break whenever a first-chance exception is being thrown. A first-chance exception is an exception that might be caught by the program itself later in some surrounding catch block. First-chance exceptions are usually non-fatal and handled (or swallowed) by the user, so they might not even be visible to the end user during normal execution. However, if a first-chance exception is not handled in the code and bubbles up to the CLR/OS, then it becomes a crash.


So, to break on first-chance exceptions, you can go to Debug | Exceptions to invoke the Exceptions dialog:


image


Here you can put a checkmark on Common Language Runtime Exceptions for the debugger to break every time a managed exception is thrown. This way you will see more hidden exceptions, some of them originating deep in the .NET framework class library. Sometimes there are so much first-chance exceptions that you can become overwhelmed, but they are incredibly useful to get to the root cause of the problem, because the debugger will show exactly where the exception originated preserving the original surrounding context. Another great advantage of breaking on first-chance exceptions is that the call stack is not unwound yet and the problem frame is still on the stack.


Debugging tool windows


OK, so now that we know how to attach a debugger and how to set the debugger options, let’s see what happens after we’ve attached a debugger. For our exercise, you can open two instances of Visual Studio, attach the second instance’s debugger to the first one, and then crash the first one using the lambda-expression crash code above. Instead of the Watson dialog on the debuggee process, you will see the following window in the debugger:


image


Now you have the chance to break and see where the exception happened. Continue is useful if the exception is actually a first-chance exception and you’d like to pass it on to user code to handle it. Let’s hit Break and examine what tool windows are available under debugger.


Processes window


All the debugger windows are available from menu Debug | Windows. The Processes window shows the list of processes that the debugger is currently attached to. A nice trick is that you can actually attach to multiple processes at the same time. Then you can use this window to switch the “current” debuggee process and all other tool windows will update to show the content of the “current” process.


image


Note: on our team, we use this window a lot, because our tests run out-of-process. Our test process starts Visual Studio in a separate process and automates it using DTE, remoting and other technologies. Once there is a failure in the test, it’s useful to attach to both the test process and the Visual Studio process under test. The most fun part is when I was debugging a crash in the debugger, and attached the debugger to the debugger process that is attached to some other process. Now if the debugger debugger crashes on you on the same bug, then you’re in trouble 😉 Sometimes I definitely should blog more about the fun debugging stories from my day-to-day job. But I digress.


Threads


As we all know, processes have multiple threads. If you break into a debuggee process, you will most likely end up with a list of threads that were active at the moment when you did break. Main thread is the green one – this is the UI thread of your application (in our example, Visual Studio). Main thread executes the application message loop, and is pumping the windows messages for the application’s UI. If a message box or a dialog is shown, you will see this dialog’s message loop on the main thread.


image


To switch threads, double-click the thread name that you’re interested in. In most of the cases you’ll be interested in the main thread. But if you start your own threads, give them a name so that they are easy to find in the threads list.


Call stack


Every thread has a call stack. Call stack is probably the most important thing you’d like to know about a crash – what was the sequence of function calls that lead to a crash? If the program is hanging, you’d like to know what function is it hanging in and how did it get there. Oftentimes by just glancing at a callstack you immediately know what’s going on. “Is this callstack familiar?” or “Who is on the callstack?” is probably the question we ask most often during the bug triage process.


Anyway, here’s the call stack window:


image


In our example we see that the C# language service module is on the stack (.dlls and .exes are called “modules” in the debugging world).


However instead of function names from cslangsvc.dll we see the addresses of the procedures in memory. This is because the symbols for the cslangsvc.dll module are not loaded. We’ll look into how to load symbols in a moment.


Modules


The Modules window shows a list of .dlls and .exes loaded into the debuggee process:


image


There are multiple ways to load symbols for a given module. You can right-click on a module for a list of options:


image


Symbols for modules are stored in .pdb files and are produced with every debug build of the binary. pdb files contain mapping from compiled binaries back to the original source code, so that the debugger can display rich information (function names, source code locations, etc.) for the binary being debugged. Without symbols, debugging is only possible at the assembly level and registers window, you can’t map the binaries back to the source code.


Symbols


A very useful dialog is the Tools | Options | Debugging | Symbols:


image


Here you can set paths where to look for the .pdb files. Normally, the .pdb files will be directly next to the binaries, in which case they are usually found and loaded automatically. Loading symbols takes some time, so Visual Studio supports caching symbol files to some directory. Also, if you don’t want all the symbols for all the binaries loaded (it can take a while), you can check the checkbox “Search the above locations only when symbols are loaded manually”. Load symbols from Microsoft symbol servers provides symbols for Microsoft products, such as Windows and Visual Studio. You can load symbols from here, or also from the Modules or Call Stack windows, by right-clicking on a module and choosing Load Symbols From. Since we’re debugging into Visual Studio, the symbols are located on the Microsoft Symbols servers:


image


When we load public symbols, the following little dialog is showing:


image


After we’ve loaded the symbols for cslangsvc.dll we notice that the Call Stack window is now much more informative:


image


Given this call stack, anyone of our developers will now easily be able to pinpoint the problem. In our example we see that the problem happens when we try to show the Smart Tag for the Generate Method refactoring:


image


As you see, there are more modules for which the symbols haven’t been loaded yet. You can load the symbols for those modules by right-clicking their stack frame and choosing Load Symbols From. Loading all the symbols is usually recommended for more detailed information.


That is why the call stack is so precious during bug reports. To save the call stack, you can Ctrl+A and Ctrl+C to copy all the stack into the clipboard. When you click Send Error Report in the Watson dialog, the call stack is included in the error report.


Also you see that the call stack is not very useful without the symbols – it’s the symbols + the callstack that provide rich information about the crash.


Minidump


Another very useful information that you can provide about the crashed process is the memory dump (heap dump, minidump) – this is basically a snapshot of the memory state of the crashed process. To create a minidump, go to Debug | Save Dump As. Visual Studio will offer you to save the dump to a location on disk, and an option to just save the Minidump or Minidump with Heap:


image


Minidump with Heap will save more information than just the minidump, but will take a whole lot more disk space (for Visual Studio – possibly hundreds of megabytes – depending on the process working set size during the crash).


Those are the three components that we usually send to our developers during crash bug reporting:



  1. Call stack

  2. Symbols

  3. Minidump with heap

In most cases, they are able to understand the problem given this information. A list of repro steps is usually even better, because they can reproduce the problem themselves and enable first-chance exceptions to break early into the problem area.


Debugging hangs


If an application hangs, you can attach the debugger to it and hit break to see what thread and what call stack blocks execution. Usually looking at the call stack can give you some clues as to what is hanging the application. Debugging hangs is no different than debugging crashes.


Microsoft shares customer pain


Finally, I’d recommend everyone to watch this video:




kick it on DotNetKicks.com

Comments (40)

  1. N.P. says:

    I would like to know though why when debugging C++ the Quick Watch and Immediate windows many times do not execute what I type into them. For example, I have a C++ map and so to get the first item I type myMap[0] in the Immediate or the Quick Watch window. And instead of getting the first item, I get "Error: The overloaded operator is not supported" or something similar. Half and more of the times I use the Watch windows or the Immediate window to debug like this I get errors. Why is this the case and if VS can evaluate more complex expressions while debugging why can’t it evaluate simple ones. The same problem I have when creating a filter for a break-point. I might want to break only if a specific variable evaluates to something. Well, many times not even the == operator is recognized making the break-point filters (you know the Break on Condition) feature useless in many cases.

  2. You’ve been kicked (a good thing) – Trackback from DotNetKicks.com

  3. Kirill Osenkov says:

    Hi N.P.

    this looks pretty much like a bug – can you please file a new bug under http://connect.microsoft.com/visualstudio/feedback – someone from the C++ or Debugger team will help you out.

    Thanks,

    Kirill

  4. Ivan Tarapov says:

    Hello, Kirill :)

    Nice post. I shuddered for a moment when I saw your first screenshot :)

    I was wondering – which version of VS you use to do all those things described here?

    Is there any good trying to debug a crashed 3rd party (not sure if it’s the right term) app that you don’t have the source code or pdb for? Like windows explorer that keeps crashing on me from time to time. I tried to find the culprit shell extension (or something like that) and all I could get was call stack (not very useful since it contained calls from kernel32.dll and some unresolved calls represented by adresses) and a bunch of asm code that I’m not very good debugging at :)

  5. greg says:

    Our offshore team managed to crash VS by creating a constant outside the scope of any member functions that got its value from a web.config file.  This crashes VS when you attempt to design the web page since there is no runtime context (i.e., no config file to read).  I do not know why one would use a function call which might fail to set a contant’s value.

    Here’s the way we found the problem:

    1. Load VS

    2. Load project in VS

    3. Load second copy of VS and debug by attaching to the first copy of VS.  Set debugging to break on all exceptions.

    4. Go to the first copy of VS and attempt to load the bad web page in the designer.

    5. This will throw an exception and break in the second VS’s debugger.

  6. Kirill Osenkov says:

    Hi Ivan :)

    ha ha, I know this dialog has a horror-movie surprise effect because it brings in certain negative associations :)

    This post applies to Visual Studio 2008 Service Pack 1, but is pretty much the same for VS 2005 and VS 2008 as well.

    You can totally attach the debugger to explorer.exe and load symbols for Windows libraries such as kernel32.dll and user32.dll. They are available on the Microsoft Symbols Servers. This should give you a callstack good enough.

    Hope this helps,

    Kirill

  7. gOODiDEA.NET says:

    Web 3 reasons why you should let Google host jQuery for you 链接A引发的思考 YUI Library, YUI Doc, and Github

  8. gOODiDEA says:

    Web3reasonswhyyoushouldletGooglehostjQueryforyou链接A引发的思考YUILibrary,YUIDo…

  9. Ivan Tarapov says:

    Kirill, thanks for the answer

    I remember an unsuccesful attempt to load symbols from MS servers in VS 2003. Shall try it in 2008, maybe something was changed there :)

  10. Good article,

    Some extra points you could include.

    1. The "private" microsoft symbols are needed to undercover more than the function name. So if its a Microsoft dll your encountering a problem with you should probably get in contact with MS. Something like 90% of bugs submitted to Microsoft Tech Support (MTS) are caused by third parties.

    2. If people want an insight into what a dump contains – I’d recommend downloading windbg, load a dump file and issue a  !analyze

    3. For ASP.Net hangs/crashs/high CPU/OOM’s DebugDiag has some great (expandable) rules using the debugging engine..

    4. There is a tool (err.exe I think) to see the description of codes like 0xC0000005 and their header file. Tip: Google the codes meaning rather than the hex code.

    5. Check out http://support.microsoft.com/kb/286350 – important to know difference between 1st and 2nd chance exceptions and using adplus to capture non VS related crashes

    6. Also check out John Robbins latest book.

    Cheers,

    Jeremy

  11. Lakshmikanth says:

    I have faced lot of issues with DataGridView crashing out everyting whenever i open the form with a DataGridView  in the production environment.

    I don’t face the same issue in the Development Environment.

    Not able to find the solution for it

    Your Kind Help is Appreciated

    Thanks in Advance

    Lakshmikanth

  12. whatispunk says:

    I was thrilled to learn about the Remote Debugging Monitor however upon looking into it further I found that it was truly dissapointing.

    The possibility of being able to debug processes running on our staging/development servers was tantalizing only to be crushed by the following requirement (from MSDN):

    To debug a process that is running under another account name: You Must Have Administrator privilege on the remote computer. If the ASP.NET worker process aspnet_wp.exe is running as SYSTEM or ASPNET, for example, you have to be an administrator on the computer where that process is running.

    How many developers actually have these kinds of permissions on a staging/dev server?

    Is there any workaround?

    Great article though, thanks!

  13. Kirill Osenkov says:

    Jeremy: thanks so much for these valuable comments.

    Lakshmikanth: unfortunately I cannot help people from all over the world with their debugging issues – I have my work to do. I did what I could by writing this article. You’re on your own from here :)

    whatispunk: unfortunately I’m not aware of any workaround. You might ask around on the forums though…

  14. Erik Cox says:

    Oh I got the Watson <a href="http://www.notionsolutions.com">window</a&gt; alright…after pasting that code that is..this has been a fun and educating exercise for me..and I gotta say that after reading and going through what you did I just realized that I knew very little about debugging…

  15. I recently found a new trick from 101 visual studio tricks site.

    I have enabled this setting and it throws the exact location of exception.

    goto Debug–>exceptions and

    check common language runtime exceptions.

    that is it wherever u have exceptions unhandled ur ide will show you while executing the application.

    Happy coding.

  16. Mac says:

    You ask why programmers don’t send in Error Reports – answer is simple – Blue Screens. Many of us, myself included, started out using VS under C/C++ (I think I started on v2 under NT 4). If you tried to send in a error report more often than not you crashed your machine. There was also a period (v5-6???) when an error report would disable any other development environment you had on your PC, forcing you to reinstall it. When I was building Mac/PC dual install code that was a real pain. So old timers avoid error reports. But I’ll give it a try and see if it’s really safe next time I crash VS.

  17. Guy kolbis says:

    Kirill Osenkov from the Visual Studio QA team posted a nice and very detailed post on how to debug crashes

  18. My latest in a series of the weekly, or more often, summary of interesting links I come across related to Visual Studio. Channel 9 Stuff: XAML Guidelines, Part 2 10-4 Episode 2- Welcome to Visual Studio 2010 Greg Duncan posted a link to the downloads

  19. Brondill says:

    Here’s another elegant way to stop execution and bring in the debugger: When a certain condition is met, use the following statement:

    System.Diagnostics.Debugger.Break();

    That’s so cool!

  20. Igor Grigoryev says:

    Kirill,

    thanks for this post, but I do not see Threads UI, how to see this?

  21. Kirill Osenkov says:

    Hi Igor,

    in Visual Studio, after you break into Debugger, go to the Debug menu -> Windows -> Threads.

    Alternatively, you can press Ctrl+D,T. Almost all Debug window shortcuts are Ctrl+D,SOMETHING.

    Hope this helps,

    Kirill

  22. Igor Grigoryev says:

    Hi Kirill,

    I see this UI now, thank you very much

    Igor

  23. Nice article – just the right kind, well documented and consise. Please put another one together sometime soon – sharing this kind of expertise and knowledge is exactly why I’ve become a better developer.

    Thank you!

  24. Rick D says:

    Nice article – I’ve sent the link to all of the developers in my team.  I’ve given some debugging training, and this emphisizes the concepts I’ve been trying to get across to my entire team.  Thanks and keep up the good work.

  25. Balaji says:

    A very useful article kirill.. Thanks for sharing it here!

  26. He,YuanHui says:

    Hi Kirill,

    It’s a very useful article. Thank you!

    As the same reason, I transship and translated it to Chinese in my blog so that all the others can enjoy it. The address is http://www.cnblogs.com/khler/archive/2009/02/08/1386462.html, hope for your support!

    Thanks again!

  27. Big thanks to He,YuanHui who has translated my debugging tutorial into Chinese: http://www.cnblogs.com/khler/archive/2009/02/08/1386462.html

  28. Pramod Pawar, MCTS says:

    Awsome. It is really nice article.

  29. I was a little sceptical, but after applying a few tricks it really boosted my Visual. Thank you very much.

  30. Reelix says:

    Whenever a windows program crashes (Or explorer itself) you get a VERY similar dialog box.

    Now, those who have read up on the matter realize that many hundreds of thousands of people click "Send Error Report", and, due to the horrifically large volume of these being sent, they logically cannot be individually attended to.

    Now, when Visual Studio crashes, people see a near-identical dialog box, assume it’s just the generic Microsoft one, and click "Don’t Send" since they are aware the generic ones are never actually attended to (In their opinions)

    If you could possibly modify the dialog box to be less "Microsofty" (Which might be slightly difficult, since it is indeed a Microsoft product), people might be more inclined to click "Send Error Report" – Maybe something small, such as "Optional E-Mail Address for notification" or something, to make people realize it’s not the generic dialog box.

    I hope this helps!

    – Reelix

  31. Jon Galloway says:

    We’d been getting sporadic reports of high CPU usage in Witty (a WPF Twitter client). I’d tried running

  32. Civa says:

    Hi Kiril can you provide me address where I can send some info about IDE crash?I already did over microsoft connect but they redirected me to your location.

    Thank you in adnvance.

    Best regards,

    Civa

  33. Kirill Osenkov says:

    Hi Civa,

    if you have a callstack and a minidump then MS Connect is the right place. I don’t know why they redirected you to my blog. They should give you an FTP address where you should upload your info. If this is not the case, let me know the Connect ID and I will take a look myself.

    A Connect ID is the bug number, for example in https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=279093 it’s 279093.

    Thanks,

    Kirill

  34. Ken Powers says:

    Kirill,

    Thanks for an informative article.  I was wondering if you are aware of any courses that focus primarily on debugging.  I’m a DBA who is doing more development work and I’m a little lost when it comes to using the Visual Studio debugger.

  35. Kirill Osenkov says:

    Hi Ken,

    check this out:

    http://www.safaribooksonline.com/events/2010/debugging.html

    Hope this helps!

    Kirill

  36. Ken Powers says:

    Thanks Kirill, I will check it out.

  37. Visual Studio Crashing when opened says:

    Krill,

    I had logged this issue with Microsoft Connect. My VS IDE crashes the moment I try to open it. Microsoft COnnect wanted to get the call stack for this. How can I get the call stack when I do not even have a chance to attach the debugger to the process. To attach the debugger I should be able to successfully open VS and click Ctrl+Alt+P . Mine crashes beore VS completed its opening sequence.

    Regards,

    Prabhat

  38. Does not work VS 2010 says:

    Hi. I will not open projects in Visual Studio 2010. This is screen shots:

    http://rghost.ru/45444588.view

    http://rghost.ru/45444622.view

    I don't know what to do

  39. hans meiser says:

    Great article, thank you!

    I was no longer thrown into the debugger on memory exceptions, my debugging executable would just stop and there was no debug session. Enabling the right exceptions in the Debug menu brought it all right back!

    Either the defaults for C development in VS2014 are wonky, or they were migrated wrong from my old 2008 install.

  40. Alistair says:

    The link to the video appears to be broken.