.NET Debugging Demos Lab 1: Hang


This is the first in a series of about 10 labs on .NET debugging.  The lab will use a site called BuggyBits, and as the name suggests the bits are extremely buggy.

To get started, follow the setup instructions posted here.

I have a feeling that these hands-on-labs may generate a lot of questions and although I will try to answer any questions posted in the comments I can’t promise to answer them all so please feel free to answer other readers comments if you know the answer, and make sure that you have followed all the installation instructions.

Note: The questions in the labs (Q: … ) are only meant as an aid when troubleshooting the problem.  I will moderate any comments containing answers to these questions until I have released the lab review (about a week after the orignal lab post in order to give everyone a shot at the labs without answers)

Feel free to comment on the lab format good or bad so that I know what works well and what doesn’t for future labs.

Without further a due, here comes Lab 1:

Reproduce the problem:

1. Browse to http://localhost/BuggyBits/FeaturedProducts.aspx
This should take about 5 seconds to show, you can see the start time and execution time the bottom of the page.

2. Open up 5 browsers, all browsing to this site and refresh them simultaneously

Note the execution time for each of them and make sure that the start time is pretty much the same on all (otherwise you probably didn’t run the reg file)

Q:  What are the execution times?

Q:  What is the CPU usage of the w3wp.exe process when reproing the problem? High or low CPU?

Q:  What are some potential reasons for a hang with these symptoms?

Get a memory dump:

1. Start a command window and browse to your debuggers directory.  Type the following command to prepare taking the dump but don’t hit enter quiet yet.
adplus –hang –pn w3wp.exe –quiet

2. Reproduce the problem either by browsing with 5 browsers as you did before or by stressing the site with tinyget with the following command line

tinyget -srv:localhost -uri:/BuggyBits/FeaturedProducts.aspx -threads:30 -loop:50

3. Hit enter in the adplus command window to take the memory dump while the requests are still executing.

Q: In adplus hang mode, what triggers the generation of the memory dump?

Q: What permissions do you need to take a memory dump of a process?

Q: Where are the dumps created? Hint: check the windbg help for adplus/hang mode

Open the dump in Windbg.exe

1. Open windbg and open the memory dump (.dmp file) with File/Open Crash dump.

2. Set up the symbol path (see Information and Setup Instructions for more info)

3. Load sos (see Information and Setup Instructions for more info)

Examine the stacks

1. Examine the native callstacks

~* kb 2000

2. Examine the .net callstacks

~* e !clrstack

Q:  Do you see any patterns or recognize any of the callstacks that suggests a thread is waiting for a synchronization mechanism?

Troubleshoot the hang

1. Determine the ID of the thread owning the lock

Q: What thread owns the lock?

Q: How many threads are waiting for the lock?
Hint: MonitorHeld = 1 for each owner and 2 for each waiter.

2. Pick one of the waiters (Hint: waiters will sit in AwareLock::Enter) and take a look at what it is doing.

~5s                          (move to thread 5, replace 5 with actual thread ID)
kb 2000                    (examine native stack)
!clrstack                    (examine .net stack)

Q: In which .net function is it waiting for the lock?

3. Determine what the owning thread is doing 

~5s                          (move to thread 5, replace 5 with actual thread ID)
kb 2000                    (examine native stack)
!clrstack                    (examine .net stack)

Q: Why is it blocking?

4. Examine the code for .NET method owning the lock to verify your theory.


The following articles may be useful when troubleshooting this hang:

Things to ignore when debugging an ASP.NET Hang – Update for .NET 2.0

A Hang Scenario, Locks and Critical Sections

.NET Hang Debugging Walkthrough

Automated .NET Hang Analysis

Have fun debugging 

Comments (55)

  1. Justin King says:

    Fantastic, now I’ve got to eagerly await part 2

  2. My colleague Tess Ferrandez just posted this excellent lab that provides you with a sample application

  3. Tess Ferrandez har skapat en serie med labbar där hon presenterar vanliga post mortem debugging scenarios.

  4. Andrew Lomakin says:

    Great post!

    I’m trying to apply the knowledge shared here to some .NET dumps i have lying around to see if i can get to the bottom of the problem and i’m having quite some success in drilling down the debug info. Waiting impatiently for lab part 2 – the crash πŸ™‚

  5. γ€εŽŸζ–‡εœ°ε€γ€‘ Feb 6th Links: ASP.NET, ASP.NET AJAX, Visual Studio, .NET, WPF γ€εŽŸζ–‡ε‘θ‘¨ζ—₯ζœŸγ€‘ Wednesday, February 06

  6. Jason says:

    Very nice!

  7. γ€εŽŸζ–‡εœ°ε€γ€‘ Feb 6th Links: ASP.NET, ASP.NET AJAX, Visual Studio, .NET, WPF γ€εŽŸζ–‡ε‘θ‘¨ζ—₯ζœŸγ€‘ Wednesday, February 06

  8. Collegamenti del 6 Febbraio, ASP.NET, ASP.NET AJAX, Visual Studio, .NET, WPF

  9. ASP.NETMVCζ‘†ζžΆθ·―ηΊΏε›Ύζ›΄ζ–°



  10. TGIF, almost time for the weekend… but before you leave, here is lab 3. Todays debugging puzzle will

  11. Lab 1: HangLab 2: CrashLab 3: MemoryLab 4: High CPU Hang

  12. Terry Voss says:

    I was just informed by MS tech support that my inability to debug many pages in ASP.NET VS2008 is a known bug which will be fixed in SP1 release date unknown at this time.

    This is with a converted application. I have done a new application with no problems.

    Can this be right? It doesn’t make sense that there wouldn’t be a lot of loud noise out there if everyone was having this problem.

  13. Tess says:

    I’m not exactly sure what issues you have debugging in VS 2008 so it is a bit hard to say.  

  14. Terry Voss says:

    When you place a breakpoint in code and run to it it crashes VS2008 upon reaching that breakpoint, or upon stepping one or 2 lines beyond the break line. IOWs debugging can’t happen in some pages of converted projects.

  15. just catching up with the tutorial now, this is really fun and I’m learning a lot, I always wanted something like this

    thank you very much for putting this together

  16. Tess says:

    Terry, can’t say I know off hand what is wrong there.  To be honest I haven’t really worked much with debugging in VS2008 so I would probably not be the best person to ask.  If the guy/gal you are working with says that it is a known issue, then I would trust that.

  17. Last week I published a debugging challenge for Lab 5. It was really interesting to see the results and

  18. Since I already posted a challenge for this lab earlier I didn’t want to wait too long with publishing

  19. Mutamblog says:

    Links 6/Fev: ASP.NET, ASP.NET AJAX, Visual Studio, .NET, WPF

  20. Chirag Shah says:

    Hi i was tryin to implement the lab one,

    as i open the dump file in win debugger i am not able to analyse as per the stpes provided in the article above

    N e help would b appreciated.


  21. Tess says:

    could you be a little bit more specific, what parts are failing?

  22. This is the last debugging lab in the .NET Debugging Demos series. By now you should have the basics

  23. We have reached the end of the .NET Debugging Demos series. And we are going to end it with a review

  24. The purpose of my presentation was to show some common pitfalls and of course to show off windbg and

  25. Test says:

    Collegamenti del 6 Febbraio, ASP.NET, ASP.NET AJAX, Visual Studio, .NET, WPF

  26. Michal says:

    First thank you for this informative and educationg blog!

    I have followed this lab but did not get any hang.  All the browsers have returned finally and when looking at the hang dump there were no deadlocks.

    Also, using the tinyget.exe to cause a hang,

    I took a hang dump during the process’s run but when opening the dump, the winDBG always get stack when trying to view certain thread’s stack (always the same threads).

    The cpu utilization is ~0 both for w3wp and tinyget process.

    Any ideas?

    Thank you, Michal

  27. Tess says:

    the cpu utilization should be pretty low, and the requests should eventually return (after 5 seconds) so that is normal…   you need to get the memory dumps while the requests are still executing to see why they are slow…

  28. I know I’m a little late in the game, but I decided that after vacation it was finally time for me to

  29. James says:

    Hey Tess,

    When I run adplus -hang -pn w3wp.exe -quiet,  the w3wp.exe memory (from task manager) jumps as the dump occurs.  For instance, the memory starts at around 48Mb and within a matter of 10 seconds after running adplus it goes up to 180Mb.  Is that supposed to happen?  And if so, doesn’t that mean a lot of "stuff" is being added that wasn’t there before (180mb after dump – 48mb before dump = 132Mb of stuff)?

    I should add, that the Virtual Memory stays at 48Mb…



    btw – You’re my memory debugging hero!

  30. Tess says:


    It adds all the pages your process touches to the dump, that is why you see a jump in taskmanager (workingset) as it loads them in to the process.  It doesnt really add any more virtual bytes or private bytes…

  31. I’ve been wanting to dig into the Test features in Visual Studio Team System for a while to β€œdebug” issues

  32. Este fin de semana, como ya sabréis casi todos, se ha celebrado la primera 'iteración' del DevCamp;

  33. Earlier this week I got an email from a reader who had a hang in an application and sent in some Debug

  34. Jason Lill says:

    I am having a problem examining the ,net call stack.   when I run the ~* e !clrstack command I get the message "No export clrstack found"

  35. Tess says:

    That will usually happen if you dont have sos loaded…

  36. Adam says:

    No matter what I run to get an XP virtual machine to serve up more than 2 requests, I cannot get IIS to handle these requests concurrently. I confirmed in regedit that MaxConnections* are both set to 10; I even tried "adsutil set w3svc/MaxConnections 40" that was mentioned elsewhere (http://www.codinghorror.com/blog/archives/000329.html). I have restarted IIS and the virtual machine itself, but multiple requests to FeaturedProducts.aspx always get queued one after another (with start time incrementing by the delay for each one). Is there something I am missing in this setup process to get XP less crippled for development work?

  37. Tess says:

    It should work just fine on XP. Not sure if w3svc/MaxConnections would really have an effect on this though…  The regfile sets up a global setting on your machine that allows you to make more than the normal 2 outgoing connections.

  38. Scott says:

    Same results as Adam. I’m not sure how to refresh mulitple IE instances simultaneously, so I used Refresh All with 5 tabs on a single IE browser. Start times were approx 5 seconds apart and execution times were all approx 5 seconds – the opposite of expected. XP SP2; .NET 2.0; Confirmed registry entries and both correctly set to 10. I proceeded with the lab using a dump generated while tinyget was running and saw expected blocking results.

    Also had to load sos differently, perhaps since I’m not using 3.5 on this machine.

  39. Tess says:


    I am not sure why you are seeing the results you are seeing, they seem to indicate that the reg. settigs are still not correct,  3.5 or not should not make a difference…

  40. Mike says:

    I kkep getting an error when tryuing to run tinyget against the website. I can browse to it fine. If I use the -t switch it  says: "You do not have permission to view this directory or pageusing the credentials taht you supplied becuase your web browser is sending a WWW-Authenticate header fileds taht the webserver is not configured to accept" Thats using Integrated security. If I set it to anonymous I just get "You are not authorized to view this page".

    So Im not abel to run tinyget to stress test.

    Unfortunately after alot searching I could not find anything related to this message when using Tinyget . Hoping that i may find the anser here

  41. Pyroxene says:


    I also got the message ‘No export clrstack found’ after running ~* e !clrstack

    However, after reading the steps again, it occurred to me that I forgot to load SOS. After doing this the results were as expected.

  42. Albert Lee says:

    Great works.

    I will read all of your labs.

  43. Jonathan says:

    For some reason when I run the app in the browser the w3wp.exe process doesn’t show up. If I run in TinyGet it says "cannot send data, because connection is closed" and "Socket Error on Receive", but I believe that just may be a firewall thing. I wonder if I’m just missing something. Thanks for the labs!

  44. Chris says:

    Likewise, I cannot get my browser instances to load concurrently.  Is there a missing step, such as logoff/logon?  Also, I am using IE8 and Win7 RTM – does that change anything?  Thanks.

  45. Tess says:

    I had the issue with the browser instances not loading concurrently as well and i believe that is because of changes in IE8 and how browsers share the same space.

    Tinyget should still work and does for me on Win7, and if you want to see two concurrent requests in the browser you can start one browser normally and then one in private mode.

  46. Stuart says:

    Thanks for these tutorials – superb!

    I’ve currently got a website which is hanging on the public IP address – but not on the local address – so when I look at call stacks in w3wp then everything looks good (because it is working).

    But something is hanging πŸ™ And at the moment it seems to be linked to some dodgy coming in from the GBPlugIn User Agent from Brazil…

    Do you have any ideas where to look? It doesn’t feel like I should be looking in ASP.Net code at the moment :/

    Any pointers appreciated – I’ve enjoyed learning things like windbg this week!


  47. Tess says:

    If you can’t find anything in the stacks, then perhaps you might want to try looking at fiddler traces and seeing if the requests look weird in some way.

  48. Stuart says:

    Thanks for the answer.

    I hope this isn’t too stupid a question… but how do I get fiddler traces on the server?

    I can’t even ask the suspect users to get Fiddler traces – as far as I can tell, the requests from the users are coming through their work network – and on that network something called WEBWASHED is doing something to the HTTP requests to "protect" the users

    I’ll keep looking…

  49. Tess says:

    you would get fiddler traces on the client so if that is not an option then you would have to find anohter way

  50. Stuart says:

    Thanks – I’ll keep looking.

    If I can’t think of anything else I’ll have to think about installing WinPCaps on the server – just have to get permission to do that in production environment!

  51. Shyam says:

    Hi Tess,

               To troubleshoot a problem with our web services, I am trying to learn the skills by following your blog. I started with this demo on Windows 7 which has IIS7.5 version. I copied the tinyget from one of windows server 2003 and run the command as specified. But I couldn't see the load increased more than 3 threads executing at any point of time. There are no wait calls on any threads , no requests in queue, cpu is low and there are free threads.  I dont see any timeouts in IIS logs as well.

    I did run the InternetConnections.reg file and verified the registry settings.  While searching on the net, I came across the below link


    which says for windows 7 home premium version, IIS limits the concurrent requests to 3.  Is this correct and if so, is there any way to run these demo labs on windows7 ( I saw your comment as tested fine with windows 7). Am I missing something.

    I thought of  changing aspnet.config file as per the below link (but the load is not even touching the default 12 )


    Can you please guide me on this.



  52. Lavanya says:


    We are facing one issue i.e. Kernelbaseswitchtothread just hangs and the thread gets blocked.

    Any idea on how this can occur.

    So far we are able to reproduce it 2 times.

  53. Sashi says:

    Hi Tess, i am working on a dump to determine the reason for High CPU hang…i ran the command ~* e !clrstack. i noticed the following

    System.Reflection.Assembly.InternalGetSatelliteAssembly(System.Globalization.CultureInfo, System.Version, Boolean) line on all the stacks…

    In the code, i do subscribe to AppDomain.CurrentDomain.AssemblyResolve and in the handler i load the dll from a specific folder…

    In the handler, i do ignore loading the dlls when the assembly name ends with .resources.. (..assemblyName.EndsWith("resources"))).

    i loaded sosex and ran the !dlk command…i dont see any deadlocks..

    Any suggestions?