We have talked about high memory quite a bit so thought it would be a good idea to move onto hangs. So first some background on what you may see:
- Slow performance
- Web pages timing out
- Event logs mentioning deadlocks
- Event logs mentioning not responding to a ping
All of these are signs that you are experiencing a hang. So what do we do about them? The best place to start is with a dump at the time of the hang. You can use DebugDiag to capture the dump or just use the Windows Debuggers to get it using Adplus.
If you aren’t getting this data from the source, maybe you should check out the original: http://blogs.msdn.com/tom
One of the first things to check for in a hanging situation is threads that are deadlocked. There are two types of deadlocks, it could be that we have 2 or more threads deadlocked waiting on native locking mechanisms. Or they could be waiting on a managed locking mechanism. So to check them we do the following.
We run the command !sos.syncblk this will show us any managed locking objects and which thread holds onto the object. If we use the one out of the clr10 directory of the debugger package, it will also show us any threads that are waiting on that object:
This helps us a lot to tell if we are actually deadlocked or just running slow. In this case, we can see that Thread 14 holds onto one lock and Thread 70 holds onto the other. Neither of these are waiting on each other so this wouldn’t be a deadlock. It also shows you the actual object are we are locking on.
We run the command !locks in see the native locks that are being held onto.
From this we can see the owning thread and how many objects are trying to get into each critical section. So if we have a thread that is waiting, but holds onto a lock, we could have a deadlock. From this output, you have to look at each thread and see if thread A is holding a lock, and waiting on thread B. And thread B is waiting on thread A.
To resolve a deadlock, assuming that we have one, is rather simple. We just need to go to the two threads which are waiting on each other and make sure that they acquire their locks as late as possible in their code and release them as early as possible. It is also usually a good idea to not try to acquire a lock while you hold onto another lock and if you do, to make sure that in all cases, you acquire those two locks in the same order at all times. Otherwise two threads could end up deadlocked.
There is some really good additional info found here:
- Deadlock (critical sections)
- Deadlock (executive resources)
- Deadlock (mixed objects)
- Deadlock (LPC)
- Deadlock monitor
- No More Hangs- Advanced Techniques To Avoid And Detect Deadlocks In .NET Apps
- Detecting Deadlocks in Multithreaded Win32 Applications
- Under the Hood — MSJ, January 1996