Hangs and how to solve them – part 1 – Deadlocks


We have talked about high memory quite a bit so thought it would be a good idea to move onto hangs.  So first some background on what you may see:

  1. Slow performance
  2. Web pages timing out
  3. Event logs mentioning deadlocks
  4. Event logs mentioning not responding to a ping

All of these are signs that you are experiencing a hang.  So what do we do about them?  The best place to start is with a dump at the time of the hang.  You can use DebugDiag to capture the dump or just use the Windows Debuggers to get it using Adplus.

After we get our dump, there are a number of different things we can look at for a hang.  Tess has one problem documented here.  And a really interesting way to look at some hangs here.

Note If you aren’t getting this data from the source, maybe you should check out the original: http://blogs.msdn.com/tom

One of the first things to check for in a hanging situation is threads that are deadlocked.  There are two types of deadlocks, it could be that we have 2 or more threads deadlocked waiting on native locking mechanisms.  Or they could be waiting on a managed locking mechanism.  So to check them we do the following.

Managed deadlock

We run the command !sos.syncblk this will show us any managed locking objects and which thread holds onto the object.  If we use the one out of the clr10 directory of the debugger package, it will also show us any threads that are waiting on that object:

syncblk

This helps us a lot to tell if we are actually deadlocked or just running slow.  In this case, we can see that Thread 14 holds onto one lock and Thread 70 holds onto the other.  Neither of these are waiting on each other so this wouldn’t be a deadlock.  It also shows you the actual object are we are locking on.

Native deadlock

We run the command !locks in see the native locks that are being held onto.

locks

From this we can see the owning thread and how many objects are trying to get into each critical section.  So if we have a thread that is waiting, but holds onto a lock, we could have a deadlock.  From this output, you have to look at each thread and see if thread A is holding a lock, and waiting on thread B.  And thread B is waiting on thread A.

Resolution

To resolve a deadlock, assuming that we have one, is rather simple.  We just need to go to the two threads which are waiting on each other and make sure that they acquire their locks as late as possible in their code and release them as early as possible.  It is also usually a good idea to not try to acquire a lock while you hold onto another lock and if you do, to make sure that in all cases, you acquire those two locks in the same order at all times.  Otherwise two threads could end up deadlocked.

There is some really good additional info found here:

kick it on DotNetKicks.com


Comments (10)

  1. Lance Hunt says:

    +1 on DebugDiag.   It is an awesome tool for simplifying capturing dumps.

  2. tomchris says:

    One thing I forgot to mention, DebugDiag does a very good job of showing you native hangs and what is happening and who is at fault.  It shows deadlocks very well in it’s report.

  3. So in a previous post, we began discussing hangs, Hangs and how to solve them – part 1 – Deadlocks .

  4. 【原文地址】 March 28th Links: ASP.NET, ASP.NET AJAX, ASP.NET MVC, Visual Studio, Silverlight, .NET 【原文发表日期

  5. Dflying Chen says:

    摘要本期共有9篇文章:提高C#和VisualStudio2008生产力的10个技巧ASP.NETMVCAction过滤器:缓存和压缩程序停止工作及其解决方法:第一部分:死锁调用D…

  6. Jacky_Xu says:

    摘要

    本期共有9篇文章: 提高C#和VisualStudio2008生产力的10个技巧

    ASP.NETMVCAction过滤器:缓存和压缩

    程序停止工作及其解决方法:第一部分…

  7. Collegamenti del 28 Marzo, ASP.NET, ASP.NET AJAX, ASP.NET MVC, Visual Studio, Silverlight, .NET

  8. Aqui está a última versão da minha lista de links . Verifique também minha página de Dicas, Truques e