Deadlock due to two versions of same assemblies loaded


 


This time we will talk about a very strange deadlock problem. Customer has an ASP.NET 2.0 application running on IIS 6. This application hangs randomly. We collected hang dumps to investigate, here are the interesting findings.


Based on the result of “!syncblk”, it is a typical deadlock. Thread 43 was blocked by thread 13, and thread 13 was waiting on a lock owned by 43.


!syncblk


Index SyncBlock MonitorHeld Recursion Owning Thread Info  SyncBlock Owner


 1321 06dfa67c           13         1 02935f58  88d0  43   16338564 System.Object


                                Waiting threads: 13 32 64 65 67 70


 1322 06df0c64            3         1 00125540  9ec0  13   10332338 System.Object


                                Waiting threads: 43


 


Here is the stack of thread 43. I deleted unrelated information to make it clear.


0:043> !clrstack


037bea08 7c82ed54 System.Threading.Monitor.Enter(System.Object)


037bea60 02afa464 Deaklock.LockFunction2 (System.String, System.Object, System.Web.Caching.CacheItemRemovedReason)


………………………………


037bec40 02af35f8 Deadlock.LockFunction1()


…..


 


Here is the stack of thread 13.


01d4e570 7c82ed54 System.Threading.Monitor.Enter(System.Object)


01d4e5c8 02af347d Deadlock.LockFunction1 ()


………………………..


01d4eeb4 02af4d9c Deadlock.LockFunction1 ()


…………


 


Let’s take a look at the code by reflector. I can’t see any problem here as there is no problem to require the same lock owned by the thread itself. Then here coming the strange problem, why thread 13 and 43 owned different objects? In fact, they should use the same lock object.


public static void LockFunction1 ()


{


    ……..


    lock (lockObj)


    {


        …….


        OtherFunction();


    }


    …….


 }


 


OtherFunction called LockFunction2 eventually.


 


public static void LockFunction2()


{


    lock (lockObj)


    {


        ……….


    }


}


Using “!ip2md” command, we found the problem. There are two versions of deadlock modules loaded into same application. Then, this result in a typical deadlock scenario like below.


 







Thread 13


Lock(OldVersionDeadLock:lockObj)


{


…..


//dead lock here


Lock(NewVersionDeadLock: lockObj)


{   


    ……


}


}


Thread 43


Lock(NewVersionDeadLock:lockObj)


{


    ……


    //dead lock here


    Lock(OldVersionDeadLock:lockObj)


    {   


        ……


    }


}


 


Resolution:


This may due to an un-successfully upgrade of application. Problem resolved by update the application again and remove old version of assembly from GAC.



Reference:


892277          Troubleshooting ASP.NET using WinDbg and the SOS extension


http://support.microsoft.com/default.aspx?scid=kb;EN-US;892277


 



Best Regards,


Wei Zhao

Comments (1)

  1. AtulGupta says:

    Interesting. though I don’t completely understand why there were different objects that the code accessed. When the first part had locked old version lock, why didn’t the inner lock get the same object? Shouldn’t this be something that the framework should garuntee?

Skip to main content