It is an ASP.NET application and randomly experience 100% CPU issue. We can’t get a relief until restart the worker process.
To troubleshooting this problem, we collected two dumps one CPU is high. But compare the “!runaway” results, we find several threads consumed most of the CPU time. And all these threads were doing similar task – get item from a Dictionary.
Here is the call stack
OS Thread Id: 0xa18 (42)
04eae498 7939f8f5 System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].FindEntry(System.__Canon)
04eae4cc 04792ae1 MyPage.Page_Load(System.Object, System.EventArgs)
From first dump, the thread was finding in the Dictionary, and from the second dump, the same thread was still finding in the same Dictionary. This is unbelievable as the Dictionary has 3 items only. So, definitely these threads were entered an endless loop.
By review the code, we found this application modify/read the Dictionary object without any lock. This is the cause of the problem by a simple look at the FindEntry code via Reflector. Below is the information from MSDN – Dictionary is not thread safe.
A <(Of <(TKey, TValue>)>) can support multiple readers concurrently, as long as the collection is not modified. Even so, enumerating through a collection is intrinsically not a thread-safe procedure. In the rare case where an enumeration contends with write accesses, the collection must be locked during the entire enumeration. To allow the collection to be accessed by multiple threads for reading and writing, you must implement your own synchronization.
The solution is Sync the access to Dictionary object.