The case of the not so ConcurrentDictionary


I was looking at our DevOps dashboards and saw some really weird patterns:

So I pinged my colleague who owns this service and he noticed it was actual very predictable:

Like clockwork, once a minute - he went further and got a PerfView which showed high contention on a newly added ConcurrentDictionary:

He then asked me to take a look since that ConcurrentDictionary was added on my suggestion to work around another issue (which I will blog about one day). Having had that problem before, I figured we either had a hot spot or a hashing function problem - so I got a dump of the process to see which (I could have saved some time and looked at the source...but as they say, there's nothing like a good dump).

0:000> !do 0000015d19e676f0
Name: System.Collections.Concurrent.ConcurrentDictionary`2
MethodTable: 00007ff8e18f9728
EEClass: 00007ff8e18c5de0
Size: 64(0x40) bytes
File: D:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
 MT              Field   Offset Type                 VT Attr     Value            Name
00007ff8e187d930 4001830 8      ....Byte, mscorlib]] 0  instance 0000015e3a819078 m_tables
00007ff93b3c4300 4001831 10     ...Canon, mscorlib]] 0  instance 0000000000000000 m_comparer
00007ff93b3b1f28 4001832 30     System.Boolean       1  instance 1                m_growLockArray
00007ff93b3a9288 4001833 20     System.Int32         1  instance 0                m_keyRehashCount
00007ff93b3a9288 4001834 24     System.Int32         1  instance 256              m_budget
0000000000000000 4001835 18     SZARRAY              0  instance 0000000000000000 m_serializationArray
00007ff93b3a9288 4001836 28     System.Int32         1  instance 0                m_serializationConcurrencyLevel
00007ff93b3a9288 4001837 2c     System.Int32         1  instance 0                m_serializationCapacity
00007ff93b3b1f28 400183b 10     System.Boolean       1  static   <no information>
0:000> !do 0000015e3a819078
Name: System.Collections.Concurrent.ConcurrentDictionary`2+Tables
MethodTable: 00007ff8e18fafd0
EEClass: 00007ff8e18c6ab0
Size: 48(0x30) bytes
File: D:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
 MT              Field   Offset Type            VT Attr     Value            Name
0000000000000000 400341d 8      SZARRAY         0  instance 0000015e3a816ca8 m_buckets
00007ff93b3a6fc0 400341e 10     System.Object[] 0  instance 0000015e3a816290 m_locks
00007ff93b3a9220 400341f 18     System.Int32[]  0  instance 0000015e3a817e28 m_countPerLock
00007ff93b3c4300 4003420 20...Canon, mscorlib]] 0  instance 0000015d19e677a0 m_comparer
0:000> !DumpArray 0000015e3a816ca8
Name: System.Collections.Concurrent.ConcurrentDictionary`2+Node
MethodTable: 00007ff8e18fae50
EEClass: 00007ff93ad6aa00
Size: 4480(0x1180) bytes
Array: Rank 1, Number of elements 557, Type CLASS
Element Methodtable: 00007ff8e18fad88
[0] null
[1] null
<...>
[428] null
[429] 0000015d46de8280
[430] null
<...>
[556] null
0:000> !do 0000015d46de8280
Name: System.Collections.Concurrent.ConcurrentDictionary`2+Node
MethodTable: 00007ff8e18fad88
EEClass: 00007ff8e18c6990
Size: 40(0x28) bytes
File: D:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
 MT              Field   Offset Type             VT Attr     Value            Name
00007ff93b3abf10 4003421 8      System.__Canon   0  instance 0000015d46de8240 m_key
00007ff93b3a8940 4003422 1c     System.Byte      1  instance 0                m_value
00007ff8e187d800 4003423 10 ....Byte, mscorlib]] 0  instance 0000015e47c672e8 m_next
00007ff93b3a9288 4003424 18     System.Int32     1  instance 37103870         m_hashcode
0:000> !do 0000015e47c672e8
Name: System.Collections.Concurrent.ConcurrentDictionary`2+Node
MethodTable: 00007ff8e18fad88
EEClass: 00007ff8e18c6990
Size: 40(0x28) bytes
File: D:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
 MT              Field   Offset  Type            VT Attr    Value            Name
00007ff93b3abf10 4003421 8       System.__Canon  0 instance 0000015e47c672a8 m_key
00007ff93b3a8940 4003422 1c      System.Byte     1 instance 0                m_value
00007ff8e187d800 4003423 10 ....Byte, mscorlib]] 0 instance 0000015e4781f3f8 m_next
00007ff93b3a9288 4003424 18      System.Int32    1 instance 37103870         m_hashcode

Sure enough, there's only one bucket occupied, and all of the items in that bucket have the same hash code, so our ConcurrentDictionary is really a giant linked list, with a giant lock on top...

Our ConcurrentDictionary's keys are System.EventHandler which is really a delegate - which default HashCode implementation is...the hash code of the underlying type which means all of our delegates have the same hashcode, hence the same bucket....DOH!


Comments (0)

Skip to main content