High Memory due to System.WeakReference

  • Article

 

We recently saw an issue that was manifesting as very high memory utilization by a w3wp.exe process that was hosting an ASP.net application. To start troubleshooting, we gathered a memory dump of the process when its memory usage was very high.

 

A standard way of starting out the debugging of a high memory problem in a managed application is to run the !dumpheap –stat command. When we did so, we found millions of 16 byte System.WeakReference objects (I’ve truncated the output for brevity’s sake):

 

 

0:000> !dumpheap -stat

Loading the heap objects into our cache.

total 28,603,114 objects

Statistics:

        MT Count TotalSize Class Name

0x648efc44 4,199 83,980 System.Configuration.ConfigurationValue

0x6639b220 4,368 87,360 System.Web.VirtualPath

0x6639e3fc 4,534 90,680 System.Web.Caching.CacheKey

0x663b4074 2,063 90,772 System.Web.HttpCookie

0x648ea704 2,003 96,144 System.Configuration.ConfigurationValues

0x79131840 99 104,612 System.DateTime[]

0x663b27f4 2,066 107,432 System.Web.HttpValueCollection

0x663aa5e8 2,072 107,744 System.Web.HttpModuleCollection

0x66435948 2,073 107,796 System.Collections.Generic.Dictionary`2

0x65409d70 1,772 113,408 System.Data.SimpleType

0x663ada24 4,126 115,528 System.Web.Hosting.RecyclableCharBuffer

0x663aa6f8 4,126 132,032 System.Web.HttpAsyncResult

0x7910d7e8 4,148 132,736 System.AsyncCallback

0x6641f33c 3,028 133,232 System.Web.UI.Control+OccasionalFields

0x65405fdc 1,102 141,056 System.Data.SqlClient._SqlMetaData

0x7a766474 8,912 142,592 System.ComponentModel.CollectionChangeEventArgs

0x79111038 2,748 153,888 System.Reflection.RuntimePropertyInfo

0x7910efbc 13,608 163,296 System.Runtime.Remoting.Messaging.CallContextSecurityData

0x663af054 2,063 165,040 System.Web.HttpWriter

0x653fe4d4 2,080 208,000 System.Data.SqlClient.SqlParameter

0x663b08c8 4,126 231,056 System.Web.HttpCookieCollection

0x02525e84 2,073 240,468 ASP._global_asax

0x79109778 4,395 246,120 System.Reflection.RuntimeMethodInfo

0x663af964 2,072 248,640 System.Web.SessionState.SessionStateModule

0x79102290 25,426 305,112 System.Int32

0x663aa0a0 2,063 354,836 System.Web.HttpRequest

0x663a9dac 2,063 387,844 System.Web.HttpContext

0x663a4adc 2,063 404,348 System.Web.HttpResponse

0x79108ce0 11,536 415,296 System.Collections.Hashtable+HashtableEnumerator

0x7911a2d0 13,610 489,960 System.Runtime.Remoting.Messaging.LogicalCallContext

0x79104de8 13,616 490,176 System.Threading.ExecutionContext

0x663b205c 31,080 497,280 System.Web.HttpApplication+SyncEventExecutionStep

0x663ad7c0 2,063 503,372 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6

0x7912d7c0 7,403 522,620 System.Int32[]

0x7a7566e8 26,959 539,180 System.ComponentModel.EventHandlerList+ListEntry

0x65412bb4 514 539,672 System.Data.RBTree`1+Node[[System.Data.DataRow, System.Data]][]

0x7a75a878 35,598 569,568 System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry

0x79104368 25,337 608,088 System.Collections.ArrayList

0x0252788c 2,027 770,260 ASP.default_aspx

0x654088b4 5,281 781,588 System.Data.DataColumn

0x79108afc 11,541 877,116 System.Threading.ExecutionContext+ExecutionContextRunData

0x654359c8 1,038 1,125,544 System.Data.RBTree`1+Node[[System.Int32, mscorlib]][]

0x66437650 2,073 1,252,092 System.Collections.Generic.Dictionary`2+Entry

0x791044dc 53,803 1,721,696 System.EventHandler

0x79101fe4 31,106 1,741,936 System.Collections.Hashtable

0x000e2dc0 3,644 3,621,108 Free

0x7912d9bc 31,396 4,984,728 System.Collections.Hashtable+bucket[]

0x790fd8c4 95,552 9,502,548 System.String

0x7912dd40 8,376 12,904,904 System.Char[]

0x7912d8f8 53,376 336,430,944 System.Object[]

0x79104c38 27,851,257 445,620,112 System.WeakReference

Total 28,603,114 objects, Total size: 836,223,424

 

 

NOTE: For more information on the !dumpheap debugger command, have a look at the following blog post:

 

 

https://blogs.msdn.com/tess/archive/2005/11/25/496973.aspx

 

 

The next step was to dump out some of the System.WeakReference instances to try to dig in to why they’re there:

 

0:000> !dumpheap -mt 0x79104c38

Using our cache to search the heap.

   Address MT Size Gen

0x025d7ab8 0x79104c38 16 2 System.WeakReference

0x025dc2d0 0x79104c38 16 2 System.WeakReference

0x025dc540 0x79104c38 16 2 System.WeakReference

0x025dd330 0x79104c38 16 2 System.WeakReference

0x025dd8d8 0x79104c38 16 2 System.WeakReference

0x025def80 0x79104c38 16 2 System.WeakReference

0x025def90 0x79104c38 16 2 System.WeakReference

0x025defa0 0x79104c38 16 2 System.WeakReference

0x025defb0 0x79104c38 16 2 System.WeakReference

0x025defc0 0x79104c38 16 2 System.WeakReference

0x025defd0 0x79104c38 16 2 System.WeakReference

0x025defe0 0x79104c38 16 2 System.WeakReference

 

0:000> !do 0x025defa0

Name: System.WeakReference

MethodTable: 79104c38

EEClass: 79104bd4

Size: 16(0x10) bytes

GC Generation: 2

 (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)

Fields:

      MT Field Offset Type VT Attr Value Name

791016bc 40005a9 4 System.IntPtr 1 instance 33389776 m_handle

7910be50 40005aa 8 System.Boolean 1 instance 0 m_IsLongReference

 

 

All of the objects we dumped out looked the same, and none of them appeared to be rooted. Looking around for other signs of problems, we found an components built in debug mode:

 

0:000> !sos.FindDebugModules

Loading all modules.

Searching for modules built in debug mode...

 

custom_component.DLL not built release

custom_component2.DLL not built release

custom_component3.DLL not built release

 

 

Having modules built in debug mode running on a Production server is never a good idea. And as it turned out in this case, the debug mode modules combined with the fact that these modules implement the __ENCLIST helper class for Visual Studio’s Edit and Continue feature.

 

When a component is built in debug mode, the Edit and Continue debugging feature of Visual Studio is enabled. The Edit and Continue debugging feature in Visual Studio 2005 and 2008 maintains a list of weak references to objects that have been created. If the object that has been created is a class that contains an event, these weak references are maintained for the duration of the program. This behavior increases memory usage. To run into the memory usage problem, the component must be built in DEBUG mode, and in must use the __ENCLIST help class that allows edit and continue.

 

In this scenario, the problem got resolved by rebuilding those components in Release mode instead of Debug mode.