ASP.NET Memory Investigation

This is a bit of a continuation of ASP.NET Memory Issue: High memory usage in a 64bit w3wp.exe process so if you haven’t checked it out you might want to just glance over it before reading this one to get the context of the problem and some notes on 64 bit debugging.

Before I go into the details I just want to mention that what I will talk about does not only apply to 64 bit debugging even though I am using a 64 bit dump,  you can just as easily see the problems I will talk, about and use the same techniques to discover them in a 32 bit process/dump.

In my previous posts the dumps have normally been pretty clean, and a lot of times they have only been suffering from one single memory issue, to make it easier to illustrate how a small detail can cause a big havoc.  As we know though, the world is seldom or never black and white and in many cases there are more than one reason for high memory usage in a process.

This particular dump has various high memory consumers (apart from the issue described in the previous post) so it gives me an opportunity to show a few different common high memory culprits. 

In order to not give out any proprietary information I have changed the contents of strings and names of certain custom classes in this dump, so if you are a nitpicker and find inconsistencies in string lengths or similar, that is why:)

Debugging details of the memory investigation

I am going to concentrate on the .NET GC heaps in this case, and freely ignore any objects that are not rooted, and for now attribute them to the issue described in the previous post.

The GC Heaps account for 1,502,201,224 bytes out of the 1,6 GB used for the dump, and out of this approximately 800 MB-1GB is Gen 0 objects (see the previous post), so the interesting thing is to figure out where the rest is going, I.e. the memory that is really sticking around.

Looking at the bottommost ~20 lines of !dumpheap -stat, showing the individual objects that used up the most memory, we get…

0:000> !dumpheap -stat

0x0000064253bda8f8 23,805 5,713,200 System.Xml.Schema.XmlSchemaElement
0x00000642bcf83908 126,788 6,085,824 System.Web.UI.WebControls.ListItem
0x00000642bcee8240 53,173 6,380,760 System.Web.UI.LiteralControl
0x00000642b77fcfb0 13,351 6,835,712 System.Data.DataTable
0x00000642bcee9a28 245,535 7,857,120 System.Web.UI.Pair
0x00000642bceabd08 263,432 8,429,824 System.Web.UI.StateBag
0x00000642788e5a88 172,973 9,714,456 System.Int32[]
0x00000642788405d8 124,338 10,941,744 System.Collections.Hashtable
0x00000642b7800d20 249,035 11,953,680 System.Data.DataRowView
0x0000064274e57ce8 300,268 12,010,720 System.Collections.Specialized.HybridDictionary
0x00000642bcea8920 137,011 12,056,968 System.Web.UI.Control+OccasionalFields
0x0000064274e57e70 278,134 13,350,432 System.Collections.Specialized.ListDictionary
0x0000064278827060 566,228 13,589,472 System.Int32
0x0000064253bcd750 466,140 14,916,480 System.Xml.Schema.BitSet
0x00000642788bd790 466,141 14,916,544 System.UInt32[]
0x00000642bceca168 514,257 16,456,224 System.Web.UI.StateItem
0x0000064253bcd090 444,728 17,789,120 System.Xml.XmlQualifiedName
0x000006427883e400 457,745 18,309,800 System.Collections.ArrayList
0x00000642bceb4f98 137,019 20,826,888 System.Web.UI.WebControls.TableCell
0x0000064274e57fe0 559,345 22,373,800 System.Collections.Specialized.ListDictionary+DictionaryNode
0x0000064253bc1c80 571,126 22,845,040 System.Xml.NameTable+Entry
0x00000642b78659d0 14,096 24,055,168 System.Data.RBTree`1+Node[[System.Int32, mscorlib]][]
0x0000064278824860 784,204 25,094,528 System.Decimal
0x000006427883f770 565,270 27,132,960 System.Collections.ArrayList+ArrayListEnumeratorSimple
0x00000642b77fe2f0 124,259 28,828,088 System.Data.DataColumn
0x00000642788dbf50 30,668 29,910,520 System.Byte[]
0x00000642b7820d60 19,193 34,319,192 System.Data.RBTree`1+Node[[System.Data.DataRow, System.Data]][]
0x00000642b77fe7c0 377,440 36,234,240 System.Data.DataRow
0x00000642788db830 71,095 38,538,856 System.Char[]
0x00000642788fe918 135,194 50,900,832 System.Collections.Hashtable+bucket[]
0x00000642788e3b70 48,047 138,801,256 System.Decimal[]
0x00000642788d4758 897,213 172,527,160 System.Object[]
0x000006427881aaf8 3,013,208 197,958,896 System.String
0x0000000000146e90 12,107 220,141,008 Free
Total 15,904,890 objects, Total size: 1,502,201,224

Fragmented blocks larger than 0.5 MB:
Addr Size Followed by
00000000934858a0 25.3MB 0000000094dc8d50 System.Threading._ThreadPoolWaitCallback
0000000094dc8dc0 11.4MB 000000009592b4e0 System.Threading._ThreadPoolWaitCallback
000000009592b550 2.3MB 0000000095b6be48 System.Threading._ThreadPoolWaitCallback
0000000095b6beb8 2.1MB 0000000095d7a828 System.Threading._ThreadPoolWaitCallback
0000000095d7a898 0.7MB 0000000095e350b8 System.Threading._ThreadPoolWaitCallback
0000000095e35128 2.7MB 00000000960e7968 System.Threading._ThreadPoolWaitCallback


I have grouped them together by color to get a bit of an overview of what we are looking at… and categorized them like this

  • XML Related Items
  • Data Related Items
  • UI + Web Controls Related Items
  • State and Cache related items
  • Free

In some cases, like in the case of Data, i have added UInt32[] for example because I know from experience that a lot of these arrays will be arrays of UInt32 objects in datasets.  The same goes for HashTables and ListDictionaries in Cache and Session State.   Of course not all of these will be directly related to Data or Cache but for rough categorization it works well to put them in those buckets.

Byte[], Object[], Char[] and String are all types of objects that you will always see at the bottom of the !dumpheap -stat output since they are used everywhere, and you can read some of my previous memory investigation posts to get more details about this.

64-bit specific memory information

Since pointers are quad-word rather than DWORDs in 64-bit processes, a lot of the objects here, such as the Object[] and some of the collections (excluding strings and Byte[] for example) are double the size they would be in a 32-bit process.


A lot of Free objects (objects that are marked collected, but the space has not been reused or compacted) usually means one of two things. 

1. The garbage collector hasn’t compacted in a while (usually the case on the workstation build)

2. You have a lot of pinned object disabling the GC from compacting the heaps

In this case, the reason is #1, but not because we are running the workstation build, but because we are running 64-bit framework RTM and have very infrequent collections.  See the previous post for a lengthier discussion and resolution.

Xml Related Items

When we have a lot of Xml Related items in the bottom part of !dumpheap -stat I usually look for a more top-level object type like XmlDocument since all other Xml related objects usually tend to be members or members of members of XmlDocuments.  In other words, if you were to get rid of the XmlDocument, the other ones would go as well.   The reason I look for a top-level object is because there are usually fewer of them so it is easier to work with them in order to find out why they stick around.

0x0000064253bc0c10      5,048     1,534,592 System.Xml.XmlDocument

And then i dump out a few and do !gcroot on them to find out why they are rooted…

0:000> !dumpheap -mt 0x0000064253bc0c10
Using our cache to search the heap.
Address MT Size Gen
0x00000000800dfb98 0x0000064253bc0c10 304 2 System.Xml.XmlDocument
0x00000000800e3578 0x0000064253bc0c10 304 2 System.Xml.XmlDocument
0x00000000800e6f58 0x0000064253bc0c10 304 2 System.Xml.XmlDocument
0x00000000800ea938 0x0000064253bc0c10 304 2 System.Xml.XmlDocument
0x00000000800ee318 0x0000064253bc0c10 304 2 System.Xml.XmlDocument
0x00000000800f1cf8 0x0000064253bc0c10 304 2 System.Xml.XmlDocument
0x00000000800f56d8 0x0000064253bc0c10 304 2 System.Xml.XmlDocument
0x00000000800f90b8 0x0000064253bc0c10 304 2 System.Xml.XmlDocument
0x00000000800fb8a0 0x0000064253bc0c10 304 2 System.Xml.XmlDocument
0x00000000800fe5a8 0x0000064253bc0c10 304 2 System.Xml.XmlDocument
0x0000000080102c28 0x0000064253bc0c10 304 2 System.Xml.XmlDocument
0x0000000080106f58 0x0000064253bc0c10 304 2 System.Xml.XmlDocument
0x000000008010a938 0x0000064253bc0c10 304 2 System.Xml.XmlDocument

0:000> !gcroot 00000000800dfb98

DOMAIN(000000000017E210):HANDLE(Strong):4b41308:Root: 00000000c0020990(System.Threading._TimerCallback)->

…so most of them are rooted in an InProcSessionState object which in turn is rooted in cache.  (This is always the case for inproc session objects).  So in other words, these XmlDocuments are stored in Session scope and more specifically in the session variable called Classes.

0:000> !do 00000000800e09e8
Name: System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry
MethodTable: 0000064274e6c8c8
EEClass: 0000064274efd388
Size: 32(0x20) bytes
GC Generation: 2
MT Field Offset Type VT Attr Value Name
000006427881aaf8 400116d 8 System.String 0 instance 00000001400dd3a8 Key
0000064278818fb0 400116e 10 System.Object 0 instance 00000000800dfb98 Value

0:000> !do 00000001400dd3a8
Name: System.String
MethodTable: 000006427881aaf8
EEClass: 000006427892f7d8
Size: 46(0x2e) bytes
GC Generation: 2
String: Classes
MT Field Offset Type VT Attr Value Name
0000064278827060 4000096 8 System.Int32 1 instance 11 m_arrayLength
0000064278827060 4000097 c System.Int32 1 instance 10 m_stringLength
00000642788216d8 4000098 10 System.Char 1 instance 50 m_firstChar
000006427881aaf8 4000099 20 System.String 0 shared static Empty
>> Domain:Value 0000000000120a60:00000642787c36e0 000000000017e210:00000642787c36e0 <<
00000642788db830 400009a 28 System.Char[] 0 shared static WhitespaceChars
>> Domain:Value 0000000000120a60:00000000bfff0550 000000000017e210:00000000bfff7390 <<

If you want to dig deeper into this you may be interested in the following two posts:

ASP.NET Memory – How much are you caching? + an example of .foreach

ASP.NET Memory Leak Case Study: Sessions Sessions Sessions…

A few of the Xml related items such as some of the XmlQualifiedName objects for example actually turn out to be rooted in DataSets but I just want to do a pretty quick and dirty memory investigation here to get a good overview of what is going on.

Data Related Items

For the data related items i usually use System.Data.DataSet as a top-level item, and if I would do that in this case (following the same procedure as for XmlDocument) I would find that some of them are stored in session scope and some in cache directly. 

State Related Items

If we have a lot of state related items that could be an indication of a couple of things

  1. Lots of caching
  2. Lots of in-proc session state (stored in cache)
  3. Lots of aspx or ascx pages around with related viewstate (System.Web.UI.Pair, System.Web.UI.StateBag)

I am going to leave #3 for a bit and discuss the other two, since we already know that we have a lot of DataSets and XmlDocuments in Session and cache.

If we take a look at the cache and it’s size we find that it is ~550 MB…

0x00000642bcece978          1            24 System.Web.Caching.Cache

0:000> !dumpheap -mt 0x00000642bcece978
Address MT Size Gen
0x00000000c001b288 0x00000642bcece978 24 2 System.Web.Caching.Cache

0:000> !objsize 0x00000000c001b288
sizeof(00000000c001b288) = 557,714,296 ( 0x213e0b78) bytes (System.Web.Caching.Cache)

This is a pretty big chunk of the memory so I think it is safe to say that other than the issue described in the previous post, this is where we should focus our efforts if we want to reduce the amount of memory used.

If we then go one step further to try to see how much this is session objects we can dump out the InProcSessionState objects and check the size of one of those…

0x00000642bcedf988        601        38,464 System.Web.SessionState.InProcSessionState

0:000> !dumpheap -mt 0x00000642bcedf988
Address MT Size Gen
0x00000000800d9ab8 0x00000642bcedf988 64 2 System.Web.SessionState.InProcSessionState
0x00000000800dba40 0x00000642bcedf988 64 2 System.Web.SessionState.InProcSessionState
0x00000000800dbb58 0x00000642bcedf988 64 2 System.Web.SessionState.InProcSessionState
0x00000000800dbcc0 0x00000642bcedf988 64 2 System.Web.SessionState.InProcSessionState
0x00000000800dbdd8 0x00000642bcedf988 64 2 System.Web.SessionState.InProcSessionState
0x00000000800dbef0 0x00000642bcedf988 64 2 System.Web.SessionState.InProcSessionState
0x00000000800dc008 0x00000642bcedf988 64 2 System.Web.SessionState.InProcSessionState

0:000> !objsize 0x00000000800dc008
sizeof(0x00000000800dc008) = 557,714,296 ( 0x213e0b78) bytes (System.Web.SessionState.InProcSessionState)

Ok, so we have 601 active sessions and the size of just one of those sessions is the same amount as the cache itself which just doesn’t make sense, and in fact offline I had taken !objsize of a sample of the datasets and XmlDocuments and although it is a sizeable chunk they will not rack up to 550 MB together, so something is fishy.

What is actually happening is that we are storing something in session scope that eventually has a link to a HttpContext, which has a link to cache, causing the actual size of the cache to be included in !objsize for the individual session objects.

When you see something like this, you should pay attention, because although the “circular reference” will be broken by taking the object out of session state, the problem is that this normally happens when you have added something to session state which has a link to an aspx page or ascx page or similar.  Very much like in the case of the EventHandlers issue or in the CacheItemRemovedCallback, but as you will soon see, this is slightly different.

UI + Web Controls Related Items

The final stage of this memory investigation is to look at the UI and Web Controls related items.  We already know that our memory can roughly be divided into memory usage caused by the issue in this post, and memory used for session state.  However what we don’t know is causing us to have so much session state since the data and xml related items don’t seem to match the total size.

If we pick the TableCells and work out who is using them, just like in the data and xml cases, we get…

0x00000642bceb4f98 137,019 20,826,888 System.Web.UI.WebControls.TableCell

0:000> !dumpheap -mt 0x00000642bceb4f98
Using our cache to search the heap.
Address MT Size Gen
0x00000000806321b0 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell
0x0000000080632850 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell
0x0000000080632a38 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell
0x0000000080632b90 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell
0x0000000080632ce8 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell
0x0000000080632e40 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell
0x0000000080632f98 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell
0x00000000806331b0 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell
0x0000000080633308 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell
0x0000000080633460 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell
0x00000000806335b8 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell
0x0000000080633710 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell

0:000> !gcroot 0000000080632850

DOMAIN(000000000017E210):HANDLE(WeakSh):4b412f8:Root: 00000000c0020b80(System.Threading._TimerCallback)->

So it appears that for some reason we are storing a web control (MyControls.MyCustomGridControl) in session state…

This may not look like a very bad idea, in fact I can completely see why this is done, since it is a fast and easy way to store the state of the data that this particular user was working with and use this throughout the session, but there are some major issues with this…

All the data related to the control will now be linked to session and can’t be garbage collected before this object goes out of session, while what we really wanted was to just store the actual data and perhaps the rowfilters.   And among this data is not only UI related items, like WebControls.DataGridItem etc. which will be relatively useless to us once the page that held the grid has finished executing… but there is also a link to the page it was loaded on in multiple ways.

1. In the parent field of the control
2. As the target for many of the events related to the grid (like in the EventHandlers memory investigation)

And since we have a link to the page, we also keep everything that belongs to it in session state, such as its viewstate, its other controls etc. which can definitely be a whole lot of data, and this data can not be collected until the session expires.

This also explain why we have a circular reference with the cache going on, since the page has a context which links to the cache…

So what should you do instead of storing controls in session scope? The best would probably be to store the data you need in a separate object and store that in session scope if it takes too long time or is otherwise impractical to gather when needed.

In summary

When it comes to memory investigations where most of the memory is .NET GC memory, the memory investigation is often a paint-by-numbers process where the steps go

  1. Run !dumpheap -stat and look for items that stick out, or try to group/categorize the bottommost items in the list to figure out who are your main memory hoggers
  2. !gcroot the objects to figure out why they are still around… (Note!  Make sure you !gcroot a couple of them before making any big decisions based on the results)
  3. Rinse and repeat until you have accounted for most of the memory.

The tricky part is of couse figuring out what sticks out, grouping them properly, and understanding the results of !gcroot:)  but hopefully this blog helps you with some of that.

Until next time,


Comments (15)

  1. Tess –

     Can you post the offending code fragment?

     I’m assuming its something simple like:

    Session.Add("control", MyControls.MyCustomGridControl);

    And a valid work around (as you are suggesting) would be to do something like:

    Session.Add("Control_Sort", MyControls.MyCustomGridControl.SortExpression);

    Session.Add("Control_SelectedIndex", MyControls.MyCustomGridControl.SelectedIndex)


  2. Tess says:

    Hi Christopher,

    Yes the offending code fragment would look something like your Session.Add…

    What you can actually store in session depends on if it has some links back to the control itself or not.  For example a SortExpression or a SelectedIndex is usually a string or an integer so that would work well.  Depending on how much state you would need to keep you could create a class containing the data from the control and sort expressions and selected index or whatever you need to store. The point being that the control itself has several links back to the page which makes it unsuitable for session storing.

  3. Tess –

     Thanks – I noted your statement about a separate object for state, I just didn’t want to write the full code in the comment ;-)

     Is there any winDB statement that would list all objects with a reference to a given object such as System.Web.SessionState.InProcSessionState?  Sort of an opposite of gcroot?

  4. Tess says:

    There isnt really an opposite of gcroot and it is hard to invision one since there is usually a very small roots to each object but could be almost an infinite number of leaf nodes.

    It would perhaps be interesting to see them in a dumpheap -stat fashion, and i believe there are some commands in the works like that for specific types of objects like datasets, sessionstate etc.

    Until then, you will have to work with windbg scripting like .foreach

  5. Josh says:

    Hi Tess,

    We have Windows Server 2003 Service Pack 2 and are having high memory issues you mentioned. I got the hotfix from Microsoft, but the dlls in the hotfix are all older revisions than the ones in Service Pack 2. Regardless, I installed the hotfix, no dlls were replaced and we still have the high GC memory issues.

  6. Jack says:

    Is this a copy-paste error?

    In "state related items" you ask for the size of the same address/object 2 times. The result is the same, but the type of object is different.

    1st try:

    0:000> !objsize 0x00000000c001b288

    sizeof(00000000c001b288) =  557,714,296 (  0x213e0b78) bytes (System.Web.Caching.Cache)

    2nd try:

    0:000> !objsize 0x00000000c001b288

    sizeof(0x00000000c001b288) =  557,714,296 (  0x213e0b78) bytes (System.Web.SessionState.InProcSessionState)

  7. Tess says:

    Nice catch Jack,  yes it must have been a copy-paste error…  I changed the object address now

  8. Jon says:

    Where are you entering these commands? These wont work at the dos prompt level:

    C:>!dumpheap -stat

    ‘!dumpheap’ is not recognized as an internal or external command,

    operable program or batch file.


  9. Tess says:

    Hi Jon,

    Those are commands to enter in windbg.exe (part of debugging tools for windows).  Look at the memory lab (on the left hand side) for better instructions on how to do this.

  10. I get several emails every week through the blog asking for help on various issues.&#160; Unfortunately

  11. alikl says:


    You are my hero!

    Another great post that helped me to resolve my customer’s problem…


  12. I have put together a quick and dirty debug diag script for troubleshooting .net memory leaks. (attached

  13. Hi Tess,

    Thanks for your labs – they were a great help. We are facing some serious memory increase in our application in production. A few weeks ago the w3wp processes never increased beyond 620 MB in task manager but now they go upto 750 MB-800MB.  I took a dump using procdump. I ran !eeheap -gc and it said total size is 523 MB but at the time I took the dump the process size was 780 MB in task manager. What would this mean?

    Eventually the app domain recycles saying that the private bytes memory limit has been reached.

    I also ran the command – !dumpheap -type *ascx to find that there are many ascx pages in Generation 2 but when I run a gcroot it looks like nothing is holding onto these pages. Please advise as we are facing huge losses in production.

  14. Tess says:

    Hi Vandana,

    The difference between the memory used for the process and !eeheap -gc can be anything from memory used for the dlls loaded or other native memory.  Given that it's a very small difference comparatively it is probably just the dlls.   It's hard to say without looking at it why they wouldnt be rooted, I would continue looking at them to see if you find any of them rooted as it might just have been that you picked one that wasn't rooted.   It could of course also be that you just recently had many requests so you have a lot of them in memory that are ready for collection.