I am getting OutOfMemoryExceptions. How can I troubleshoot this?
Problem:
You've written an ASP.NET application that is getting OutOfMemoryExceptions.
Cause:
Let's find out...
Resolution:
Use Windbg to take a look at the heap.
Is it a leak?
Take a look at the memory usage of you application using perfmon. If memory is slowly increasing and never released, then you have a leak. If it is going up and down like a rollercoaster, then you are most likely using huge amounts of memory for certain operations which is later garbage collected.
How do I troubleshoot this?
Below I'll try to give you a step by step description of how to troubleshoot this scenario
1. Get a memory dump
This is done using Windbg and Adplus. If you don't have Windbg installed, check out my previous post.
2. Open up the dump and load SOS
Open the dump in Windbg and load the SOS extension. You'll find it in the framework directory so if you're debugging an application for Framework 2.0 look in "C:\Windows\Microsoft.NET\Framework\v2.0.50727". Anyway, load SOS by typing:
.load [path]\sos
3. Run dumpheap
Execute the following command:
!dumpheap -stat
It will show statistics for the objects on the heap in a nice little summary divided in tho four columns.
- The method table of the object
- The number of objects of this type on the heap
- The total size of these objects in bytes
- The name of the object type
Be careful not to omit the -stat parameter. If you do then Windbg will dump the address of each object in the entire heap to your screen, which will be a lot of information to say the least.
Analyzing the information provided by !dumpheap
Here's a sample heap from one of my cases...
0:000> !dumpheap -stat
Statistics:
MT Count TotalSize Class Name
7a787cc4 1 12 System.IO.FileSystemWatcher+FSWAsyncResult
7a75904c 1 12 System.Diagnostics.OrdinalCaseInsensitiveComparer
7a7575cc 1 12 System.CodeDom.Compiler.CodeDomConfigurationHandler
7a7571a8 1 12 System.Net.DefaultCertPolicy
7a75661c 1 12 System.Diagnostics.TraceListenerCollection
7a755834 1 12 System.Diagnostics.PerformanceCounterCategoryType
........CONTINUED.........
68a66a88 227,559 12,743,304 System.Web.UI.WebControls.Literal
68a2f7fc 399,272 14,373,792 System.Web.UI.ControlCollection
68a92e2c 768,731 33,824,164 System.Web.UI.Control+OccasionalFields
68a884a0 641,952 38,517,120 System.Web.UI.LiteralControl
79124228 879,515 43,394,976 System.Object[]
790fa3e0 1,431,594 122,806,484 System.String
Total 10,389,625 objects, Total size: 463,313,540
So in this particular dump I have 1,431,594 strings whose total size is 122 MBytes, 879,515 Objects with a total size of 43 MBytes, etc.
Objects in heap may be larger than they appear
The TotalSize column isn't 100% true. Look at the LiteralControls that are on third place. They only use a total of 38 MBytes, or do they? The TotalSize refers to the object structure, but the member variables like strings, integers and other child objects are not included. It kind of makes sense, since otherwise the Total size would be completely off the scale. The LiteralControl objects contain a number of child objects and three of those are strings. Their respective size is listed with the System.String object.
Using !dumpobj
Let's take a closer look at one of the System.Web.UI.LiteralControls. We list all of the controls with the following command !dumpheap -type System.Web.UI.LiteralControl and quickly press Ctrl+Break before the screen fills with too many lines:
0:000> !dumpheap -type System.Web.UI.LiteralControl
Address MT Size Gen
023ea0a8 68a884a0 60 2 System.Web.UI.LiteralControl
023ea0e4 68a884a0 60 2 System.Web.UI.LiteralControl
023ea374 68a884a0 60 2 System.Web.UI.LiteralControl
023ea460 68a884a0 60 2 System.Web.UI.LiteralControl
023ea510 68a884a0 60 2 System.Web.UI.LiteralControl
023eab3c 68a884a0 60 2 System.Web.UI.LiteralControl
........CONTINUED........
023fe31c 68a884a0 60 2 System.Web.UI.LiteralControl
023fe414 68a884a0 60 2 System.Web.UI.LiteralControl
023fe4c4 68a884a0 60 2 System.Web.UI.LiteralControl
023fe500 68a884a0 60 2 System.Web.UI.LiteralControl
As you can see each LiteralControl is 60 bytes in size. Like I said before that's the size of the object structure alone, not it's referenced objects and properties. We now pick the address of one of the LiteralControls and execute the !dumpobj command (!do for short). This gives us the following result:
0:000> !do 023ea0a8
Name: System.Web.UI.LiteralControl
MethodTable: 68a884a0
EEClass: 68a88428
Size: 60(0x3c) bytes
GC Generation: 2
(C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
MT Field Offset Type VT Attr Value Name
790fa3e0 4001fe0 4 System.String 0 instance 00000000 _id
790fa3e0 4001fe1 8 System.String 0 instance 00000000 _cachedUniqueID
68a2af44 4001fe2 c ...em.Web.UI.Control 0 instance 023e8864 _parent
68a91070 4001fe3 2c System.Int32 0 instance 0 _controlState
68a85ea0 4001fe4 10 ...m.Web.UI.StateBag 0 instance 00000000 _viewState
68a2af44 4001fe5 14 ...em.Web.UI.Control 0 instance 023e8864 _namingContainer
68a273d0 4001fe6 18 System.Web.UI.Page 0 instance 01df4514 _page
68a92e2c 4001fe7 1c ...+OccasionalFields 0 instance 00000000 _occasionalFields
68a2b378 4001fe8 20 ...I.TemplateControl 0 instance 00000000 _templateControl
68a14528 4001fe9 24 ...m.Web.VirtualPath 0 instance 00000000 _templateSourceVirtualDirectory
68a8bb48 4001fea 28 ...rs.ControlAdapter 0 instance 00000000 _adapter
68a3a8f8 4001feb 30 ...SimpleBitVector32 1 instance 023ea0d8 flags
790f9c18 4001fda c70 System.Object 0 shared static EventDataBinding
>> Domain:Value 000f0d00:NotInit 0011a720:01df0028 <<
790f9c18 4001fdb c74 System.Object 0 shared static EventInit
>> Domain:Value 000f0d00:NotInit 0011a720:01df0034 <<
790f9c18 4001fdc c78 System.Object 0 shared static EventLoad
>> Domain:Value 000f0d00:NotInit 0011a720:01df0040 <<
790f9c18 4001fdd c7c System.Object 0 shared static EventUnload
>> Domain:Value 000f0d00:NotInit 0011a720:01df004c <<
790f9c18 4001fde c80 System.Object 0 shared static EventPreRender
>> Domain:Value 000f0d00:NotInit 0011a720:01df0058 <<
790f9c18 4001fdf c84 System.Object 0 shared static EventDisposed
>> Domain:Value 000f0d00:NotInit 0011a720:01df0064 <<
79124228 4001fec c88 System.Object[] 0 shared static automaticIDs
>> Domain:Value 000f0d00:NotInit 0011a720:01df0070 <<
790fa3e0 4002211 34 System.String 0 instance 02238664 _text
Cool, now we can take a look at the specifics. For example the value of the text-property, which is located in the string down at the bottom with address 02238664. To get it's value, simply perform a !do on the address:
0:000> !do 02238664
Name: System.String
MethodTable: 790fa3e0
EEClass: 790fa340
Size: 158(0x9e) bytes
GC Generation: 2
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: </td>
</tr>
</table>
<!-- end of content table -->
Fields:
MT Field Offset Type VT Attr Value Name
790fed1c 4000096 4 System.Int32 0 instance 71 m_arrayLength
790fed1c 4000097 8 System.Int32 0 instance 70 m_stringLength
790fbefc 4000098 c System.Char 0 instance 3c m_firstChar
790fa3e0 4000099 10 System.String 0 shared static Empty
>> Domain:Value 000f0d00:790d6584 0011a720:790d6584 <<
79124670 400009a 14 System.Char[] 0 shared static WhitespaceChars
>> Domain:Value 000f0d00:01d413b8 0011a720:01d44f80 <<
Okay, so we can see that the string contains some data to close off a table. We can also take a look at the other properties for the object and examine them if we wish. But there is another command that is really useful...
Using !objsize
Is there a way to get the total size of a specific System.Web.UI.LiteralControl? - Simple! We use the !objsize -command. !objsize looks at all the pointers within an object and calculate their total size. Below is the built in documentation for the !objsize -command:
0:000> !help objsize
-------------------------------------------------------------------------------
!ObjSize [<Object address>]
With no parameters, !ObjSize lists the size of all objects found on managed
threads. It also enumerates all GCHandles in the process, and totals the size
of any objects pointed to by those handles. In calculating object size,
!ObjSize includes the size of all child objects in addition to the parent.
For example, !DumpObj lists a size of 20 bytes for this Customer object:
0:000> !do a79d40
Name: Customer
MethodTable: 009038ec
EEClass: 03ee1b84
Size: 20(0x14) bytes
(C:\pub\unittest.exe)
Fields:
MT Field Offset Type Attr Value Name
009038ec 4000008 4 CLASS instance 00a79ce4 name
009038ec 4000009 8 CLASS instance 00a79d2c bank
009038ec 400000a c System.Boolean instance 1 valid
but !ObjSize lists 152 bytes:
0:000> !ObjSize a79d40
sizeof(00a79d40) = 152 ( 0x98) bytes (Customer)
This is because a Customer points to a Bank, has a name, and the Bank points to
an Address string. You can use !ObjSize to identify any particularly large
objects, such as a managed cache in a web server.
Objects in heap may also be smaller than they appear
So what do we get if we run !objsize on our LiteralControl? This is really interesting, because what happens is; the debugger gets really busy for quite some time and eventually we get this:
0:000> !objsize 023ea0a8
sizeof(023ea0a8) = 456918136 ( 0x1b3c0478) bytes (System.Web.UI.LiteralControl)
456 MBytes! How is that possible? Well if you scroll up to where we ran the !do command on the LiteralControl, you'll see that the control holds a reference to the page. The page in turn has a reference to the cache, and before long we'll have referenced almost the entire heap.
Summary
Hopefully this is enough to give you a quick glimpse of what is possible with three relatively simple commands from the sos extension. The commands were:
- !dumpheap
- !dumpobj
- !objsize
Over and out
/ Johan