ASP.NET Memory Leak Case Study: Sessions Sessions Sessions…
In ASP.NET 1.1 as you probably know, there are 3 different locations to store session objects.
In proc which stores session variables in the cache, State server which stores it in the state service and finally SQL Server.
There are of course pros and cons of each but no matter which one you use, you should be careful with how much you store in session state if you want your application to be scalable.
For in-proc session state, storing too much in session means high memory usage. For state server and SQL server you have the performance cost of serializing and de-serializing the session data.
One thing that may not be readily apparent is that if you use either of the out of process storage locations you will be reading in and de-serializing all the session data for a particular user on each web request.
Lot’s of data in out of proc session state may not only cause your performance to suffer, but depending on what you store in session scope you may be suffering a memory hit there too. Take for example a dataset, this will normally be serialized with the binary formatter (unless you don’t create your own dataset surrogate) into an XML format. The problem here is that when using a generic tool to do a specific task a lot of generalizations have to be made.
Say for example that you have a dataset like so:
| ID | CustomerFirstName | CustomerLastName |
| 1 | John | Doe |
| 2 | Jane | Doe |
Instead of just storing the data “1, Jane, Doe, 2, John, Doe” you store xml markup for the dataset, tables, records etc. and each individual data item looks something like <CustomerFirstName>Jane</CustomerFirstName>.
Add to this, that the binary formatter calculates approximately how many bytes it think its going to need to store this and then adds a little (sometimes a lot) to make sure that it wont have to reallocate the buffer.
Long story short… I have seen situations where the buffer used for serialization was around 10 to 15 times as big as the original dataset because of the length of column names vs. actual data etc.
See this article https://support.microsoft.com/default.aspx?scid=kb;en-us;829740 for a more in-depth discussion on the issue.
So if we take a pretty common scenario… You develop an on-line shop application for a customer with relatively few concurrent users. The initial setup is one standalone web server, using in-proc session state, and your application is storing some datasets per user with info about their past orders etc.
Suddenly the application gets more popular and a larger company wants to use it. It needs to work on a multi-server web farm, so you move to SQL Server session state.
What is wrong with this picture???
The original idea of storing the datasets in session scope was probably based on that it would be faster to get the datasets from cache rather than by doing database requests. Now you are stuck in a situation where not only do you still have to retrieve the datasets from a database, but you do it for every single request whether you are going to use it or not on that page… and every time it has to be de-serialized and/or serialized.
Makes you think twice about what you are storing in session state doesn’t it:)
Let’s take a look at a case study…
Problem description
The memory usage for the ASP.NET process is very high (800MB – 1GB), sometimes resulting in out of memory exceptions or unwanted recycles of the process.
Gathering data
As in the case of the event handlers post I would suggest multiple dumps as the memory increases, but even if that is not possible, a dump when the memory is very high will give us a good clue about what is going on.
Debugging
The dump size is 1.473.913 bytes so we are using close to 1.4 GB private bytes which is pretty excessive.
Usually one of the first things I do is to run !eeheap –gc to get a feel for if most of the memory is on the managed heap, in which case I can for the most part rule out native leaks or problems with the loader heap.
0:023> !eeheap -gc
Number of GC Heaps: 2
------------------------------
Heap 0 (0x000b7198)
generation 0 starts at 0x022104d4
generation 1 starts at 0x022037c0
generation 2 starts at 0x02170030
ephemeral segment allocation context: none
segment begin allocated size
0x2170000 0x2170030 0x224a4e0 0xda4b0(894,128)
Large object heap starts at 0x0a170030
segment begin allocated size
0x0a170000 0x0a170030 0x0acf0b20 0x00b80af0(12,061,424)
0x0d490000 0x0d490030 0x0e3d2450 0x00f42420(16,000,032)
0x12010000 0x12010030 0x12f52460 0x00f42430(16,000,048)
0x13010000 0x13010030 0x13f52460 0x00f42430(16,000,048)
0x15010000 0x15010030 0x15f52460 0x00f42430(16,000,048)
0x1a010000 0x1a010030 0x1af52460 0x00f42430(16,000,048)
…
0x71ca0000 0x71ca0030 0x72be2470 0x00f42440(16,000,064)
0x748b0000 0x748b0030 0x757f2470 0x00f42440(16,000,064)
0x7d0e0000 0x7d0e0030 0x7d881250 0x007a1220(8,000,032)
Heap Size 0x2d5b4e10(760,958,480)
------------------------------
Heap 1 (0x000ede88)
generation 0 starts at 0x06249b58
generation 1 starts at 0x0623e190
generation 2 starts at 0x06170030
ephemeral segment allocation context: none
segment begin allocated size
0x6170000 0x6170030 0x6283b64 0x113b34(1,129,268)
Large object heap starts at 0x0b170030
segment begin allocated size
0x0b170000 0x0b170030 0x0b9f1c90 0x00881c60(8,920,160)
0x0c3e0000 0x0c3e0030 0x0d322460 0x00f42430(16,000,048)
0x0e490000 0x0e490030 0x0f3d2460 0x00f42430(16,000,048)
0x11010000 0x11010030 0x11f52460 0x00f42430(16,000,048)
0x14010000 0x14010030 0x14f52450 0x00f42420(16,000,032)
0x16010000 0x16010030 0x16f52480 0x00f42450(16,000,080)
0x17010000 0x17010030 0x17b81b60 0x00b71b30(12,000,048)
0x18010000 0x18010030 0x18b81b60 0x00b71b30(12,000,048)
0x19010000 0x19010030 0x19f52460 0x00f42430(16,000,048)
0x1b010000 0x1b010030 0x1bf52460 0x00f42430(16,000,048)
…
0x61010000 0x61010030 0x61f52470 0x00f42440(16,000,064)
0x62db0000 0x62db0030 0x63cf2470 0x00f42440(16,000,064)
0x657e0000 0x657e0030 0x66722470 0x00f42440(16,000,064)
0x685c0000 0x685c0030 0x69502470 0x00f42440(16,000,064)
0x6e110000 0x6e110030 0x6ec81b70 0x00b71b40(12,000,064)
0x72ca0000 0x72ca0030 0x73be2470 0x00f42440(16,000,064)
0x77ff0000 0x77ff0030 0x78f32470 0x00f42440(16,000,064)
0x7e0e0000 0x7e0e0030 0x7f022470 0x00f42440(16,000,064)
Heap Size 0x286a4124(678,052,132)
------------------------------
GC Heap Size 0x55c58f34(1,439,010,612)
So !eeheap –gc tells us that a) the GC Heap Size is around 1.4 GB which is very close to the total amount of memory, meaning that most of our memory is on the managed heap, and b) that most of our memory is in the large object segments, i.e. objects over 85000 bytes.
I have shortened the output here to save space. In the original output we had around a hundred large object segments. In fact this example is a bit exaggerated, it would be very rare to see this situation (hardly anything in the small-object segments, and tonnes in the large object segments), but the case study should still give you an idea of how to troubleshoot these types of issues in the more general case.
Naturally from here the next step is to find out what is on the large object heaps.
We can start off with a summary of the objects stored on the large object heap, to get a feel for what objects we are looking for.
Since the objects stored on the large object heap are all 85000 bytes or above we can dump out the large object heaps by using the –min 85000 switch for !dumpheap, and then –stat to show just the statistics.
0:023> !dumpheap -min 85000 -stat
Using our cache to search the heap.
Statistics:
MT Count TotalSize Class Name
0x000eda20 1 920,144 Free
0x01b2209c 33 132,000,528 System.Object[]
0x01b226b0 163 1,304,001,956 System.Int32[]
Total 197 objects, Total size: 1,436,922,628
Most of the memory is in Int32 arrays but a good chunk (around 132 MB) is used for byte arrays.
We should note here that this is just for the structures (i.e. the arrays themselves). The 132 MB does not include the size of the objects stored in the Object arrays, same for the Int32 arrays. To find out the actual size of each object array or int32 array including the objects stored in them we would have to run !objsize.
The next step is to take a closer look at some of the individual arrays more closely by dumping all objects on the large object heap (without the –stat switch).
0:023> !dumpheap -min 85000
Using our cache to search the heap.
Address MT Size Gen
0x0b170030 0x000eda20 920,144 -1 Free
0x0a920210 0x01b2209c 4,000,016 -1 System.Object[]
0x247b1250 0x01b2209c 4,000,016 -1 System.Object[]
0x277b1250 0x01b2209c 4,000,016 -1 System.Object[]
0x287b1250 0x01b2209c 4,000,016 -1 System.Object[]
…
0x51af0030 0x01b226b0 8,000,012 -1 System.Int32[]
0x52291250 0x01b226b0 8,000,012 -1 System.Int32[]
0x53b40950 0x01b226b0 8,000,012 -1 System.Int32[]
0x56620030 0x01b226b0 8,000,012 -1 System.Int32[]
0x57620030 0x01b226b0 8,000,012 -1 System.Int32[]
0x5a440030 0x01b226b0 8,000,012 -1 System.Int32[]
0x5abe1250 0x01b226b0 8,000,012 -1 System.Int32[]
0x5c660030 0x01b226b0 8,000,012 -1 System.Int32[]
0x5ce01250 0x01b226b0 8,000,012 -1 System.Int32[]
0x61010030 0x01b226b0 8,000,012 -1 System.Int32[]
0x617b1250 0x01b226b0 8,000,012 -1 System.Int32[]
0x62db0030 0x01b226b0 8,000,012 -1 System.Int32[]
0x63551250 0x01b226b0 8,000,012 -1 System.Int32[]
0x657e0030 0x01b226b0 8,000,012 -1 System.Int32[]
0x65f81250 0x01b226b0 8,000,012 -1 System.Int32[]
Picking one of the Int32 arrays at random, we can do !gcroot on it to find out where it is rooted to find out why it won’t get garbage collected.
0:023> !gcroot 0x72ca0030
Scan Thread 16 (0xbd8)
ESP:1a3f5e0:Root:0x6280d38(System.Web.HttpContext)->
0x62809ec(System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6)->
0x61b7030(System.Web.HttpWorkerRequest/EndOfSendNotification)->
0x61b47d4(System.Web.HttpRuntime)->
0x61b4ca0(System.Web.Caching.CacheMultiple)->0x61b4cc4(System.Object[])->
0x61b4cdc(System.Web.Caching.CacheSingle)->
0x61b4dac(System.Web.Caching.CacheExpires)->0x61b4ff8(System.Object[])->
0x61b5ab8(System.Web.Caching.ExpiresBucket)->
0x222e63c(System.Web.Caching.ExpiresEntry[])->
0x220292c(System.Web.Caching.CacheEntry)->
0x22028fc(System.Web.SessionState.InProcSessionState) ->
0x2202690(System.Web.SessionState.SessionDictionary)->
0x220273c(System.Collections.Hashtable)->
0x2202770(System.Collections.Hashtable/bucket[])->
0x2202810(System.Collections.Specialized.NameObjectCollectionBase/NameObjectEntry)->
0x72ca0030(System.Int32[])
Scan Thread 20 (0x89c)
Scan Thread 22 (0xa5c)
Scan HandleTable 0xdc9d8
Scan HandleTable 0xea6e8
Scan HandleTable 0x1531e8
The root chain tells us that this particular Int32 array is rooted in cache, more specifically in the InProcSessionState object, indicating that it is stored in in-proc session scope.
Once we get here a search through the code for session assignment, focusing on Int32 arrays may be in order. However if the code base is large or there is another reason we can’t just search the code, we can dig further to get a bit more information.
Debugging Tip: From the HttpContext object above you can information about the application that the particular session object belongs to (path etc) if you have many different applications.
Taking a look at the cache (where the sessions are stored) using !dumpaspnetcache –stat we get this
0:023> !dumpaspnetcache -stat
Going to dump the ASP.NET Cache.
MT Count TotalSize Class Name
0x0211cc9c 1 20 System.Web.Security.FileSecurityDescriptorWrapper
0x020c242c 2 56 System.Web.UI.ParserCacheItem
0x0206c66c 5 260 System.Web.Configuration.HttpConfigurationRecord
0x0c2e7014 1 316 System.Web.Mobile.MobileCapabilities
0x79b94638 4 376 System.String
0x0c2eaeb4 151 7,248 System.Web.SessionState.InProcSessionState
Total 164 objects, Total size: 8,276
So we have 151 concurrent sessions (each InProcSessionState object holds the objects for one session)
To find out how much is stored in them we can use a for each loop using the method table for InProcSessionState from above and run !objsize on each InProcSessionState object.
0:023> .foreach (obj {!dumpheap -mt 0x0c2eaeb4 -short}){!objsize ${obj}}
sizeof(0x22028fc) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x2202a10) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x2202cfc) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x2202fe8) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x22032d4) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x22035c0) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x2203a38) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x2203d24) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x2204010) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
…
sizeof(0x6248080) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6248334) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62485e8) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x624a84c) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x624d7b8) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6250724) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6253690) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62565fc) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6259568) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x625c4d4) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x625f440) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62623ac) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6265318) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6268284) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x626b1b8) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
Wow, 151 sessions and each holds about 8-12 MB of data, yeah that could certainly cause a memory issue:)
Let’s pick one of them and see what we actually store in session …
0:023> !do 0x626b1b8
Name: System.Web.SessionState.InProcSessionState
MethodTable 0x0c2eaeb4
EEClass 0x0c1c5660
Size 48(0x30) bytes
GC Generation: 0
mdToken: 0x02000132 (c:\windows\assembly\gac\system.web\1.0.5000.0__b03f5f7f11d50a3a\system.web.dll)
FieldDesc*: 0x0c2eae0c
MT Field Offset Type Attr Value Name
0x0c2eaeb4 0x40009f2 0x4 CLASS instance 0x06269f74 dict
0x0c2eaeb4 0x40009f3 0x8 CLASS instance 0x00000000 staticObjects
0x0c2eaeb4 0x40009f4 0xc System.Int32 instance 20 timeout
0x0c2eaeb4 0x40009f5 0x18 System.Boolean instance 0 isCookieless
0x0c2eaeb4 0x40009f6 0x10 System.Int32 instance 0 streamLength
0x0c2eaeb4 0x40009f7 0x19 System.Boolean instance 0 locked
0x0c2eaeb4 0x40009f8 0x1c VALUETYPE instance start at 0x0626b1d4 utcLockDate
0x0c2eaeb4 0x40009f9 0x14 System.Int32 instance 1 lockCookie
0x0c2eaeb4 0x40009fa 0x24 VALUETYPE instance start at 0x0626b1dc spinLock
Each InProcSessionState object has a dict (dictionary) member variable that holds the actual session objects in its _entriesArray.
0:023> !do 0x06269f74
Name: System.Web.SessionState.SessionDictionary
MethodTable 0x0c2e0c54
EEClass 0x0c1c1308
Size 44(0x2c) bytes
GC Generation: 0
mdToken: 0x0200013b (c:\windows\assembly\gac\system.web\1.0.5000.0__b03f5f7f11d50a3a\system.web.dll)
FieldDesc*: 0x0c2e0b30
MT Field Offset Type Attr Value Name
0x0206b338 0x4000a8b 0x24 System.Boolean instance 0 _readOnly
0x0206b338 0x4000a8c 0x4 CLASS instance 0x06269fb8 _entriesArray
0x0206b338 0x4000a8d 0x8 CLASS instance 0x06269fa0 _hashProvider
0x0206b338 0x4000a8e 0xc CLASS instance 0x06269fac _comparer
0x0206b338 0x4000a8f 0x10 CLASS instance 0x0626a020 _entriesTable
0x0206b338 0x4000a90 0x14 CLASS instance 0x00000000 _nullKeyEntry
0x0206b338 0x4000a91 0x18 CLASS instance 0x00000000 _keys
0x0206b338 0x4000a92 0x1c CLASS instance 0x00000000 _serializationInfo
0x0206b338 0x4000a93 0x20 System.Int32 instance 4 _version
0x0c2e0c54 0x4000a0f 0x25 System.Boolean instance 1 _dirty
0x0c2e0c54 0x4000a0e 0 CLASS shared static s_immutableTypes
>> Domain:Value 0x000dad08:NotInit 0x00104f30:0x021be0dc <<
0:023> !do 0x06269fb8
Name: System.Collections.ArrayList
MethodTable 0x79ba2ee4
EEClass 0x79ba3020
Size 24(0x18) bytes
GC Generation: 0
mdToken: 0x02000100 (c:\windows\microsoft.net\framework\v1.1.4322\mscorlib.dll)
FieldDesc*: 0x79ba3084
MT Field Offset Type Attr Value Name
0x79ba2ee4 0x4000362 0x4 CLASS instance 0x06269fd0 _items
0x79ba2ee4 0x4000363 0xc System.Int32 instance 3 _size
0x79ba2ee4 0x4000364 0x10 System.Int32 instance 3 _version
0x79ba2ee4 0x4000365 0x8 CLASS instance 0x00000000 _syncRoot
The entries array is an ArrayList, and to print out all the objects in an arraylist in one shot we can run !do –v on it’s _items member variable
0:023> !do -v 0x06269fd0
Name: System.Object[]
MethodTable 0x01b2209c
EEClass 0x01b22018
Size 80(0x50) bytes
GC Generation: 0
Array: Rank 1, Type CLASS
Element Type: System.Object
Content: 16 items
------ Will only dump out valid managed objects ----
Address MT Class Name
0x0626a81c 0x0206b784 System.Collections.Specialized.NameObjectCollectionBase/NameObjectEntry
0x0626a82c 0x0206b784 System.Collections.Specialized.NameObjectCollectionBase/NameObjectEntry
0x0626a83c 0x0206b784 System.Collections.Specialized.NameObjectCollectionBase/NameObjectEntry
----------
We can either dump out these objects individually or we can let the debugger do the work for us.
If we use !do –v –short instead of just !do –v, we just get the addresses of the NameObject entries so we can use .foreach on them.
0:023> !do -v 0x06269fd0 -short
0x0626a81c
0x0626a82c
0x0626a83c
In this specific case the .foreach might seem like overkill but it’s good to know how to do it for larger collections.
For each object, we print out the session variable name at offset 0x4, the object at offset 0x8 and the size of the object.
(To learn more about how this works, read the post on “how much are you caching”)
0:023> .foreach (obj {!do -v 0x06269fd0 -short}){.echo ***;!do poi(${obj}+0x4);!do poi(${obj}+0x8);!objsize ${obj}}
***
String: somestring
String: this is a string i stored in session scope
sizeof(0x626a81c) = 160 ( 0xa0) bytes (System.Collections.Specialized.NameObjectCollectionBase/NameObjectEntry)
***
String: alargeintarray
Name: System.Int32[]
MethodTable 0x01b226b0
EEClass 0x01b22638
Size 8000012(0x7a120c) bytes
GC Generation: 3
Array: Rank 1, Type System.Int32
Element Type: System.Int32
Content: 2,000,000 items
sizeof(0x626a82c) = 8,000,076 (0x7a124c) bytes (System.Collections.Specialized.NameObjectCollectionBase/NameObjectEntry)
***
String: sometimesbig
Name: System.Object[]
MethodTable 0x01b2209c
EEClass 0x01b22018
Size 4000016(0x3d0910) bytes
GC Generation: 3
Array: Rank 1, Type CLASS
Element Type: System.Object
Content: 1,000,000 items
sizeof(0x626a83c) = 4,000,076 (0x3d094c) bytes (System.Collections.Specialized.NameObjectCollectionBase/NameObjectEntry)
So for this user we are storing a pretty small string “this is a string i stored in session scope“ in Session[“somestring”], an 8 MB Int32 array in Session[“alargeintarray”] and a 4 MB object array in Session[“sometimesbig”].
At this point we have the issue pretty well defined and the task will be how to reduce the sizes of these objects, but that I will leave to the developers:) My work here is done:)
Based on the number of issues I have seen with high memory caused by storing too much in cache or sessions, I have made it a standard step in my troubleshooting of high mem issues to look at the size and contents of the cache and session state.
Side note
If the number of users on your site is pretty constant but the number of concurrent sessions (in performance monitor) keeps increasing abnormally, you may be running in to a problem with timers not firing https://support.microsoft.com/kb/900822/en-us. You can obtain this hotfix by contacting Microsoft Product Support.
Over and out…