Debugging Script: Dumping out ASP.NET Session Contents


In my last post I wrote a script to dump out all the ASP.NET requests on the heap.  Since one of the most common memory issues I encounter is too much cache or session state I figured that showing you how to retreive session data would be good.


A word of caution, since this script uses !dumpheap (to dump out the objects on the heap) and !objsize (to figure out the size of an object including the size of its membervariables) it may take a long time to execute if you have a very large dump, which is typically the case when you are worried about how much you store in session state. 


There is a lot more data that you can get, other than what I am displaying, and the script can pretty easily be modified to display cache objects instead so if you find some cool use or modification of it, please post it in the comments so all of us can benefit from it.


Note: This script only works if you use in-proc session state since it is doing its magic by dumping out all the System.Web.SessionState.InProcSessionState items on the heap.  On the other hand, if you have high memory usage due to session state it is probably the in-proc session state items you will be worried about.  Btw, if this is what you are running into, check out the post index for other posts on ASP.NET Session memory issues.


Output and comments on output


This script is divided into two parts. 


Part #1 (DumpSessions.txt) displays all the System.Web.SessionState.InProcSessionState items with the session address as a DML (Debugger Markup Language) link so that you can click it to get more detailed info.



So in this case we have 3 active sessions, or at least 3 sessions that are still on the heap, the session timeout is set to 20 minutes and the first session holds an amazing ~268 MB (actually more like 256,001 MB, but what’s a few MB in the sceme of things).


Part #2 (DumpSessionVar.txt) dumps out more detailed information about each sessions, i.e. the session variables for that particular session and this is what gets executed when you click the DML link.



So in this case the 256 MB session contains 3 session variables.  UserName = Joe Smith (88 Bytes), UserType = User (80 Bytes) and ProductsOrdered = ArrayList (268,435,600 bytes).   The address of the object stored in session is printed so you can dump it out or explore it further…


An idea for a modification of the script would be to print out all the session variables at once, or add some parsing to get the total size of all sessions. Oh well, there is endless possiblities here…


Finding the data


All in-proc sessions are represented on the heap as System.Web.SessionState.InProcSessionState objects.

0:013> !do 03093d38
Name: System.Web.SessionState.InProcSessionState
MethodTable: 663ac580
EEClass: 663ac510
Size: 48(0x30) bytes
GC Generation: 1
(C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
MT Field Offset Type VT Attr Value Name
663ecfbc 4001edc 4 …ateItemCollection 0 instance 03093b18 _sessionItems
663a3390 4001edd 8 …ObjectsCollection 0 instance 00000000 _staticObjects
791018e0 4001ede c System.Int32 1 instance 20 _timeout
79107584 4001edf 18 System.Boolean 1 instance 0 _locked
791084f8 4001ee0 1c System.DateTime 1 instance 03093d54 _utcLockDate
791018e0 4001ee1 10 System.Int32 1 instance 1 _lockCookie
663aad40 4001ee2 24 …ReadWriteSpinLock 1 instance 03093d5c _spinLock
791018e0 4001ee3 14 System.Int32 1 instance 0 _flags

and given a particular System.Web.SessionState.InProcSessionState object we can get to the information we want like this…


Size of the session  

0:013> !objsize 03093d38
sizeof(03093d38) = 268,437,272 ( 0x10000718) bytes (System.Web.SessionState.InProcSessionState)

Individual Session Variables


We get to the individual session variables by dumping out the _sessionItems member variable, following the steps below…


Note, the number of session variables in the session is available from the _size membervariable of the ArrayList 

0:013> !do 03093b18
Name: System.Web.SessionState.SessionStateItemCollection
MethodTable: 66411bb4
EEClass: 66411b3c
Size: 60(0x3c) bytes
GC Generation: 2
(C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
MT Field Offset Type VT Attr Value Name
79107584 4001173 24 System.Boolean 1 instance 0 _readOnly
79105cd4 4001174 4 …ections.ArrayList 0 instance 03093b60 _entriesArray
791138d4 4001175 8 …IEqualityComparer 0 instance 03049ce0 _keyComparer
79101634 4001176 c …ections.Hashtable 0 instance 03093b78 _entriesTable
7a761668 4001177 10 …e+NameObjectEntry 0 instance 00000000 _nullKeyEntry
7a78e3b0 4001178 14 …se+KeysCollection 0 instance 00000000 _keys
7910ec98 4001179 18 …SerializationInfo 0 instance 00000000 _serializationInfo
791018e0 400117a 20 System.Int32 1 instance 4 _version
790fc35c 400117b 1c System.Object 0 instance 00000000 _syncRoot
79115538 400117c 538 …em.StringComparer 0 shared static defaultComparer
>> Domain:Value 0019b498:NotInit 001cc780:NotInit <<
79107584 4001f2b 25 System.Boolean 1 instance 1 _dirty
664228fc 4001f2c 28 …n+KeyedCollection 0 instance 00000000 _serializedItems
79100f74 4001f2d 2c System.IO.Stream 0 instance 00000000 _stream
791018e0 4001f2e 34 System.Int32 1 instance 0 _iLastOffset
790fc35c 4001f2f 30 System.Object 0 instance 03093b54 _serializedItemsLock
79101634 4001f2a c54 …ections.Hashtable 0 shared static s_immutableTypes
>> Domain:Value 0019b498:NotInit 001cc780:0708be1c <<

0:013> !do 03093b60
Name: System.Collections.ArrayList
MethodTable: 79105cd4
EEClass: 79105c28
Size: 24(0x18) bytes
GC Generation: 2
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Fields:
MT Field Offset Type VT Attr Value Name
7912ad90 40008df 4 System.Object[] 0 instance 03093c90 _items
791018e0 40008e0 c System.Int32 1 instance 3 _size
791018e0 40008e1 10 System.Int32 1 instance 3 _version
790fc35c 40008e2 8 System.Object 0 instance 00000000 _syncRoot
7912ad90 40008e3 1c0 System.Object[] 0 shared static emptyArray
>> Domain:Value 0019b498:03031ff8 001cc780:030371e4 <<

0:013> dc 03093c90
03093c90 7912ad90 00000004 790fc35c 03093c80 …y….\..y.<..
03093ca0 03093cb0 03093ce4 00000000 00000000 .<…<……….
03093cb0 7a761668 030939c8 03093c64 00000000 h.vz.9..d<……
03093cc0 79105cd4 31180038 00000000 020bb7a8 .\.y8..1……..
03093cd0 020bb7a8 00000000 00194d40 00000000 ……..@M……
03093ce0 00000000 7a761668 03093a14 03093cc0 ….h.vz.:…<..
03093cf0 00000000 790fcb30 0000001a 00000019 ….0..y……..
03093d00 0061006a 006e0030 00710078 007a006e j.a.0.n.x.q.n.z.

0:013> !do 03093c80
Name: System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry
MethodTable: 7a761668
EEClass: 7a7ce36c
Size: 16(0x10) bytes
GC Generation: 2
(C:\WINDOWS\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll)
Fields:
MT Field Offset Type VT Attr Value Name
790fcb30 400117d 4 System.String 0 instance 030939a4 Key
790fc35c 400117e 8 System.Object 0 instance 03093c40 Value

0:013> !do 030939a4
Name: System.String
MethodTable: 790fcb30
EEClass: 790fca90
Size: 34(0x22) bytes
GC Generation: 2
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: UserName
Fields:
MT Field Offset Type VT Attr Value Name
791018e0 4000096 4 System.Int32 1 instance 9 m_arrayLength
791018e0 4000097 8 System.Int32 1 instance 8 m_stringLength
790fe534 4000098 c System.Char 1 instance 55 m_firstChar
790fcb30 4000099 10 System.String 0 shared static Empty
>> Domain:Value 0019b498:790d81bc 001cc780:790d81bc <<
7912b1d8 400009a 14 System.Char[] 0 shared static WhitespaceChars
>> Domain:Value 0019b498:030303f4 001cc780:03034558 <<

0:013> !do 03093c40
Name: System.String
MethodTable: 790fcb30
EEClass: 790fca90
Size: 36(0x24) bytes
GC Generation: 2
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: Joe Smith
Fields:
MT Field Offset Type VT Attr Value Name
791018e0 4000096 4 System.Int32 1 instance 10 m_arrayLength
791018e0 4000097 8 System.Int32 1 instance 9 m_stringLength
790fe534 4000098 c System.Char 1 instance 4a m_firstChar
790fcb30 4000099 10 System.String 0 shared static Empty
>> Domain:Value 0019b498:790d81bc 001cc780:790d81bc <<
7912b1d8 400009a 14 System.Char[] 0 shared static WhitespaceChars
>> Domain:Value 0019b498:030303f4 001cc780:03034558 <<
 

Running the scripts


To run the scripts copy the text below and put it in two text files somewhere on your harddrive.  I’ve put mine in c:\tools\extensions\DumpSessions.txt and c:\tools\extensions\DumpSessionVar.txt.   If you put them somewhere else, make sure you change the path in the DumpSessions.txt script so that you get to the right script file when you click the DML links.  


To run it, open up your dump in windbg and load sos.dll, and then run $><c:\tools\extensions\DumpSessions.txt


Script code


DumpSessions.txt 

$$
$$ Dumps all sessions on the heap
$$
$$
$$ Written by: Tess
$$
$$ Run as: $><c:\tools\extensions\DumpSessions.txt
$$

$$ CLEAR ALL ALIASES (VARIABLES)
$$ ———————————————————————————-
ad /q *

r @$t0=0;
r @$t1=0;

$$ GET ALL SESSION ITEMS
.foreach (CurrentSession {!dumpheap -type System.Web.SessionState.InProcSessionState -short}){
$$ Increment # of sessions
r @$t0 = @$t0+1

.printf /D “Session Address:\t<?dml?><exec cmd=\”$$>a< c:\\tools\\extensions\\DumpSessionVar.txt ${CurrentSession}\”>${CurrentSession}</exec>\n”;
.printf “Session Timeout:\t%d\n”, poi(${CurrentSession}+0xc);
.foreach /pS 2 /ps 99 (token {!objsize ${CurrentSession}}){.printf “Session Size:\t${token} bytes\n”}
.printf “_______________________________________________\n”;
}

.printf “Number of Sessions: %d\n\n\n\n\n\n\n\n\n”, @$t0;

 


 DumpSessionVar.txt

$$
$$ Dumps details of a particular session
$$
$$
$$ Written by: Tess
$$
$$ Run as: $$>a<c:\tools\extensions\DumpSessionVar.txt SessionAddress
$$

.printf “Session:\t\t${$arg1}\n”;
.foreach /pS 2 /ps 99 (token {!objsize ${$arg1}}){.printf “Session Size:\t${token} bytes\n”}

$$ Number of session variables
r @$t0 = poi(poi(poi(${$arg1}+0x4)+0x4)+0xc);


.for(r @$t1=0; @$t1 < @$t0; r @$t1=@$t1+1){
.printf “\n================================================\n”
.printf “Session Variable\n”
.printf “================================================\n”

$$ NameObject Entry (actual session variable)
r @$t2 = poi(poi(poi(poi(${$arg1}+0x4)+0x4)+0x4)+0x8+0x4*(@$t1+1));
$$ Size
.foreach /pS 2 /ps 99 (token {!objsize @$t2}){.printf “Sessionvar Size:\t${token} bytes\n”}
$$ Key
.foreach /pS 5 (tk {.foreach /pS 9 (token {!do -nofields poi(@$t2+0x4)}){.printf “${token} “}}) {
.printf /D “Session Key:\t<?dml?><col fg=\”emphfg\”>${tk}</col>\n”
};
$$ Value
.printf “Session Variable:\t%p\n****\n”, poi(@$t2+0x8);
!do poi(@$t2+0x8)
}

.printf “\n\n\n\n\n\n\n\n\n”;
ad /q *


 


Final words


If you have any ideas for scripts that you think would be useful please let me know and if I have the time and energy I will probably implement and post them…


Have fun,
Tess 








Comments (19)

  1. Link Listing – September 18, 2007

  2. This is very good info, Tess, thanks !

    I don’t have any suggestions for useful scripts right now, but I do have two questions that have been nagging me since your blog got me started using SoS :

    1) How does !objsize exactly work and how reliable is it ? It seems you could have many sessions that each report being 100 MB in size while in fact they are all pointing to the same 100 MB block of data ?

    2) Is there an easy way to filter out unreachable objects from !dumpheap etc. ? I’m asking this because I once had a situation where some code was creating hundreds of thousands of objects very fast, and every dump was full of these – it was very difficult to discern which were temporary and which were actual "leaks". I know of !gcroot but it’s slow and not really practical to run it on every object. Ideally there would be a !collect command that would force or simulate a GC on the current heap. But perhaps I am thinking of this in entirely the wrong terms…

    Well that’s it, if you feel like addressing either or both I’d be grateful. Otherwise it’s no matter, your blog has already been of tremendous help.

    Cheers,

    –Jonathan

  3. Tess says:

    Hi Jonathan,

    1.  You are completely right, you cant add upp the result of !objsize since it could potentially be pointing to the same data. i.e. if two items (a and b) both have a pointer to the same object (c) !objsize for a would include the size of c and !objsize for b would also include the size of c.

    However if you have a toplevel object d, pointing to a and b, then the size of c and it’s membervariables should only be counted once.  

    The same thing is true if you have a double-link, i.e a->c and c->  (eg. a control points to the parent page and the page has a list of controls which point to the control)

    In this case the size of the control will only be counted once…

    So in short, the !objsize of an item is pretty reliable, but the sum of all objsizes will exceed the total amount of memory on the heap.

    2.  No, there isn’t an easy way to do this, and yes, gcroot usually takes a long time because it has to traverse a lot of objects…

    The only recommendation I can make there is to run

     GC.Collect

     GC.WaitForPendingFinalizers

     GC.Collect

    before taking the dump, if you are in a testing environment.  That way you will clean out as much as possible, leaving you with almost only reacheable objects.

    Hope this helps,

    Tess

  4. Here is the latest in my link-listing series . Also check out my ASP.NET Tips, Tricks and Tutorials page

  5. ASP.NET says:

    Here is the latest in my link-listing series . Also check out my ASP.NET Tips, Tricks and Tutorials page

  6. Great blog.

    Somehow in Java it seem to be much easier…

  7. Niktu says:

    Awww, my eyes!!!

    Never again textual screenshoots in low quality jpeg, please! :D

    Png or Gif heve their best use right in such situations :)

    … after manadatory amount of complaining …

    Great info, Thanks!!!

  8. Tess says:

    :) I completely see what you’re saying… I’ll update with better pics shortly…

  9. Tess says:

    Niktu,

    I found the problem with the pics, turns out that the tool i am using for blogging didn’t choose to link to the external source but rather stored the pics on the blog but after having turned them into some low res thumbs of the pics.  Now it should look a lot better:)  I didn’t realize this was happening so the pic quality bothered me too, so i’m glad i figured it out…

  10. parag medsinge says:

    I was trying to use the script dumpsessions.txt , i have win2000 Advance server on my machine

    I m getting following error :

    Address expression missing from ‘:AspnetdumpsScriptsDumpsessions.txt’

    What will be the possible reason… Its urgent

  11. Tess says:

    sounds like the path to your script may be incorrect, perhaps you could post the command you use to load the script so that me or someone else reading the blog can take a look to figure out whats wrong

  12. The purpose of my presentation was to show some common pitfalls and of course to show off windbg and

  13. womp says:

    Tess,

    Thanks so much for your awesome labs/tutorials.  I went through all of them and have gotten to the point where I’m trying to debug my own asp.net sessions.

    Running these scripts seems to corrupt my dump.  When I run DumpSessions.txt I get the following:

    0:000> $$><DumpSessions.txt

    Session Address: 015c8420

    Session Size: 208 bytes

    _______________________________________________

    Session Address: 017e05a8

    Session Size: 208 bytes

    _______________________________________________

    Session Address: 017ed374

    Session Size: 208 bytes

    _______________________________________________

    Session Address: 01823aa4

    Session Size: 208 bytes

    _______________________________________________

    Session Address: 01826570

    Session Size: 208 bytes

    _______________________________________________

    Session Address: 05693848

    Session Size: 6466088 bytes

    _______________________________________________

    Session Address: 05859788

    Session Size: 208 bytes

    _______________________________________________

    Session Address: ——————————

    Session Size: 1 bytes

    _______________________________________________

    This last entry will hang until I break in Windbg.  If I try to run DumpSessionVar on my large session, I get a few good variable outputs and then it starts dumping

    ================================================

    Session Variable

    ================================================

    Sessionvar Size: not bytes

    Session Variable: 0167e128

    ****

    <Note: this object has an invalid CLASS field>

    Invalid object

    If I try to run either of the scripts, or even a !dumpheap after that, I get the following error message:

    0:018> !dumpheap -stat

    The garbage collector data structures are not in a valid state for traversal. It is either in the "plan phase," where objects are being moved around, or we are at the initialization or shutdown of the gc heap. Commands related to displaying, finding or traversing objects as well as gc heap segments may not work properly. !dumpheap and !verifyheap may incorrectly complain of heap consistency errors.

    Error requesting GC Heap data

    Unable to build snapshot of the garbage collector state

    Any idea?  My dump is from an IIS5 aspnet_wp.exe process with .net 2.0 runtime…

  14. Tess says:

    Hi Womp,

    based on the output of the 2nd script it looks like your version is not matching the version that the script is written for so the offsets are a little bit off… so you would have to change the offsets in the scripts to match your version…

    either way, to get back to a good state you can unload and reload the sos extension, i.e.   .unload sos (or path to sos) and then .load sos (or path to sos) again

  15. womp says:

    Well I figured out the first part of the problem…

    !dumpheap … -short was printing out a line of "———" at the end of the output, which was being taken as one of the sessions in the for loop.  Here’s some fixed up code for the script:

    .foreach (CurrentSession {!dumpheap -type System.Web.SessionState.InProcSessionState -short}){

     .if (‘${CurrentSession}’ == ‘——————————‘)

     {

      $$.printf /D "Not an Addressn";

     }

    .else

    {

    $$ Increment # of sessions

    r @$t0 = @$t0+1

    .printf /D "Session Address:t<?dml?><exec cmd="$$>a< DumpSessionVar.txt ${CurrentSession}">${CurrentSession}</exec>n";

    .printf "Session Timeout:t%dn", poi(${CurrentSession}+0xc);

    .foreach /pS 2 /ps 99 (token {!objsize ${CurrentSession}}){.printf "Session Size:t${token} bytesn"}

    .printf "_______________________________________________n";

     $$.printf /D "An addressn";

    }

    }

    Now DumpSessionVar.txt gets quite a bit further before bogging down with the invalid class field errors…. but it still doesn’t complete properly and ends up in some kind of corrupted state….

  16. slmoloch says:

    Hello,

    Could you help me? I have problem similar to the problem above:

    0:000> !dumpheap -stat

    The garbage collector data structures are not in a valid state for traversal.

    It is either in the "plan phase," where objects are being moved around, or

    we are at the initialization or shutdown of the gc heap. Commands related to

    displaying, finding or traversing objects as well as gc heap segments may not

    work properly. !dumpheap and !verifyheap may incorrectly complain of heap

    consistency errors.

    Error requesting GC Heap data

    Unable to build snapshot of the garbage collector state

    I am using SOS for .NET 2.0 and dump is got from the same version.

  17. Tess says:

    like the message says you are probably in the middle of a GC in the dump so you can’t traverse the heap and thus we can’t pick up the requests from the heap

  18. A few weeks back me and Micke (one of our Architect Evangelists) had a session at TechDays where we talked

  19. Shantanu says:

    I used both the script seems they are not working. The command executes for sometime and returns no output. I have given full permission to the folders where the script is placed.