High Memory continued – Datatables


So in a previous post, High memory, CPU, or other performance problems with .NET, we began to give some things to look for when dealing with High Memory.  So what if debug isn’t set and we aren’t using debug modules?


So the best thing to do first is to try to figure out where all the memory is.  For example, it could be in managed (.NET memory) or it could be native, or we could have a problem like Dynamic Assemblies and what to do about them that makes it look like high memory but it really isn’t the cause of our issue.


The best command to use to look at memory as a whole is !address -summary.  It will give you output that looks like:


 

0:000> !address -summary

——————– Usage SUMMARY ————————–
TotSize ( KB) Pct(Tots) Pct(Busy) Usage
4a434000 ( 1216720) : 58.02% 89.42% : RegionUsageIsVAD
2cf36000 ( 736472) : 35.12% 00.00% : RegionUsageFree
3d0b000 ( 62508) : 02.98% 04.59% : RegionUsageImage
d80000 ( 13824) : 00.66% 01.02% : RegionUsageStack
36000 ( 216) : 00.01% 00.02% : RegionUsageTeb
41c2000 ( 67336) : 03.21% 04.95% : RegionUsageHeap
0 ( 0) : 00.00% 00.00% : RegionUsagePageHeap
1000 ( 4) : 00.00% 00.00% : RegionUsagePeb
1000 ( 4) : 00.00% 00.00% : RegionUsageProcessParametrs
1000 ( 4) : 00.00% 00.00% : RegionUsageEnvironmentBlock
Tot: 7fff0000 (2097088 KB) Busy: 530ba000 (1360616 KB)

——————– Type SUMMARY ————————–
TotSize ( KB) Pct(Tots) Usage
2cf36000 ( 736472) : 35.12% :
3d0b000 ( 62508) : 02.98% : MEM_IMAGE
c10000 ( 12352) : 00.59% : MEM_MAPPED
4e79f000 ( 1285756) : 61.31% : MEM_PRIVATE

——————– State SUMMARY ————————–
TotSize ( KB) Pct(Tots) Usage
3ffd2000 ( 1048392) : 49.99% : MEM_COMMIT
2cf36000 ( 736472) : 35.12% : MEM_FREE
130e8000 ( 312224) : 14.89% : MEM_RESERVE

Largest free region: Base 685ba000 – Size 04b36000 (77016 KB)


 


So from this, we can see if the problem is in NT Heaps, Virtual Memory, Images (DLL’s) or if none of them are that high and the issue is memory fragmentation.  Another important thing to look at is at the bottom.  If the largest free region is low, that is always a good sign that we are out of memory.  In this case, the dump is mostly RegionUsageIsVAD which is Virtual Memory, where .NET allocates things.  Had this been in RegionUsageHeap mostly, it would be a native NT Heap leak and we would troubleshoot that differently, we will address that in a future post with using DebugDiag.


Now that we know the problem is with Virtual Memory, we will see if the issue is with .NET itself (as other programs can also allocate memory in the Virtual Address Space).  There are a few ways to do this, such as running !clr10\sos.eeheap -gc and look at the amount of space that the heaps are taking up.  You can also just run the command which prints out all the objects in the .NET heap as it totals them all up at the end.


 

0:000> !clr10\sos.dumpheap -stat
Loaded Son of Strike data table version 5 from “C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\mscorsvr.dll”
Loading the heap objects into our cache.
——————————
Heap 0
total 3,234,104 objects
——————————
Heap 1
total 1,753,696 objects
——————————
Heap 2
total 2,309,441 objects
——————————
Heap 3
total 2,128,943 objects
——————————
Heap 4
total 2,554,432 objects
——————————
Heap 5
total 3,801,613 objects
——————————
Heap 6
total 3,083,516 objects
——————————
Heap 7
total 2,645,112 objects
——————————
total 21,510,857 objects
Statistics:
MT Count TotalSize Class Name
0x79be7078 1 12 System.Runtime.Remoting.Proxies.ProxyAttribute
0x79bc08e0 1 12 System.Empty
0x79b9f448 1 12 System.Byte
0x03179c0c 1 12 System.InvariantComparer

0x79bab93c 1,934 100,568 System.Collections.Hashtable
0x025ea844 2,400 115,200 System.Web.Caching.ExpiresBucket
0x027958c4 4,992 119,808 System.Xml.NameTable/Entry
0x030653d4 2,491 139,496 System.Data.SimpleType
0x79ba2ee4 6,868 164,832 System.Collections.ArrayList
0x032a5b5c 14,795 236,720 System.Data.DataRowChangeEventArgs
0x025e1294 123 253,380 System.Decimal[]
0x0306309c 13,033 312,792 System.Data.ParentForeignKeyConstraintEnumerator
0x0317f934 5,348 406,448 System.Data.SqlClient._SqlMetaData
0x03060024 4,113 526,464 System.Data.DataColumn
0x01ea2970 2,275 610,128 System.Collections.Hashtable/bucket[]
0x79baaf78 75,231 902,772 System.Int32
0x03064b44 22,838 1,461,632 System.Data.DataColumnPropertyDescriptor
0x01ea375c 244 1,603,180 System.Boolean[]
0x01ea236c 1,464 1,950,192 System.Char[]
0x79ba3adc 95,948 2,302,752 System.Collections.ArrayList/ArrayListEnumeratorSimple
0x032a57b4 256,739 5,134,780 System.Data.DataColumnChangeEventArgs
0x01ea2c3c 1,425 8,196,224 System.Byte[]
0x025e1da4 153 12,738,360 System.DateTime[]
0x01ea26b0 2,090 20,273,700 System.Int32[]
0x030bb59c 1,319,665 58,065,260 System.Data.DataRow
0x000e1de8 1,004,003 81,543,256 Free
0x01ea209c 21,974 127,692,108 System.Object[]
0x79b94638 18,579,050 554,497,748 System.String
Total 21,510,857 objects, Total size: 881,488,504

Fragmented blocks larger than 0.5MB:
Addr Size Followed by
0x2ff9f2c0 1.0MB large free object followed by 0x3009cfd8 System.String
0x303ec4cc 0.6MB large free object followed by 0x30484af0 System.Object[]
0x0b608a8c 0.6MB large free object followed by 0x0b6a3d4c Travelers.CL.LPE.BoilerWorkstation.Datasets.PolicyInfo/PolicyLocRow
0x3ad2b8a4 0.7MB large free object followed by 0x3ade5ad8 Travelers.CL.LPE.BoilerWorkstation.Datasets.PolicyInfo/PolicyLocRow
0x3d6598f4 0.8MB large free object followed by 0x3d732f40 System.String
0x3fe78f04 0.6MB large free object followed by 0x3ff1e72c System.String
0x401d3a94 0.7MB large free object followed by 0x40283c94 System.Web.HttpInputStream


 


From this output we can see that the .NET heap is taking up 881,488,504 bytes or 840.7MB.  We will look at some other possible causes later, but here we see that strings are taking up 554,497,748 bytes.  So that is where the bulk of our problems are.  Now strings can be caused by a number of things.  Like not using the StringBuilder class (KB306822).  But in this case, we can notice the high number of DataRow objects and DataColumn objects and run another command to look at the System.Data.DataTable objects


 

0:000> !clr10\sos.dumpdatatables
DataTable Rows Columns DataSet nextRowID ColumnCount
———————————————————————————————–
0x55807f2c 0x558081e4 0x55808408 0x55807e24 1 11

0x407454e0 0x40745774 0x407459a8 0x40745358 218 2
0x4074347c 0x40743710 0x40743944 0x407432f4 218 2
0x408969a8 0x40896c24 0x40896e58 0x40895ee8 854 12
0x168731bc 0x168734a4 0x16873678 0x1686ffb0 402,521 23
0x123596fc 0x123599e4 0x12359bb8 0x123564f0 446,028 23
0x1439ba30 0x1439bd18 0x1439beec 0x14398824 471,117 23
Total 189 DataTable objects

 


Most of the time, all cells of a DataTable aren’t strings, but each of those bottom three tables have over 10 million cells.  So a lot of the Strings are coming from these tables.  Now we can’t get a lot of info on these tables, but we can get the names of the columns.  Below are the commands, along with a few new commands that make the process much shorter. Note: !do is short for !dumpobj


 

0:000> !clr10\sos.do 0x1439beec
Name: System.Data.DataColumnCollection
MethodTable 0x03062254
EEClass 0x02d8d868
Size 52(0x34) bytes
GC Generation: 2
mdToken: 0x0200001a (c:\windows\assembly\gac\system.data\1.0.5000.0__b77a5c561934e089\system.data.dll)
FieldDesc*: 0x03061e94
MT Field Offset Type Attr Value Name
0x0302b7f8 0x4000331 0 CLASS shared static RefreshEventArgs
>> Domain:Value 0x000dcbb0:NotInit 0x00104fa0:0x1e2fb450 0x026c12a0:0x1e34a790 0x02f6b6a0:0x1a314a24 0x02fed358:0x12309044 0x03f78600:0x103cbad0 <<
0x03062254 0x4000376 0x4 CLASS instance 0x1439ba30 table
0x03062254 0x4000377 0x8 CLASS instance 0x1439bf20 list
0x03062254 0x4000378 0x28 System.Int32 instance 1 defaultNameIndex
0x03062254 0x4000379 0xc CLASS instance 0x00000000 delayedAddRangeColumns
0x03062254 0x400037a 0x10 CLASS instance 0x1439d490 columnQueue
0x03062254 0x400037b 0x14 CLASS instance 0x1439bf38 columnFromName
0x03062254 0x400037c 0x18 CLASS instance 0x1439bf6c hashCodeProvider
0x03062254 0x400037d 0x1c CLASS instance 0x00000000 onCollectionChangedDelegate
0x03062254 0x400037e 0x20 CLASS instance 0x00000000 onCollectionChangingDelegate
0x03062254 0x400037f 0x24 CLASS instance 0x00000000 onColumnPropertyChangedDelegate
0x03062254 0x4000380 0x2c System.Boolean instance 0 fInClear
0:000> !clr10\sos.do 0x1439bf20
Name: System.Collections.ArrayList
MethodTable 0x79ba2ee4
EEClass 0x79ba3020
Size 24(0x18) bytes
GC Generation: 2
mdToken: 0x02000100 (c:\windows\microsoft.net\framework\v1.1.4322\mscorlib.dll)
FieldDesc*: 0x79ba3084
MT Field Offset Type Attr Value Name
0x79ba2ee4 0x4000362 0x4 CLASS instance 0x1439cab4 _items
0x79ba2ee4 0x4000363 0xc System.Int32 instance 23 _size
0x79ba2ee4 0x4000364 0x10 System.Int32 instance 24 _version
0x79ba2ee4 0x4000365 0x8 CLASS instance 0x00000000 _syncRoot
0:000> .foreach ( myobj { !clr10\sos.do -v 0x1439cab4 -short } ) {
!clr10\sos.dumpfield -field _columnName myobj }
String: Loc1

String: Loc2

String: SLoc

String: Num

String: Address

String: Address2

String: City

String: County

String: State

String: Zip

String: Country

String: ContactNm

String: ContactTitle

String: ContactPhone

String: ContactEmail

String: DataSource

String: Status

String: Occupancy

String: TotalAmt

String: NewInd

String: UniqueId

String: EspLocation

String: PrevInsp


 


We will continue to look at high memory issues in future posts.  So stay tuned and feel free to ask questions or give any feedback.

Comments (6)

  1. So let’s continue our digging into Memory problems and how to track down what is happening.&#160; So

  2. So the last high memory post, High Memory part 3 , focused on native memory that was the problem, let’s

  3. So here is the situation.&#160; I will lead you down the debugging path and get us to a certain place

  4. KB says:

    I am doing sos from 1.1 but the system is saying

    0:000> !dumpfield

    No export dumpfield found

    0:000> !dumpdatatables

    No export dumpdatatables found

    Am I doing something wrong?

  5. tomchris says:

    KB,

    Make sure you are using the SOS out of the clr10 directory under the debugger package.  It is the version that has that command.