High Memory part 3 - Native Heap
So let's continue our digging into Memory problems and how to track down what is happening. So our last post, High Memory continued, went into how to look into the managed heap and if the problem is a System.Data.DataTable, we can look at it and see what the columns are.
So what if the memory isn't all in the Virtual Memory. What if it is in the native NT Heap? In that case, !address -summary may look like:
0:000> !address -summary
-------------------- Usage SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Pct(Busy) Usage
41c2000 ( 67336) : 03.21% 04.95% : RegionUsageIsVAD
2cf36000 ( 736472) : 35.12% 00.00% : RegionUsageFree
3d0b000 ( 62508) : 02.98% 04.59% : RegionUsageImage
d80000 ( 13824) : 00.66% 01.02% : RegionUsageStack
36000 ( 216) : 00.01% 00.02% : RegionUsageTeb
4a434000 ( 1216720) : 58.02% 89.42% : RegionUsageHeap
0 ( 0) : 00.00% 00.00% : RegionUsagePageHeap
1000 ( 4) : 00.00% 00.00% : RegionUsagePeb
1000 ( 4) : 00.00% 00.00% : RegionUsageProcessParametrs
1000 ( 4) : 00.00% 00.00% : RegionUsageEnvironmentBlock
Tot: 7fff0000 (2097088 KB) Busy: 530ba000 (1360616 KB)
-------------------- Type SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
2cf36000 ( 736472) : 35.12% :
3d0b000 ( 62508) : 02.98% : MEM_IMAGE
c10000 ( 12352) : 00.59% : MEM_MAPPED
4e79f000 ( 1285756) : 61.31% : MEM_PRIVATE
-------------------- State SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
3ffd2000 ( 1048392) : 49.99% : MEM_COMMIT
2cf36000 ( 736472) : 35.12% : MEM_FREE
130e8000 ( 312224) : 14.89% : MEM_RESERVE
Largest free region: Base 685ba000 - Size 04b36000 (77016 KB)
From here we can see that most of the memory now is native. The best way to find out what is causing all of these memory to be allocated is to use a tool from the IIS team called DebugDiag. When you run this, you can use the Memory and Handle Leak rule to have DebugDiag track native memory and try to help you find out what is causing the high memory.
After you have run the test under DebugDiag, you load the dump file into DebugDiag and run the analysis report Memory Pressure Analyzers. When you look at the report it creates, you will see some summary information about memory, such as:
mscorwks.dll (a known Windows memory manager) is responsible for 761.87
MBytes worth of outstanding allocations. These allocations appear
to have originated from the following module(s) and function(s):
System.Threading._TimerCallback.TimerCallback_Context(System.Object)
From this we can click on the name of the DLL and jump to the report for the file. The main part to look at are the Top 5 functions by allocation count and Top 5 functions by allocation size.
Top 5 functions by allocation size
mscorwks!EEHeapAlloc+12d 454.89 MBytes
mscorwks!EEvirtualAlloc+104 306.94 MBytes
mscorwks!DebuggerHeap::Alloc+2f 39.16 KBytes
mscorwks!CExecutionEngine::CheckThreadState+14d 288 Bytes
mscorwks!NLSTable::OpenOrCreateMemoryMapping+120 0 Bytes
Each function, if we click on it, will give us a Leak Probability and also is then further broken down to show the Top 10 allocation sizes by allocation count and Top 10 allocation sizes by total size for each of these functions. So we will get something like:
Top 10 allocation sizes by allocation count
24 Bytes 44,766 allocation(s)
28 Bytes 8,973 allocation(s)
40 Bytes 8,404 allocation(s)
20 Bytes 7,388 allocation(s)
16 Bytes 6,761 allocation(s)
64 Bytes 5,584 allocation(s)
5.26 KBytes 5,580 allocation(s)
19.54 KBytes 5,580 allocation(s)
2.35 KBytes 5,580 allocation(s)
36 Bytes 4,895 allocation(s)
So now we can see for the function, we should focus on allocations of a particular size. So if we want to focus on the 24 Byte allocations, we then scroll a little farther down in the report and we will see callstacks. Just scroll until you find one of the size you are looking for. For example:
Call stack sample 3
Address 0x1c4704d0
Allocation Time 00:05:02 since tracking started
Allocation Size 24 Bytes
Function
mscorwks!EEHeapAlloc+12d
mscorwks!EEHeapAllocInProcessHeap+51
System.Threading._TimerCallback.TimerCallback_Context(System.Object) System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
webengine!HashtableIUnknown::AddCallback+a
webengine!HttpCompletion::ProcessRequestInManagedCode+1a3
webengine!HttpCompletion::ProcessRequestInManagedCode+1a3
webengine!HttpCompletion::ProcessCompletion+3e
webengine!CorThreadPoolWorkitemCallback+18
mscorwks!ThreadpoolMgr::intermediateThreadProc+49
kernel32!BaseThreadStart+34
So now we have callstacks that allocated the memory that we need to focus on. There is some more information about DebugDiag on it's blog: https://blogs.msdn.com/debugdiag/