I was debugging a weird hang at device boot and I used the command “meminfo kernel” in CeDebugX to get more info, but I realized right away that I didn’t know what this command was showing me. So, I did a bit of investigation into what the output meant and thought I’d write down what I learned so I’d remember it later. And, of course, there’s no better place to write things down than on the HopperRx blog, so here’s what I learned.
Intro to Kernel Heap
To understand the output of "meminfo kernel", I need to explain a bit about the kernel's internal heap. As with all heaps, the kernel's heap allocates a page of memory and then divides that memory into blocks of various sizes as needed. The kernel uses the following 8 different block sizes (in bytes): 16, 36, 64, 168, 228, 524, 576, 1024. Each block size directly corresponds to the actual size of at least one kernel data structure. To demonstrate this, I used a handy trick in PB's Watch Window to show the structure of the kernel's EVENT data type (see image below). If you add up all the bytes in the structure (pProxHash has an array size of 32) you'll see that sizeof(EVENT) is 168 bytes, which fits perfectly into the bucket that is 168 bytes. It turns out that all of the kernel's data types exactly matches one of the 8 buckets.
Unlike the heap used by LocalAlloc(), the kernel's private heap doesn't do compaction. This means that when a chunk of memory is allocated as a certain block, it stays that size until the device reboots. The block will be marked as free when the object it contains is deleted and could be re-used by another object later on, but the blocks will never be resized. If the kernel needs to create a new object but no empty block exists, it will simply create a new block. Consequently, the kernel heap will grow as necessary, but never shrink. This has some important implications I'll touch on later.
Dissecting "Meminfo Kernel"
Okay, that's the basic stuff on the heap, now let's look at what meminfo shows. Here's the output I got from meminfo on DeviceEmulator after a clean boot:
Size Used Max Extra Entries(Max) Name
0: 576 98496 99648 1152 171( 173) Thrd
1: 228 54036 54948 912 237( 241) Mod
2: 36 91944 92412 468 2554( 2567) API/CStk/ClnEvt/StbEvt/Prxy/HData/KMod
3: 168 598920 604800 5880 3565( 3600) Crit/Evt/Sem/Mut/ThrdDbg
4: 64 33920 34048 128 530( 532) FullRef/FSMap/ThrdTm
5: 16 32864 33120 256 2054( 2070) MemBlock
6: 524 0 1572 1572 0( 3) Name
7: 1024 174080 175104 1024 170( 171) HlprStk
Total Used = 1084260 bytes
Total Extra = 11392 bytes
Here's my quick break down of what each column is showing:
First column: The left most column is just a number for the table row. The block sizes aren't listed in any special order so the numbering doesn't mean anything special, but the line numbers make the table easier to read and discuss.
Size: The size, in bytes, of the block described on that particular row.
Used: Number of bytes currently in use at this particular size (i.e. "Used" = "Size" * "Entries")
Max: Number of bytes allocated at this size (i.e. "Max" = "Size" * "(Max)")
Extra: Number of bytes that aren't being used (i.e. ""Extra" = "Size" * "((Max) - Entries)).
Entries: Number of blocks of this size that are currently in use.
(Max): Number of blocks of this size that have been created.
Name: Friendly names of the data types that fit into this block
At the bottom of the "Total Used" shows the sum of all the "Used" columns.
The "Total Extra" shows the sum of all the of all the "Extra" columns.
And, the "heapptr" is just what is sounds like, the pointer to the start of kernel's heap.
What the output actually means
In my table above, row 0 shows the blocks of size 576 bytes. There are currently 173 of these blocks created, and 171 of those blocks are in use. That means, 99,648 bytes are allocated as this block size, and 98,496 of those bytes are in use. This particular block is only used to hold THREAD objects, so this information also tells us there are currently 171 threads in the system.
Contrary to the implications of their names, the Max and (Max) columns don't reflect any kind of upper limit. These columns are actually reporting the number of blocks of that size that have been created. Going back to row 0 as an example again, the Max column shows that 173 blocks have been created at size 576. However, when I took this snapshot, only 171 threads were still around, so only 171 of those blocks were used. If a new application was started and created 3 new threads, the kernel would re-use the 2 empty blocks for 2 of the new threads, and then create a new block for the other new thread. After this happened, "Entries" and "(Max)" would both be match at 174, indicating there are 174 blocks for that size and all 174 of those blocks are in use.
How To use the output
This table is a great way to get an idea of where to start a new investigation. It doesn't provide much help with root causing a specific issue though, so don't try and read too much into the information.
For example, if the table shows rows 2 and 3 have a huge number of entries, then some process is probably leaking handles to a critical section, event, semaphore, or mutex. When one of these objects is created, the object stays around until all handles are closed. If the handle is leaked, then the handle (block size 36) and the object (block size 168) are both going to stay around and never get deleted. On the other hand, if row 2 has a huge number of entries but all the other blocks have a normal number of entries, then it's possible someone is leaking a thread or module handle. Since threads exit on their own, and there's only one instance of a DLL, neither of these buckets will grow even though there are a bunch of handles pointing at them. If all of the columns look fine, then the issue probably isn't a handle leak or object.
As you can see, this info can provide a helpful guide on where to start debugging, which is very valuable sometimes, but it doesn't tell you anything specific about the root cause.