Understanding Output From “meminfo kernel”

I was debugging a weird hang at device boot and I used the command “meminfo kernel” in CeDebugX to get more info, but I realized right away that I didn’t know what this command was showing me.  So, I did a bit of investigation into what the output meant and thought I’d write down what I learned so I’d remember it later.  And, of course, there’s no better place to write things down than on the HopperRx blog, so here’s what I learned.

Intro to Kernel Heap

To understand the output of "meminfo kernel", I need to explain a bit about the kernel's internal heap.  As with all heaps, the kernel's heap allocates a page of memory and then divides that memory into blocks of various sizes as needed.  The kernel uses the following 8 different block sizes (in bytes): 16, 36, 64, 168, 228, 524, 576, 1024.  Each block size directly corresponds to the actual size of at least one kernel data structure.  To demonstrate this, I used a handy trick in PB's Watch Window to show the structure of the kernel's EVENT data type (see image below).  If you add up all the bytes in the structure (pProxHash has an array size of 32) you'll see that sizeof(EVENT) is 168 bytes, which fits perfectly into the bucket that is 168 bytes.  It turns out that all of the kernel's data types exactly matches one of the 8 buckets.LPEVENT structure, from the Watch Window

Unlike the heap used by LocalAlloc(), the kernel's private heap doesn't do compaction.  This means that when a chunk of memory is allocated as a certain block, it stays that size until the device reboots.  The block will be marked as free when the object it contains is deleted and could be re-used by another object later on, but the blocks will never be resized.  If the kernel needs to create a new object but no empty block exists, it will simply create a new block.  Consequently, the kernel heap will grow as necessary, but never shrink.  This has some important implications I'll touch on later.

Dissecting "Meminfo Kernel"

Okay, that's the basic stuff on the heap, now let's look at what meminfo shows.  Here's the output I got from meminfo on DeviceEmulator after a clean boot:

      Size     Used      Max   Extra   Entries(Max)   Name
 0:    576    98496    99648    1152    171(   173)   Thrd
 1:    228    54036    54948     912    237(   241)   Mod
 2:     36    91944    92412     468   2554(  2567)   API/CStk/ClnEvt/StbEvt/Prxy/HData/KMod
 3:    168   598920   604800    5880   3565(  3600)   Crit/Evt/Sem/Mut/ThrdDbg
 4:     64    33920    34048     128    530(   532)   FullRef/FSMap/ThrdTm
 5:     16    32864    33120     256   2054(  2070)   MemBlock
 6:    524        0     1572    1572      0(     3)   Name
 7:   1024   174080   175104    1024    170(   171)   HlprStk
Total Used  =   1084260 bytes
Total Extra =     11392 bytes
heapptr: 0x80076278
Here's my quick break down of what each column is showing:
First column:  The left most column is just a number for the table row.  The block sizes aren't listed in any special order so the numbering doesn't mean anything special, but the line numbers make the table easier to read and discuss.
Size:  The size, in bytes, of the block described on that particular row.
Used:  Number of bytes currently in use at this particular size (i.e. "Used" = "Size" * "Entries")
Max:  Number of bytes allocated at this size (i.e. "Max" = "Size" * "(Max)")
Extra:  Number of bytes that aren't being used (i.e. ""Extra" = "Size" * "((Max) - Entries)).
Entries:  Number of blocks of this size that are currently in use.
(Max):  Number of blocks of this size that have been created.
Name:  Friendly names of the data types that fit into this block
At the bottom of the "Total Used" shows the sum of all the "Used" columns.
The "Total Extra" shows the sum of all the of all the "Extra" columns.  
And, the "heapptr" is just what is sounds like, the pointer to the start of kernel's heap.

What the output actually means

In my table above, row 0 shows the blocks of size  576 bytes.  There are currently 173 of these blocks created, and 171 of those blocks are in use.  That means, 99,648 bytes are allocated as this block size, and  98,496 of those bytes are in use.   This particular block is only used to hold THREAD objects, so this information also tells us there are currently 171 threads in the system.

Contrary to the implications of their names, the Max and (Max) columns don't reflect any kind of upper limit.  These columns are actually reporting the number of blocks of that size that have been created.  Going back to row 0 as an example again, the Max column shows that 173 blocks have been created at size 576.  However, when I took this snapshot, only 171 threads were still around, so only 171 of those blocks were used.  If a new application was started and created 3 new threads, the kernel would re-use the 2 empty blocks for 2 of the new threads, and then create a new block for the other new thread.  After this happened, "Entries" and "(Max)" would both be match at 174, indicating there are 174 blocks for that size and all 174 of those blocks are in use.

How To use the output

This table is a great way to get an idea of where to start a new investigation.  It doesn't provide much help with root causing a specific issue though, so don't try and read too much into the information. 

For example, if the table shows rows 2 and 3 have a huge number of entries, then some process is probably leaking handles to a critical section, event, semaphore, or mutex.  When one of these objects is created, the object stays around until all handles are closed.  If the handle is leaked, then the handle (block size 36) and the object (block size 168) are both going to stay around and never get deleted.  On the other hand, if row 2 has a huge number of entries but all the other blocks have a normal number of entries, then it's possible someone is leaking a thread or module handle.  Since threads exit on their own, and there's only one instance of a DLL, neither of these buckets will grow even though there are a bunch of handles pointing at them.  If all of the columns look fine, then the issue probably isn't a handle leak or object. 

As you can see, this info can provide a helpful guide on where to start debugging, which is very valuable sometimes, but it doesn't tell you anything specific about the root cause.


Comments (2)

  1. jayongg says:

    Thanks for the description!

    Can you explain what all the different Names mean? e.g. what’s a FullRef, CStk, etc?

  2. JeCahill says:

    A few of the Names are obvious, but I don’t know what the rest mean.  I’ll investigate this when I get a chance and post a complete list of the Names once I have it.

    Thrd = Thread object

    Mod = Module object

    Crit = Critical section object

    Evt = Event object

    Sem = Semaphore object

    Mut = Mutex object

    HData = Handle object

Skip to main content