Basics on Windows CEMobile: an alternative way to approach the Memory Monster

Article
11/16/2009

Introduction on Windows CE\Mobile
Virtual Memory on Windows CE 5.0\Mobile (a visual approach)
Troubleshooting Memory Leaks for NETCF applications

Lately I’ve been involved on starting up my team blog: if you read Italian and want to listen about stories from Support Engineers working on various technologies, you can’t avoid signing up to the feed:

Now guess what’s been the topic of the posts I’ve been writing there… Open-mouthed yes! Memory Management on Windows CE\Mobile and NETCF!! (so far the intro and then part 0, 1, 2 – don’t ask me why I made it 0-based, even after an intro… I really don’t remember) And since I’ve been adopting an approach that is not usually the one used to explain how things work, I honestly think I can reuse the same one and translate it here… probably I’m going to repeat some concepts I’ve already blogged about, however it may be worth in order to have one single document, this post, as hopefully a possible quite-ultimate way to describe how memory is handled… is the bar too high??

INTRODUCTION ABOUT WINDOWS CE\MOBILE

So, let’s start: Windows CE and Windows Mobile are not the same thing. After working on this for a while it can be obviuos, but if you’re at your first experience with these so-called “Smart Devices” then it may not be so. We must be clear about the terminology, specifically about terms like “platform”, “operating system”, “Platform Builder”, “Adaptation Kit”, “OEM”, “ODM”, etc.. In reality the true name of “Windows CE” would nowadays be “Windows Embedded CE”, however I don’t want to mess with a product that was once known as “Windows XP Embedded” and which nowadays is differentiated in the following products:

“Windows Embedded Stardard”
“Windows Embedded Enterprise”
“Windows Embedded POSReady”
“Windows Embedded NavReady”
“Windows Embedded Server”

So, when I’ll write “Windows CE” here I’ll mean the historical name of “Windows Embedded CE”: let’s forget the other “Windows Embedded X” products (out of my scope) and let’s concentrate on Windows CE\Mobile.

Windows CE is a platform for OEMs (Original Equipment Manufacturer) . This means that we provide to the manufacturer an Integrated Development Environment very similar to Visual Studio (indeed, Windows CE 6.0 is integrated with Visual Studio), but with the aim of developing Operating Systems (instead of applications), based on the platform provided by Microsoft. The tool is called "Platform Builder for Windows CE”, and up to version 5.0 was a separate tool from Visual Studio.

Windows CE is a modular platform. This means that the OEM is totally free to include only the modules, drivers and applications of his interest. Microsoft provides about 90% of the source code of the Windows CE platform, as well as code examples for drivers and various recommendations (which the OEM might or might not follow). For example, if the device is not equipped with an audio output, then the OEM won’t add a sound driver. If it doesn’t have a display, then the OEM will not develop nor insert a video driver. And so on for the network connectivity, or a barcode-scanner, a camera, and so on. On a Windows CE-based device, the OEM can include whatever he wants. That's why, from the point of view of technical support to application-developers, sometimes we can’t help a programmer who is targeting a specific device whose operating system is based on Windows CE: furthermore, the OEM can decide whether to offer application-developers the opportunity to programmatically interact with special functions of the device through a so-called "Private SDK” (which may also contain a emulator image, for example).

An important detail: differently from Windows Embedded OSs (Standard \ Enterprise \ POSReady \ NavReady \ Server), for Operating Systems based on Windows CE the OEMs actually *COMPILE* the source code of the platform (apart from about 10% provided by Microsoft and corresponding to the core kernel and other features).

Now: Windows Mobile is a particular customization of Windows CE, but in this case the OEM needs to create an Operating System that meets a set of requirements, called "Windows Mobile Logo Test Kit". The tool used by Windows Mobile-OEM is called "Adaptation Kit for Windows Mobile", a special edition of "Platform Builder" and allows to adapt the "Windows Mobile”-Platform to the exact hardware that the OEM has built or that he requested to an ODM (“Original Device Manufacturer”). In the Windows Mobile’s scenario we can’t forget Mobile Operators also, which often "brand" a device requiring the OEM to include specific applications and usually configure the connectivity of the mobile network (GPRS, UMTS, WAP, etc.).. WARNING: nothing prohibits a WinMo-OEM to include special features such as a barcode-scanner or an RFID chip or anything else... the important thing is that the minimal set is the same. Moreover, also WinMo-OEM can provide a “Private SDK” to programmatically expose specific functionality related to their operating system (see for example Samsung SDK containing private APIs for the accelerometer and other features, which are documented and supported by Samsung itself).

Finally, one last thing before starting talking about memory: Windows Mobile 5.0, 6, 6.1 and 6.5 are all platforms based on Windows CE 5.0. So, they all share the same Virtual Memory management mechanisms, except in some details for the latter (mainly with some benefits for application-developers).

VIRTUAL MEMORY ON WINDOWS CE\MOBILE

So now we can start talking about how memory is managed on Operating Systems based on Windows CE 5.0. And I’m being specific on Windows Embedded CE *5.0* because on 6.0 memory management is (finally!) totally changed and there are no more the limitations we’re going to discuss in the remainder. Incidentally, this is part of the same limitations described for Windows CE 4.2 by Doug Boling’s Windows CE .NET Advanced Memory Management, although the article is dated 2002! Fortunately, some improvements have been introduced from Windows Mobile 6 and especially in 6.1 (and therefore also in 6.5), whereby not only the applications have more virtual memory available, but also the entire operating system is more stable as a whole.

I don’t want to repeat here what can be found in documentation and on various blogs: in contrast, I’d like to actually show the theory, because only by looking at data like the following you can realize what a good programmer a Developer for Windows Mobile has to be! Hot The following is the output of a tool developed by Symbol (later acquired by Motorola), which allowed the manufacturer to understand how the device.exe process was responsible (or not) for various problems related to memory. Why? Because

device.exe is the process that loads the DLL for the driver in its own address space;
if it hasn’t yet been loaded by other processes, a DLL will occupy a portion of the virtual memory below the "DLL Load Point";
each DLL may have dependencies, also to be loaded;
each DLL can create its own heap to store data.

The result was something like the following (I obfuscated possibly sensitive data of the software-house I worked with during the Service Request):

So, above all, what did Symbol\Motorola mean by “Code” (blue) and “Data” (green)?

• "Code": these are the *RAM* DLLs loaded or mapped into a process-slot. They start from the top of the 32MB slot and goes down. If several processes use the same DLL, the second one maps it to the same address where the first one had loaded it.
• "Data" is the executable’s compiled code + Heap(s) + Stack. It starts from the bottom and grows.

Finally, the red vertical line represents the "DLL Load Point", i.e. the address where a DLL is loaded in case it hadn’t yet by any other process.

That is the situation of only the process slots, not the whole virtual memory - in particular the contents of the Large Memory Area is not shown:

Why did I specify *RAM* DLLs? Because those placed by the OEM in the ROM (= firmware) are executed directly there, without the need of the process loading their "compiled" code into its Address Space (they’re XIP DLL, i.e. "Executed in Place" – in Slot 1).

That picture also shows that the green part (code + heap + stack) may exceed the DLL Load Point. Indeed, the problems related to lack of available virtual memory is usually of 2 types:

a process tries to load a DLL that hadn’t been loaded by any other before, trying to move the DLL Load Point in an area already occupied by code + heap + stack
a process tries to allocate memory (code + heap + stack) but the process slot has got saturated even on sections left free among loaded DLLs

That's also because in general one of the advices to avoid memory problems had always been to load all DLLs used by the application itself at application startup, through an explicit call to LoadLibrary() . Another visual example is the following:

We’ll later discuss in detail the particularities of NETCF, but it's worth at this point noting a detail: apart from the actual CLR DLLs (mscoree*.dll, netcfagl*.dll), every other assembly doesn’t waste address space in the Process slot, but is loaded into the Large Memory Area. Even more, if you are using the version of the runtime included in the ROM by the OEM, also the runtime DLLs do not affect the process’ virtual memory space. Obviously it is different when the application P/Invoke native DLLs: these will be loaded in the process slot.

Moreover, if you look at the picture showing all the alive processes, you’ll notice that in the upper bound of all the slots there’s a portion of "blue" virtual memory, which is the same for all the processes. This is the memory blocked by the Operating System whose size is equal to the sum of the binaries (the simple .EXE files) active at any given moment. So large monolithic EXEs (large for example because containing “many” resources) are not recommended at all on Windows CE 5.0! And in general supporting NETCF developers I can say I’ve seen many “big” applications... that is not a good practice for this reason!

Through those pictures it is also easy to understand why the whole system stability is a function of all active processes, and in particular it is easy to see that very often DEVICE.EXE can be a source of headaches! Think of those Windows Mobile-based devices that have the radio stack (i.e. the phone), Bluetooth, WiFi, Camera, barcode-scanner, etc. .. each of these drivers is a DLL that device.exe has to load (blue line), and each can also create its own stack and heap (green line). Some OEMs allowed developers to programmatically disable some drivers (to reduce pressure done by device.exe), but obviously we can not take for granted for example that a user manually restarts that feature (or this is done by another application...).

So, what has been done to fight device.exe’s power? In many cases, the driver-DLLs were loaded by services.exe, which is the host process for Service-DLLs on Windows CE. But very often it was not enough... What Windows Mobile 6.1 introduced is that native DLLs with size > 64KB are typically loaded into the so-called slots 60 and 61, which are part of the Large Memory Area. Another improvement in Windows Mobile 6.1 was to dedicate another slot (slot 59) to the driver stack (part of the Green Line to device.exe). Of course, this means that the memory-mapped files have now less space available (and I have recently handled a request for exactly this purpose, coming by a software company that was developing a GPS navigation software that could not load some map files in WinMo6.1), but in general the whole operating system has gained a stability that hadn’t before...

To conclude, the tool I mentioned was developed by Symbol and I don’t think it’s publicly available. But a similar tool has recently been published on CodePlex (source code included!) through the article Visualizing the Windows Mobile Virtual Memory Monster. The term "Virtual Memory Monster" was invented years ago by Reed Robison… (part 1 e part 2). I've already been using it in a couple of requests and highly recommend it!

TROUBLESHOOTING MEMORY LEAKS FOR NETCF APPLICATIONS

Instead of explaining how things work in theory, which is a task I leave to more authoritative sources like the documentation itself and various blogs – one for all that of Abhinaba Basu, who is precisely the GC Guru inside the NETCF Dev Team (Back to basic: Series on dynamic memory management), I’d like to follow the troubleshooting flow I run through when a new Service Request arrives, about for example the following issues:

OutOfMemoryException
SqlCeException: “Not enough memory to complete this operation”
The application can’t load a DLL
Other misbehaviors that may lead thinking about a problem with the memory

Firstly, we must determine whether the problem is specific to an OEM. The best approach, when possible, is to verify if the error occurs even on emulators contained in the various Windows Mobile SDK. If not, the help that Microsoft Technical Support can provide is limited, as it is possible that the error is due to a customization of the Windows Mobile platform by the OEM. In this case, it may be helpful to know about what I wrote about device.exe above.

Another initial step, in the case of applications NETCF v2 SP2, is to check if just running the application on NETCF v3.5 gives any improvement. There is no need to recompile the application with the Visual Studio 2008 - just like for .NET Desktop applications, add a configuration XML file in the same folder that contains the file TheApplication.exe, named TheApplication.exe.config and whose content is simply (I mentioned here):

 <configuration>
  <startup>
    <supportedRuntime version="v3.5.*"/>
  </startup>
</configuration>

So, after having considered possible “trivial” causes, you can proceed to the analysis... Historically NETCF developers haven’t had an easy time in troubleshooting due to lack of appropriate tools – unlike Desktop cousins! – but over the years Microsoft has released tools that have gradually evolved over the current Power Toys for .NET Compact Framework 3.5. Apart from these you must know the(freeware!) EQATEC’s ones (Tracer and Profiler) and recently a tool on CodeProject that I mentioned earlier, that displays the status of virtual memory (VirtualMemory, with source code).

Regarding power-toys, when you are dealing with a problem around memory, you have 2 of them that are of great help: the "CLR Profiler" and "Remote Performance Monitor (RPM) ”. The first one is useful in visually making problems with objects’ allocation almost immediate and allows you to notice the problem in a visual way. Info on how using it are available through The CLR Profiler for the .Net Compact Framework Series Index. The second one provides, both in real time and through an analysis a posteriori, counters about the usage of MANAGED memory; also, through the "GC Heap Viewer” it allows not only to study the exact content of the managed heap, but also allows you to compare contents of the heap in different moments, in order to bring out a possible unexpected growth of a certain type of objects. Some images are available on Finding Managed Memory leaks using the .Net CF Remote Performance Monitor, which is useful also to get an idea about which counters are available, while a list and related explanations are provided on Monitoring Application Performance on the .NET Compact Framework - Table of Contents and Index . What I'd like to do here is not to repeat the same explanations, already detailed in the links above, but share some practical experience...

For example, in the vast majority of cases I have handled about memory leaks, the problem was due to Forms (or Controls) that unexpectedly were NOT removed by the Garbage Collector. The instances of the Form class of the application are therefore the first thing to check through the Remote Performance Monitor and GC Heap Viewer. For this reason, where appropriate (e.g. if the total form are "not so many"), to avoid memory problems with NETCF applications it may be useful to adopt the so-called "Singleton Pattern": this way a single managed instance of a given form will exist throughout the application life cycle.

So, supposing to be in the following situation: I used the Remote Performance Monitor and saved different .GCLOG files during normal use of the application, and thanks to the GC Heap Viewer I noticed that an unexpected number of forms stays in memory, and also that this increases during the life of the application, although there have been a certain number of Garbage Collections. Why the memory of a Form is not cleaned up by the garbage collector? Thanks to the GC Heap Viewer you can know exactly who maintains a reference to what, in the "Root View" on the right pane. Obviously knowing application’s architecture will help in identifying unexpected roots.

A special consideration must be done for MODAL Forms in .NET (the dialogs, those that on Windows Mobile have the close button "Ok" instead of "X", and which permits a developer to prevent the user to return to the previous form). In many cases I have handled, the problem was simply due to the fact that the code was not invoking Close() (or. Dispose ()) after .ShowDialog():

 Form2 f2 = new Form2();
f2.ShowDialog();
f2.Close();

Why should it matter? Because often (not always, for example, not when you expect a DialogResult) on Windows Mobile the user clicks on 'Ok' in the top right "close" the dialog. Also on Desktop, when a dialog is "closed" in this way the window is not closed, but "hidden"! And it could happen that the code creates a new instance of the form, without removing the old one in memory. It’s documented in “Form..::.ShowDialog Method” (the doc talks about “X” but of course for Windows Mobile refers to the 'Ok' referred to above):

[…] When a form is displayed as a modal dialog box, clicking the Close button (the button with an X at the upper-right corner of the form) causes the form to be hidden and the DialogResult property to be set to DialogResult.Cancel. Unlike modeless forms, the Close method is not called by the .NET Framework when the user clicks the close form button of a dialog box or sets the value of the DialogResult property. Instead the form is hidden and can be shown again without creating a new instance of the dialog box. Because a form displayed as a dialog box is not closed, you must call the Dispose method of the form when the form is no longer needed by your application.

Anyway, we assumed so far that the memory leak is MANAGED, but in reality it may be that leak is with the NATIVE resources that are used by a .NET instance, which have not been successfully released by implementing the so-called " IDisposable Pattern . And around this there are some peculiarities in NETCF, that Desktop-developers don’t need to worry about, particularly with respect to SQL Compact objects and "graphical" objects, i.e. classes of the System.Drawing namespace. In NETCF the Font, Image, Bitmap, Pen, Brush objects are simple wrappers around their native resources, which in the Windows CE-based operating systems are handled by the GWES (Graphics, Windowing and Event Subsystem). What does this mean? It means that in their own .Dispose() they effectively release their native resources, and therefore one *must invoke .Dispose() for Drawing objects* (or invoke methods that indirectly call it, for example .Clear() in ImageList.ImageCollection – which has not the .Dispose() itself). Note that among the counters provided by the Remote Performance Monitor, the category "Windows.Forms" contains indeed:

Controls Created
Brushes Created
Pens Created
Bitmaps Created
Regions Created
Fonts Created
Graphics Created (FromImage)
Graphics Created (CreateGraphics)

Note that I’m not talking about only objects directly “born” as Brush, Pen, etc.. I’m talking also about those objects whose properties contain graphic objects, such as a PictureBox or ImageList (or indirectly, an ImageList of a ToolBar). So, when you close a form, remember to:

 this.ImageList1.Images.Clear();
this.ToolBar1.ImageList.Images.Clear();
this.PictureBox1.Image.Dispose();
//etc...

Finally, still about Forms, a simple technique I often used to identify possible problems with items not properly released at the closing of a form has been to emulate user interaction by “automatic” opening and closing of the form. I’m purely talking about a test code like this:

 int N = 1000;
for (int i = 1; i <= N; i++)
{
   if (i % 200 == 0)  MessageBox.Show(i.ToString()); //just to know it's running
   // and to block execution in case you want to save counters (.STAT)
   // and GC Heap View (.GCLOG)    
   frmTest frm = new frmTest(/*...*/);    
   frm.Show();    
   frm.Close();    
   frm.Dispose(); 
}
MessageBox.Show("Over. Done. Finito.");

After running the loop N times, the Remote Performance Monitor will be of considerable help to see what is going wrong... Nerd

A final note before concluding this paragraph. It may be that an application is so complex to require "a lot of" virtual memory. This would not be a problem, as long as there is room for the lines "green" in my previous post. But requiring "a lot of" memory means that the Garbage Collector will get kicked more frequently, thus impacting general application performance (because the GC must first “lock” the threads in a safe state). The point is that if the application is so complex to require too frequent use of garbage collection (and therefore performance may not be acceptable by end-users), then it might be worthwhile to split the application into 2 parts, such as one for memory dog-guard and another for the user interface. This process at the cost of an additional process slot, but often it is something that can be paid. Or, since the managed DLLs are loaded in the Large Memory Area without wasting precious process’ address space, an idea would be to place all classes, even those of the form, not in the EXE but in the DLLs! A simple yet very effective idea, which Rob Tiffany has discussed about in his post MemMaker for the .NET Compact Framework.

Enjoy!

~raffaele

Basics on Windows CEMobile: an alternative way to approach the Memory Monster

Additional resources