Saving VM by using OEMDRIVERSHIGH package


The release of Windows Mobile 6 (WM6) gave us a few more tools for dealing with Virtual Memory (VM) issues. Compaction of slot 1 is new for WM6, in previous releases all slot 1 modules were aligned on 64k boundaries, causing VM address space to be wasted when modules sizes aren’t perfectly divisible by the alignment size of 64KB were mapped into slot 1. The new alignment size of 4k allows more modules to be packed closer together, thus better compaction.


To illustrate this point, Figure 1 below shows 4 files totaling 142k in size taking up 320k using 64k alignment. Figure 2 shows the same 4 files only taking up 144k using 4k alignment. The white space is unused memory.



Along with compaction, the WM6 release also provides a way to map OEM driver’s modules into slot 1 with the package OEMDRIVERSHIGH. This is important because all the modules that do not fit into slot 1 rollover into slot 0, and thus reducing available VM for all slots. Further exacerbating the situation of modules rolling over into slot 0 is the fact slot 0 aligns modules on 64k boundaries. The OEMDRIVERSHIGH package enables OEMs, SVs, and system integrators to change the packaging priority of OEM drivers, which is typically near the end in the list of packages processed by makeimg.


When it comes to reconfiguring modules, OEM’s don’t have the ability to rearrange packages or to change the modules that go into a particular package. All that is available at this stage of building an image is reconfiguring the modules that go into the OEMDRIVERS package. Now with the addition of OEMDRIVERSHIGH package an OEM can target selected DLL’s to be processed, which will push larger modules out of slot 1 and into slot 0.


 


Optimizing OEMDRIVERSHIGH usage


To save VM using OEMDRIVERSHIGH we are going to identify the modules at the bottom of slot 1 on the verge of rolling over into slot 0. Through careful comparison of these modules and those currently in the OEMDRIVERS package we will reassign OEMDRIVERS modules to OEMDRIVERHIGH. By doing so the modules in OEMDRIVERSHIGH will be processed before the modules at the bottom of slot 1, and through this careful calculation the OEMDRIVERSHIGH modules targeted for slot 1 will rollover files on the slot 1 / 0 boundary. The VM savings will come from rolling over larger modules that are 100’s of KB into slot 0 which is 64KB align and filling the freed slot 1 space with smaller modules. It is the relocation, for example, of three 16KB modules to slot 1 that will free 192KB in slot 0 and only consume 48KB in slot 1.


To achieve this, the first place to start is analyzing the output of makeimg from a baseline build. The output isn’t always captured so you many need to run it once capturing the output (makeimg > makeimg.out 2>&1). Search through makeimg.out and look for “fixing up”. These are modules rolling over from slot 1 into slot 0. Below is an example of modules that just made it into slot 1 and modules that started rolling over into slot 0. This list of modules will be different for every project based on SYSGEN, BSP flags, IMG flags and the type of image.


Using the data from makeimg.out we will build a table to calculate the size of each module sitting at the bottom of slot1. Later we will use this data to determine how many modules to move into OEMDRIVERSHIGH, which will cause these modules to rollover.


 


                                                        FIGURE 3


 


Module cespell.dll          at offset 01d8a000 data, 0210b000 code


Module MsgStore.dll         at offset 01d89000 data, 02106000 code


Module msim.dll             at offset 01d87000 data, 020e4000 code


Module mscoree.dll          at offset 01d86000 data, 020d4000 code


Module netcfagl2_0.dll      at offset 01d85000 data, 02099000 code


Module netcfd3dm2_0.dll     at offset 01d84000 data, 02072000 code


Module ccoredrv.dll         at offset 01d83000 data, 02051000 code


Module ccoreutl.dll         at offset 01d82000 data, 02042000 code


Module cspvoice.dll         at offset 01d81000 data, 02032000 code


Module simsec.dll           at offset 01d80000 data, 02026000 code


Module sms.dll              at offset 01d7f000 data, 02020000 code


Code space full, fixing up supsvcs.dll to ram space


Code space full, fixing up btagsvc.dll to ram space


Code space full, fixing up BthAGPhonebook.dll to ram space


Code space full, fixing up celltsp.dll to ram space


Code space full, fixing up cplphone.dll to ram space


Code space full, fixing up phone.dll to ram space


Code space full, fixing up ril.dll to ram space


 


Taking the modules code offset from Figure 3, and subtracting it from the code offset of the previously listed module will determine the slot 1 space that can be recouped. (The data segment is need regardless of what we do and will remain the same.) So for msim.dll the code size is 0x02106000 – 0x020E4000, which is 136KB. The rollover order will be the same for every run of makeimg, the only exception is when slot 1 is low on space, it will pack in modules small enough to fit into remaining space. This is good news because one less module in slot 0 saves an additional 64KB. The consistent rollover order enables us to predict which files will rollover and in what order, this is illustrated in Table 1. The last two columns of this table illustrate the total space saved in slot 1 and total space used by these modules as they rollover. Notice how sms.dll consumes 24K of space in slot 1 and will consume 64K in slot 0.


 



























































































Table 1


 


Rollover Order


Name


Space used


in slot 1 (KB)*


Space used


in slot 0 (KB)


 if rolled over*


Incremental space freed in slot 1 (KB)


Incremental space used in slot 0 (KB)


1st


sms.dll


24


64


24


64


2nd


simsec.dll


48


64


72


128


3rd


cspvoice.dll


64


64


136


192


4th


ccoreutl.dll


60


64


196


256


5th


ccordrv.dll


132


192


328


448


6th


netcfd3dm2_0.dll


156


192


484


640


7th


netcfag12_0.dll


236


256


720


896


8th


mscoree.dll


64


64


784


960


9th


msim.dll


136


192


920


1152


10th


MsgStore.dll


20


64


940


1216


11th


cespell.dll


160


192


1100


1408


*Calculated with equation roundup(size/slot alignment)*slot alignment; sms.dll size in slot 0 is roundup((0x02026000 – 0x0202000)/1024/64)*64.


 


Now we need to generate a list of OEMDRIVERS modules that can be relocated into the OEMDRIVERSHIGH package. A dir of %_FLATRELEASEDIR%/OEMDrivers_PACKAGE_FILES/ will result in a rough list. Further refine the list by making sure the modules are not being filtered out of the image, so search makeimg.out to verify their presence.


The next step is to determine the module size that will be used in slot 1. For DLLs and EXEs this can be done with the command dumpbin /headers <DLL name> and subtracting the size of .reloc from the total size of the image. For the example below we get 0x5000KB (20KB).


 


D:\_FLATRELEASEDIR>dumpbin /headers gpio.dll

 


             6000 size of image


 


        1000 .reloc


 


 


Table 2 is a list of modules going into OEMDRIVERS package that can be moved into the OEMDRIVERSHIGH package. The table is sorted by a ratio of wasted VM / module size, size is taken from the dumpbin output. The columns “Total space freed in slot 0” and “Total space used in slot 1”list the sum total of space used by that module and the others that precede it in the table based on the compaction of each slot. The column “Wasted VM in slot 0” illustrates the VM not utilized by 64KB alignment. To populate “Modules from Table 1 rolled over into slot 0” we compare the “Incremental space freed in slot 1” column of Table 1 and the “Total space used in slot 1” of Table 2. This comparison determines how many modules we can put into OEMDRIVERSHIGH thus causing modules from Table 1 to rollover into slot 0. The intersections of the two columns are marked with an asterisk to indicate the optimal amount of savings to get an additional module listed in Table 1 to rollover. As illustrated in this table, it would only take one module (gxdma.dll) to get one module from Table 1 to rollover. However if we wanted to rollover two modules we could relocate the first 2-4 modules, four being to optimal number.


 















































































































































































Table 2


 


Name


Module size (KB) *


Total space freed in slot 0 (KB)


Total space used in slot 1 (KB)


Waist in 64k Slot (KB)


Ratio
Waste/Size


Files rolled over into slot 0


gxdma.dll


16


64


16


48


3.00


1*


usbOtg.dll


16


128


32


48


3.00


2


nleddrvr.dll


16


192


48


48


3.00


2


keypad.dll


20


256


68


44


2.20


2*


backlight.dll


20


320


88


44


2.20


3


battdrvr.dll


20


384


108


44


2.20


3


gpio.dll


20


448


128


44


2.20


3*


sdioclient.dll


24


512


152


40


1.67


4


kbdmouse.dll


36


576


188


28


0.78


4*


drvr005.dll


76


704


264


52


0.68


5


drvr001.dll


40


768


304


24


0.60


5*


uart.dll


40


832


344


24


0.60


6


touch.dll


44


896


388


20


0.45


6


drvr002.dll


48


960


436


16


0.33


6


drvr003.dll


48


1,024


484


16


0.33


6*


ddi.dll


536


1,600


1,020


40


0.07


11*


rilgsm.dll


596


2,240


1,616


44


0.07


 


usbfn.dll


60


2,304


1,676


4


0.07


 


drvr006.dll


368


2,688


2,044


16


0.04


 


drvr004.dll


64


2,752


2,108


0


0.00


 


* To achieve optimal savings when rolling over this number of files from slot 0 push this file and all pervious files into OEMDRIVERSHIGH


 


We can now use the optimal number of modules to relocate to estimate VM savings. We can assume VM usage in slot 1 stays the same, though it isn’t exactly the same as the relocated modules aren’t exactly the same size as the ones being rolled over. Finally we compare the differences in slot 0 usage. Table 3 brings together data from Table 1 and optimal data from Table 2 to calculate VM savings. Looking that the data under the column “Compaction savings in slot 0 (KB)” we can see a clear point where causing 6 modules to rollover will save 384KB in VM. We will relocate the first 15 modules of table 2 so that six modules of Table 1 rollover. We should see about 384KB of VM savings. It is fathomable, because VM is conserved in 64KB chucks that there are multiple points of optimal savings, if this were to happen in your optimization you should implement each scenario to see which one results in the better optimization.


 















































































Table 3


 


Space freed in slot 0 by relocation (KB) (From Table 2)


Space used in slot 0 by rollover (KB)


(From Table 1)


Compaction


savings in slot 0 (KB)


Modules rolled over


Modules


relocated to OEMDRIVERSHIGH


64


64


0


1


1


256


128


128


2


4


448


192


256


3


7


576


256


320


4


9


768


448


320


5


11


1024


640


384


6


15


-*


896



7



-*


960



8



-*


1152



9



-*


1216



10



1600


1408


192


11


16


* Based on the sort order of Table 2 there is a large jump between the 15th and 16th module; because of the ddi.dlls (16th) size. Relocated the ddi.dll will cause the 7th, 8th, 9th, 10th, and 11th modules of table 1 to rollover.


 


Enable OEMDRIVERSHIGH


·         Set environment variable IMGOEMDRIVERSHIGH=1


·         In platform\<platform>\files\wpc\oem.cpm.csv or platform\<platform>\files\smartfon\oem.cpm.csv add the line ‘PACKAGE_OEMDRIVERSHIGH,OEMDriversHigh’


·         In platform\<platform>\files\platform.bib change @XIPREGION [END]IF PACKAGE_OEMDRIVERS to @XIPREGION [END]IF PACKAGE_OEMDRIVERSHIGH for targeted modules.


·         Do the same for platform.reg and the corresponding settings for modules moved in previous step.


·         In platform\<platform>\files\wpc or platform\<platform>\files\smartfon make a copy of oemdrivers.pkd.xml and name it oemdrivershigh.pkd.xml


·         Use guidgen tool to create a new packageguid for oemdrivershigh.pkd.xml.


·         In oemdrivershigh.pkd.xml  change the name entry in the XML file from OEMDrivers to OEMDriversHigh.


 


Example:


This example is for the GPIO driver which is typically pretty small.


; @CESYSGEN IF COREDLL_CORESIOA


; @XIPREGION IF PACKAGE_OEMDRIVERS


IF BSP_NOGPIO !


IF IMG_NOGPIO !


    gpio.dll        $(_FLATRELEASEDIR)\gpio.dll                NK SH


ENDIF IMG_NOGPIO !


ENDIF BSP_NOGPIO !


; @XIPREGION ENDIF PACKAGE_OEMDRIVERS


; @CESYSGEN ENDIF COREDLL_CORESIOA


 


This is what the same section would look like after moving the module to the OEMDRIVERSHIGH package.


; @CESYSGEN IF COREDLL_CORESIOA


; @XIPREGION IF PACKAGE_OEMDRIVERSHIGH


IF BSP_NOGPIO !


IF IMG_NOGPIO !


    gpio.dll        $(_FLATRELEASEDIR)\gpio.dll                NK SH


ENDIF IMG_NOGPIO !


ENDIF BSP_NOGPIO !


; @XIPREGION ENDIF PACKAGE_OEMDRIVERSHIGH


; @CESYSGEN ENDIF COREDLL_CORESIOA


 


Results


For this example I modified platform.bib to move the first 15 modules of Table 2 into OEMDRIVERSHIGH, so that the last 6 modules of Table 1 would rollover into slot 0 and a goal of saving 384KB. Here is snippet from the optimized makeimg.out showing the target modules of slot 1 are now rolled over into slot 0. Occasionally, though not depicted here, the relocated VM space used in slot 1 will be smaller than the VM space being rolled over. When that happens you will see additional files this area also being assigned data and code offsets; each extra module that makes it in is saving at least an additional 64KB. We don’t see this because we rolled over 484KB by replacing it with exactly the same size, which I suspect won’t happen all that often.


 



Module MsgStore.dll         at offset 01d76000 data, 0208e000 code


Module msim.dll             at offset 01d74000 data, 0206c000 code


Module mscoree.dll          at offset 01d73000 data, 0205c000 code


Module netcfagl2_0.dll      at offset 01d72000 data, 02021000 code


Code space full, fixing up netcfd3dm2_0.dll to ram space


Code space full, fixing up ccoredrv.dll to ram space


Code space full, fixing up ccoreutl.dll to ram space


Code space full, fixing up cspvoice.dll to ram space


Code space full, fixing up simsec.dll to ram space


Code space full, fixing up sms.dll to ram space


Code space full, fixing up supsvcs.dll to ram space


Code space full, fixing up btagsvc.dll to ram space



 


Using makeimg.out from a base line image and the optimized image I got these results.


 


Results from baseline makeimg:



Module wendyser.dll         at offset 01220000 data


Module drvr007.dll          at offset 010e0000 data


Module drvr002.dll          at offset 010d0000 data


Module drvr005.dll          at offset 010b0000 data


Module drvr001.dll          at offset 010a0000 data


Module shellcelog.dll       at offset 01090000 data


***** BuildPkg Warnings *****



 


Results from optimized OEMDRIVERSHIGH package:



Module drvr006.dll          at offset 01370000 data


Module wavedev.dll          at offset 01260000 data


Module wcestreambt.dll      at offset 01250000 data


Module wendyser.dll         at offset 01240000 data


Module drvr007.dll          at offset 01100000 data


Module shellcelog.dll       at offset 010f0000 data


***** BuildPkg Warnings *****



 


The last address assigned by makeimg is for shellcelog.dll; for the baseline is 0x1090000, and 0x10F0000 when optimized for OEMDRIVERSHIGH. This is a savings of 384KB.


 


Summary


The potential amount of VM savings will change from platform to platform. Potential savings can be affected by modules sizes sitting on the slot 1 / 0 threshold and the libraries configured to be in the devices modules section. The potential saving is also dependent on how many small to medium size OEM drivers are available to be relocated.  The more you have the greater potential there is for savings.


 


This is not the only way to save VM, nor is this the first step that should be taken to preserve VM. To trouble shoot VM issues a clear understanding of the CE memory model is also needed.


 

Comments (15)

  1. Motanis says:

    Hi wesbarc :

    Thansk for this article! We do enconter VM issue by using WM6 to develop. With your way to investigate, the problem we encounter now is that after choosing some optional components, the free space of slot 1 is quite few for OEM drivers, the following is the part of our makeimg.out:

    ===========================

    Module ril.dll              at offset 01e14000 data, 020be000 code

    Module siminit.dll          at offset 01e13000 data, 020b8000 code

    Module ipsec.dll            at offset 01e12000 data, 020a6000 code

    Module ipsecsvc.dll         at offset 01e0f000 data, 02062000 code

    Module l2tp.dll             at offset 01e0e000 data, 02052000 code

    Module scard.dll            at offset 01e0d000 data, 02047000 code

    Module winscard.dll         at offset 01e0c000 data, 02040000 code

    Module OEM_SMS_Providers.dll at offset 01e0b000 data, 02039000 code

    Module OEM_BTPower.dll      at offset 01e0a000 data, 02034000 code

    Module OEM_Daemon.dll       at offset 01e09000 data, 02030000 code

    Module asyncmac.dll         at offset 01e08000 data, 0202a000 code

    Module atadisk.dll          at offset 01e07000 data, 02023000 code

    Code space full, fixing up battdrvr.dll to ram space

    Code space full, fixing up bthuart.dll to ram space

    Code space full, fixing up bthusb.dll to ram space

    Code space full, fixing up PwrButton.Dll to ram space

    ===========================

    We like to keep those optional components because that is the new and attracting features of WM6. So any good idea for us to solve the problem?

    Thanks in advance!

    Motanis

  2. wesbarc says:

    From the looks of your makeimg.out sample these are modules targeted to OEMDRIVERS and are at the threshold for rolling over into slot 0. If most of your modules were already in Slot 1 then using OEMDRIVERSHIGH probably won’t save you any VM because they were already in slot 1.

    The modules listed in the sample makeimg.out will definitely be part of your image whether they end up in slot 1 for slot 0.

  3. luckytigerwood says:

    Hi Wesbarc:

    It’s interesting and useful article, thanks!

    From the reply of Motanis quesiton, you said ‘If most of your modules were already in Slot 1 then using OEMDRIVERSHIGH probably won’t save you any VM because they were already in slot 1.’ What’s the most means, it’s means just a few files with fixing up or have other flag to determine?

    Thanks in advance.

  4. Motanis says:

    It seems that taking adv. of OEMDRIVERSHIGH package just re-arrange the order of entering slot1 within "OEM modules" during makeimg, right? However, the slot 1 is almost full and occupied by Microsoft’s modules while extracting OEMDRIVERS packages in makeimg. Therefore, most of OEMDRIVERS modules are forced to be put in slot 0 due to "Code space full".

    The skill you provide still has benefits for Virtual Memory usage in this situation?

  5. wesbarc says:

    This is a very difficult subject to explain so thank you for asking these questions to help clarify the subject.

    Your understanding of the process is correct in that we change the order of modules entering slot 1. It sounds to me like the difficulty you are encountering is that before enabling OEMDRIVERSHIGH it is the modules targeted to OEMDRIVERS that are rolling over from slot 1 to slot 0. In this situation you have to modify the documented technique because the modules rolling over and the module being reassigned to OEMDRIVERSHIGH are the same. Some VM savings could come from your effort, but I suspect that if it is OEMDRIVERS modules that are rolling over that you aren’t losing a lot of VM due to slot 0 rollover. In this example slot 0 rollover is consuming 15.4 MB of slot 0 (0x02000000-0x01090000). I suspect yours is much less.

  6. wesbarc says:

    luckytigerwood:

    You are correct. In this case “most” meant a majority of the OEMDRIVERS modules. This stems from the fact that if, for example, 50% of the OEMDRIVERS modules rollover into slot 0, then relocating modules to OEMDRIVERSHIGH would only serve to push the remaining OEMDRIVERS modules that were in slot 1 into slot 0. By taking into account compaction some VM may be saved.

  7. luckytigerwood says:

    Hi Wesbarc:

    Sorry for disturb you again with a question which isn’t related with this article, because I really don’t know who  

    can help me to understand this problem, thanks in advance if you can help me on this or recommend me to consult other expert.

    I do the hopper test with my devices,and I get such result which I never seen before.

    Build = 17740 (OS 1235)(Hop 2.0.16,6225).

    Random Seed = 132143.

    Previous runtime = 1007 mins (16 hrs 47 mins).

    Ended by: UI Unresponsiveness caused by CPU starvation!

    Boot count (prev): 1 (0)

    ACTIONS/min = 107

    Total States = 218

    ……

    ……

    ……

    STATS: System: RUNTIME = 16 hrs 35 mins; ACTIONS = 106966; ACTIONS/min = 107; Total States = 218

    FREE MEMORY: 2400 KB

    FREE DISK:   54219 KB

    EndType = UI Unresponsiveness caused by CPU starvation

    WatchDog: Highest starved priority 251 should ping 618045 ticks ago.

    Would you please help me on it, thanks!

  8. wesbarc says:

    Have you seen this Wiki page? I like using CEDebugX for system hangs.

    http://channel9.msdn.com/wiki/default.aspx/CeDeveloper.BSPHopperInvestigation

  9. luckytigerwood says:

    It’s really a good web site, thanks!

  10. Jun says:

    Hello:

    Can I ask you if we can use IMGVMCOMPACT=1 safely.

    I have noticed this definition in publicwpcoakfileswpc.sku.xml.

    It seems it can put many dlls from MODULES to FILES thus give us more Virtual Memory.

    But I found nothing about its in help.

    So I wandering if we can set IMGVMCOMPACT=1 safely to reserve more VM?

    Thanks

    Jun

  11. wesbarc says:

    I was able to dig up a little information about this flag and I posted it here. http://channel9.msdn.com/wiki/default.aspx/CeDeveloper.BSPVMOptimization. I have also filed a doc bug to get MSDN/OEM docs updated appropriately.

  12. jackdannise says:

    Thanks Wesbarc,

    I used this information to gain 0x50000 bytes by moving in/out the dlls between OEMDrivers/OEMDriversHigh packages. -Jack

  13. jackdannise says:

    Wesbarc,

    We are eliminating the XIP DLL VM here, is it possible to reduce the heap usage by reducing the Thread Stack by either linker option or CreateThread? It’s now set to 64K. 10 threads takes up to 640KB. although it’s only 40~100K used. Thanks,

    -Jack

  14. wesbarc says:

    Have you seen this link?

    http://channel9.msdn.com/wiki/default.aspx/CeDeveloper.BSPVMOptimization

    This is an excerpt:

    “You can also create a heap to allocate from shared memory. Instead of using LocalAlloc for heap allocations from the default process heap, create a heap to allocate from shared memory to preserve process VM. Use CeHeapCreate to create a new heap and define custom allocator/deallocator functions to allocate/deallocate blocks from shared memory. The custom allocator/deallocator functions will be called when the heap size needs to change. Use HeapAlloc to allocate from your heap.”

    Although this is not a complete answer, hopefully this is enough information to get you going.

  15. Bob says:

    Hi wesbarc:

    I encounter this problem,when compile WM611.

    [makecif Error:] (-1): Unable to find file D:WM611releaseWPC_CITRINE_ShipWPCPackagesOEMDRIVERSHIGH.cab.pkg. Exiting.

    please help me!

    Thank you

    Bob