The Windows CE Build Process


I've been working recently with some customers who have had to make some tweaks to our build system and they had a heck of a time of it.  We have very good reference material on MSDN, but I don't know of anyplace that ties it all together.  So using my favorite component, the web server, I'm going to through how we turn C code into DLL's into ROM images. 


If you're adding a new, custom component to CE you probably don't have to go through all this.  You should just build your EXE or DLL directly and burn it into the ROM directly.  However I hope this insight of how we do this internally at MS is useful.  Hopefully this will at least convince you we're not crazy for making things the way they are :).  Of course there's weird cases that don't exactly follow these rules, but for now we're just going to look at a straightforward component.


Also when I say "you" I'm assuming you are an OEM who has Platform Builder and creates your own platforms.


10000 FOOT OVERVIEW
* STEP 1 BUILD: Turns C code into libraries.  Microsoft does this stage and gives pre-built libraries to customers in Platform Builder.  The remaining stages you perform yourself (whether or not you realize it :)).
* STEP 2 SYSGEN: Turns libraries into DLL's, based on which libraries are required for componentization.
* STEP 3 BUILDREL: Copy all the DLL's from stage 2 into a central location.
* STEP 4: MAKEIMG: Munge all the DLL's, registry settings, etc... into a ROM image.


It's up to you to get the ROM image onto your device.  That's outside scope of this article.


As you read this, if you have PB it may be helpful to have a command line window open as I walk through this.  To do this, go to Build OS->Open Release Directory.  (The exact wording/menu location of this may change from version to version.)  Even if you totally use the IDE and never cmd line to build, the principles are the same since the IDE is just a wrapper.


Also note I'm being lazy on directory names below - when I say \private\servers\http to be technically correct I should say %_WINCEROOT%\private\servers\http.



***** STEP 1 BUILD: C CODE -> LIBRARIES *****
The Web Server (HTTPD) source code is checked into \private\servers\http.  It's part of the shared source package if you want to see it.  The first thing we need to do is turn all the C code (C++ in HTTPD's case) into intermediate libraries.


Microsoft runs a tool named cebuild to do this.  It will first run 'build' (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wcepb40/html/_wcepb_windows_ce_build_environment_tool.asp) on \private\winceos, then on public\winceos, then it will run sysgen (see step 2).  Then it runs build on \private\dcom, then \public\dcom, then sysgen -p dcom, etc...  Eventually it gets to \private\servers, the tree we care about.


In \private\servers, there is a file named dirs.  This lists the directories that build should recurse into and also the order they should be visited.  These may have other dirs files.  Node directories that contain source to build contain a file named SOURCES.  This is effectively a dumbed down makefile, listing C files to build and various other rules.  Build.exe is smart enough to figure out various dependencies, rather than requiring it be done explicitly via nmake files.


\private\servers\http contains a bunch of sub directories, each one of which builds a separate library.  For instance .\core contains basic server stuff like logging, HTTP request+response handling, and services.exe interface.  .\auth contains HTTP authentication work, and .\webdav contains the webDav piece.  They respectively build httpd.lib, httpauth.lib, and httpdav.lib.  These files are placed into \public\servers\oak\lib\<CPU>\<debug|retail>\.


These libs are then shipped to you, the customer.  We do not ship the web server as a DLL itself.  You build this during STEP 2.  Note that even if you don't build a device with the web server, you will still get all these library bits for all processors and for both debug+retail.  That's why PB takes so much space on your hard drive.  After all, we're not psychic and can't predict whether you want to build a web server or not before we send the bits to you.


There's no difference whether a LIB was built from \private\servers or \public\servers as far as the build is concerned.  This is just a shared source/licensing thing with regards to where the source code is placed.



***** STEP 2 SYSGEN: LIBRARIES -> DLLS *****
If you want all the details of the sysgen tool, check out http://msdn.microsoft.com/library/en-us/wcepb40/html/_wcepb_Sysgen_Tool.asp?frame=true.


Why don't we ship the Web Server DLL?  Wouldn't it be easier and faster to NOT have customer link libs to the DLL?  The issue is that CE is a componentized operating system.  Some DLL's have dozens of different permutations they can be built with, depending on the scenario.  MSXML is a good example - i.e. SAX, XMLHTTP, XSLT are all independently configurable.  It's impractical for MS to build all combinations of such a DLL.  That means we give you the libs and a process to turn these into DLL's on your own.


The Web Server is also a componentized.  To keep things simple, for now let's look at the standard non-componentized DLL - the telnet server.  Telnet built telnetd.lib in the same dir as the web server lib was placed in in STEP 1.


If you want the telnet server, you need 'SYSGEN_TELNETD=1' set in your command window.  (The PB IDE does this under the covers if you drag Telnet Server into the catalog.)  In the file \public\cebase\oak\misc\servers.bat (which is called when sysgen is run), there are some lines that look like this:


    if "%SYSGEN_TELNETD%"=="1" set SYSGEN_CMD=1
    if "%SYSGEN_TELNETD%"=="1" set SYSGEN_STDIO=1
    if "%SYSGEN_TELNETD%"=="1" set SYSGEN_SERVICES=1
    if "%SYSGEN_TELNETD%"=="1" set SYSGEN_FSPASSWORD=1


These lines indicate that if telnet was included, then to bring in CMD.exe, STDIO, services.exe, and the filesystem password stuff as well.  This saves you from having to figure out all its dependencies.


After that line a little ways, we have this:
if "%SYSGEN_TELNETD%"=="1"  set SERVERS_MODULES=%SERVERS_MODULES% telnetd


(Digression: This is CE 5.0 only.  In CE 4.0-4.2 we had a similar SYSGEN_XXX idea but the files were in either \public\iabase\oak\misc\cesysgen.bat or \public\hlbase\oak\misc\cesysgen.bat.  CE 3.0 was different but we have too much info already)


This means that as we go through the sysgen phase, this SERVERS_MODULES has a list of all the various servers modules (anything built in \public\servers or \private\servers).  Now we need to take some action based on this list.  In \public\servers\cesysgen\makefile we have standard makefile rules for building the component.  The lines for telnet are as follows.


telnetd::
    @set SOURCELIBS=$(SG_INPUT_LIB)\$@.lib
telnetd::
    @set TARGETNAME=$@
    @set RELEASETYPE=OAK
    @set TARGETLIBS=$(SG_INPUT_LIB)\$@.lib $(SG_OUTPUT_SDKLIB)\ws2.lib $(SG_OUTPUT_SDKLIB)\coredll.lib $(SG_OUTPUT_SDKLIB)\ceosutil.lib  $(SG_OUTPUT_SDKLIB)\authhlp.lib 
    @set DLLENTRY=_DllEntryCRTStartup
    $(MAKE) /NOLOGO $@.dll

The magic here is that we set a bunch of environment variables (SOURCESLIBS tells which libraries are "core", TARGETLIBS are what we link to, etc...) and then a call to $(MAKE) executable.  This exe looks at the variables and then spits out telnetd.dll in \public\cebase\cesysgen\oak\target\{CPU}\{debug|retail}.  (Again this dir is CE 5.0 only, though on prev version the concept is the same.)


How about the Web Server?  In this case, there are 2 separate SYSGEN variables for it.  SYSGEN_HTTPD brings in the core functionality except WebDAV, which requires SYSGEN_HTTPD_WEBDAV=1 to be set.  In either case, set SERVERS_MODULES=%SERVERS_MODULES% httpd is specified.  In order to help make figure out whether it needs to build just core HTTPD or Core+WebDAV, we introduce a new variable in public\cebase\oak\misc\servers.bat named HTTPD_COMPONENTS.  This has a bunch of HTTPD libraries (httpauth httpfilt ...) always and may include httpdav if DAV is included.


When it comes time to sysgen the web server, we tie all this together via this rule in the makefile:
httpd::
    @set TARGETLIBS=
    @set SOURCELIBS=$(SG_INPUT_LIB)\$@.lib $(SG_INPUT_LIB)\httpparse.lib
httpd::$(HTTPD_COMPONENTS)
    @set TARGETNAME=$@
    @set RELEASETYPE=SDK
    @set TARGETLIBS=%%TARGETLIBS%% $(SG_INPUT_LIB)\$@.res $(SG_OUTPUT_SDKLIB)\coredll.lib $(SG_OUTPUT_SDKLIB)\ws2.lib $(SG_OUTPUT_SDKLIB)\ceosutil.lib $(SG_OUTPUT_SDKLIB)\authhlp.lib $(SG_INPUT_LIB)\httpstubs.lib
    @set DLLENTRY=_DllEntryCRTStartup
    @echo TARGETLIBS=%TARGETLIBS%
    @echo SOURCELIBS=%SOURCELIBS%
 $(MAKE) /NOLOGO $@.dll
httpauth httpasp httpextn httpfilt httpisapi:
    @set SOURCELIBS=%%SOURCELIBS%% $(SG_INPUT_LIB)\$@.lib
httpdav:
    @set SOURCELIBS=%%SOURCELIBS%% $(SG_INPUT_LIB)\$@.lib
    @set TARGETLIBS=$(SG_OUTPUT_SDKLIB)\ole32.lib $(SG_OUTPUT_SDKLIB)\oleaut32.lib $(SG_OUTPUT_SDKLIB)\uuid.lib


What this does is that "httpd::$(HTTPD_COMPONENTS)" indicates that HTTPD has dependencies on the various libraries (i.e. httpddav) and that these rules should be executed first.  So httpauth rules adds httpauth.lib to the SOURCESLIBS variable list.  httpdav not only adds httpdav.lib to SOURCESLIBS, but it also has target libs include all the COM libs.


(UPDATE (2/20/07): My description of SOURCESLIBS and TARGETLIBS is correct in how they are used in practice to do stubs, but the actual implementation details I describe are wrong.  In the comments of this blog below I have the nitty gritty from the architect, not included it here since there's so much info already.  Note I got by here for 8 years not knowing the build magic myself so this probably isn't a critical distinction.)


(Digression: Since HTTPD always requires httpauth httpextn httpfilt and httpisapi, I should've just added them in the "@set SOURCELIBS=$(SG_INPUT_LIB)\$@.lib $(SG_INPUT_LIB)\httpparse.lib" line.  But I was young and wanted a super-componentized web server (i.e. no authentication option)!  This can't be built in reality because SYSGEN_HTTPD always sets httpauth as a target that it depends on and hence the lib is always included in the final httpd.dll.  I guess you could hack it out, but that would be foolish.  Always use web server auth!)


This is also the stage where the componentization magic takes place.  The Core Web server logic calls a function in its parser called HandleWebDav() which (surprise surprise) handles WebDav requests if the right HTTP verb is sent by a client.  However, what happens if the WebDAV library isn't linked to httpd.dll?  How do we still link, and even if we can somehow link how do we handle it at runtime?


The answer is in "@set TARGETLIBS=... $(SG_INPUT_LIB)\httpstubs.lib"  This library has a dummy stub function named HandleWebDav() which is just a few lines long and returns an error indicating WebDAV isn't supported.  This lets us link and it means we could theoretically implement custom error handling of this case at runtime if we wanted to. 


That brings up a new question -- how do we avoid multiple definitions of HandleWebDav() if it's defined in both httpdav.lib and httpstubs.lib?  The answer is that if the linker sees HandleWebDav() in both a library in the SOURCELIBS list (httpdav.lib) AND TARGETLIBS list (httpstubs.lib), then it will favor the SOURCELIBS one and disregard the TARGETLIBS assuming that this is a stub.


Besides generating DLL's, sysgen also generates .reg and .bib files that specify the initial CE registry and the DLL's and other files that need to be in the ROM.  \public\servers\oak\files\Servers.reg contains all possible registry settings for stuff from the servers project.  It's just like how we give you all the libs for CE, knowing you'll never need all of them on the same device.  Since your image won't include everything, sysgen will make a copy of servers.reg in \public\CEBase\cesysgen\oak\files\servers.reg and will strip out unused component's registry (or .bib) settings while at it. It only includes stuff in "; @CESYSGEN IF" for the modules that were built - ultimately determined based on the SYSGEN_XXX environment variables.



***** STEP 3 BUILDREL: DLL'S -> _FLATRELEASEDIR
It gets easier from here, I promise.  The sysgen process generates files all over your release tree (DLL's, .reg/.bib, etc...), which I won't get into.  The BuildRel tree simply copies them from these dirs into a single directory, specified by the %_FLATRELEASEDIR% cmd environment variable.  BuildRel is documented at http://msdn.microsoft.com/library/en-us/wcepb40/html/_wcepb_Build_Release_Directory_Tool.asp?frame=true.



***** STEP 4 MAKEIMG: _FLATRELEASEDIR -> ROM IMAGE *****
Whew!  We have all our DLL's, .reg, .bib, and other random files together in one place.  Now we just need to run makeimg.  Makeimg is documented at http://msdn.microsoft.com/library/en-us/wcepb40/html/_wcepb_Make_Binary_Image_Tool.asp?frame=true.


Makeimg looks at the .bib files to see what DLL's and other files to include in the ROM and .reg to figure out the initial registry.  There are also some environment variables that allow further tweaking, though these are outside of the scope of this entry.


And that's it - enjoy!


[Author: John Spaith]

Comments (16)

  1. JayLee says:

    It was very helpful! They should have this in the next PB.

    jl

  2. A.M. says:

    Thank you. That is helpful. Great work.

  3. G.S. says:

    Thanks!

    I wouldn’t have figured it out without this.

  4. Gildas Bayard says:

    Very interesting!

    There’s something I’m still wondering about though. I’m not sure I understand the purpose of the postproc phase.

    As an example I’ve looked at the publicshell directory. There I see in cesysgen subdirectory the makefile used for the sysgen phase.

    In it I see:

    preproc: $(SHELL_MODULES) files

    postproc: $(SHELL_REPLACE_MODULES) postlocalize

    I understand the usefulness of the postlocalize rule: move the localized ressources to the final "SG_OUTPUT_XXX" directories

    But my question is more specifically on the SHELL_MODULE vs SHELL_REPLACE_MODULES.

    If I understand correctly SHELL_MODULES contains all the modules that makes up the SHELL project (SHELL_MODULES=explorer shcore ceshell gvgap). SHELL_REPLACE_MODULES is empty by default but if I change one of the modules, say explorer, then I should remove "explorer" from SHELL_MODULES and add it to SHELL_REPLACE_MODULES, right?

    The only consequence is that my modified explorer will be assembled after the other modules. But why is it so important? Why can’t it be assembled during the preproc phase?

    Could you provide an example showing why it’s necessary?

  5. cenet says:

    Gildas – first happy that this article was useful for you.  And congratulations, you’ve managed to stump me on a build question :(.  I’ve never dealt with the replace modules.  I looked through our build tree and only saw a few examples of components actually uses replaceModules.  I believe (this is only from reviewing makefile, so take this with grain of salt) that this is if someone wanted to replace some component from Microsoft that had the same name.  

    For your question, on your development environment I don’t think it’s a huge deal whether you do the replaces modules or not.  Whatever gets you the binary that you want in a clean & reproducable way is what you should stick with.  In general people that modify MS source don’t touch the replaces stuff, they just let the standard build pick up their modified lib and sysgen it to DLL they want.

    If this isn’t helpful I’m afraid I’d have to point you at a CE newsgroup, http://blogs.msdn.com/cenet/archive/2005/12/05/500181.aspx has a list.

    Sorry I’m not more help here,

    John

  6. Mark Miller says:

    Most of the REPLACE_MODULES is so that internal teams in Microsoft (for instance Smartphone, PPC or others) can modify portions of the OS code (like an edit control, of the shell notification services) while still leveraging the moving target that is the general OS components that deliver to Platform Builder (where teams have daily builds of the OS components and daily builds of their products)

    For an OEM that get’s platform builder it’s probably easier to either replace the build rules for the component completely and/or have your replace lib be created to be linked as part of the preproc phase.  You will need to be carefull that any QFE released does not modify your changes and if so re-apply them as necessary.  

    We’re cleaning up a lot of this for our next release so hopefully some of the confusion will go away.

    Mark Miller

  7. cenet says:

    Note my description of the SOURCESLIBS vs. TARGETLIBS is not technically correct (though it does capture what they’re used for in practice).  From one of our other architects,

    <

    There is no favoritism shown to SOURCELIBS or TARGETLIBS.  What actually happens is that building a dll or an exe during sysgen is actually done in two parts.  First, all of the non-stub static libraries for the module are linked into another static library named $(TARGETNAME)_ALL.lib.  If I remember right, the linker will resolve all of the names that it can within this new lib.  This lib contains all of the โ€œrealโ€ functionality of the executable module.  Second, the real module link is done with $(TARGETNAME)_ALL.lib listed as the first library to be processed.  Only the unresolved names will be picked up from the stub library.

    >

    John Spaith

  8. smith white says:

    It’s really good work and interesting

  9. Paul Tobey says:

    John,

    I don’t think that this stuff about _all.lib is actually right, either.  In some cases, coredll, oleaut, etc., it gets done that way and it would be fine with me if MS would like to make everything work the same way!  However, for most of the items which are built into EXEs and DLLs at SYSGEN time, it’s not done that way.  The Shell, for example, in PUBLIC, doesn’t do this, and niether does NetUI (two items that I happen to have all too much experience modifying because of the size of our screens, 640×240).  If there’s a generalized statement about how, exactly, those things get built, I’d sure like to know what it is!

    Good article, though, and very useful, particularly for those just getting started with PB and trying to understand where those hundreds of DLLs and EXEs come from!

    Paul T.

  10. Sam S. says:

    John,

    Your blog post was an excellent read.

    Could you comment about how coredll.dll and coredll.lib are generated and used?  It has been challenging wrapping my head around these components that (I believe) tie everything together during runtime.

  11. John Karlsson says:

    Sam S. – Sue L wrote a blog about how WCE API calls work. It doesn’t tell you much about how coredll is generated, but it does tell you how it’s used/works. Moreover it’s an interesting read ๐Ÿ™‚ I do however know that what goes in coredll is decided by what OS components you sysgen for – others are filtered out. Take a look at the coredll.def file and you’ll see what I’m talking about.

    http://blogs.msdn.com/ce_base/archive/2006/02/02/Inside-Windows-CE-API-Calls.aspx

  12. Jimmy Carter says:

    Finally I know why the sources described in commonoak should produce .lib only but it does not. It generate .dll also. It’s all because the step2. Porting driver from publicoak to private platforms needs lots of working to indicate which part of librarys I need. Sigh~

  13. Rajasekaran says:

    This is the best article to start with, if anyone is trying to understand the Win CE Build process.

  14. Prabhu Kumar says:

    Hi,

    Thank you so much for such a clear description of the build process!! ๐Ÿ™‚ I have always used the build commands and always used to wonder how it all works. This certainly has made it a lot clear, thanks!

    Prabhu

  15. wushibin says:

    Thank you! It is very helpful!

  16. Susan Wolber says:

    Great article!

    One thing I’ve never understood in the build process is when to use just an "IF" and when to use an "@CESYSGEN IF".  So in a .dat, .bib, .reg – when do I need to use the @CESYSGEN versus just a plain if?

    For example in   <WINCE600>PUBLICWCESHELLFEOAKFILESwceshellfe.dat

    there are both being used – plain "IF" is used on LOCALE and IMG checks while MODULE checks use @CESYSGEN IF’s.

    I’m mostly doing IF’s on various SYSGEN variables, so how do I know which flavor of IF to use?

    Thanks!

Skip to main content