Marshalling Helper APIs


I am told that our APIs are not part of our documentation.  :-(  I know for sure we documented these, but I’m told there is a documentation update coming soon, so they must only have made it into the update.  My apologies on behalf of Microsoft.  Keep an eye out for update notifications inside Platform Builder / Visual Studio.


In the meantime, to help you out, I am posting the comments from some of our code.  This is part of %_WINCEROOT%\private\winceos\coreos\core\thunks which does not appear to be part of our shared source.  :-(


//


// Access-checks and marshals a buffer pointer from the source process, so


// that it may be accessed by the current process.  Returns the marshalled


// pointer.  This function allocates resources which must be freed by a


// subsequent call to CeCloseCallerBuffer.


//


// Duplication prevents asynchronous modification of the buffer by the caller.


// If duplication is not required for security purposes, don’t use it.  Then


// CeOpenCallerBuffer can select the most efficient marshalling method for


// best performance.


//


// If duplication is required, allocates a new heap buffer, copies data from the


// source buffer to the heap buffer [if necessary due to “in” descriptors


// ARG_I* or ARG_IO*], and returns the new heap buffer.  If duplication is not


// required, CeOpenCallerBuffer may still duplicate the buffer, or it may


// allocate virtual address space in the current process (VirtualAlloc) and


// point it at the caller process’ memory (VirtualCopy) to create an alias to


// the same memory.  In all cases, any required write-back to the source buffer


// will be managed by CeCloseCallerBuffer [if necessary due to “out” descriptors


// ARG_IO* or ARG_O*].


//


// This call uses ReadProcessMemory and WriteProcessMemory to do its work.  If


// your code is running at a low enough privilege level that it does not have


// access to those APIs, this call will fail with E_ACCESSDENIED.


//


// Does not allocate any resources if the call fails, or if the source buffer


// was NULL.  If this call fails for any reason, the pointer returned in


// *ppDestMarshalled is NULL.


//


// This function opens the caller buffer for synchronous access during an API


// call.  You must call CeAllocAsynchronousBuffer in order to use the buffer


// returned by CeOpenCallerBuffer asynchronously.  Do not close the caller


// buffer until after you have called CeFreeAsynchronousBuffer.


//


// This function is protected by __try/__except so as not to throw an exception


// while accessing the input pointer pSrcUnmarshalled.


//


// Possible return values:


// E_INVALIDARG    pSrcUnmarshalled was NULL, the length was 0, or some other


//                 argument was invalid.


// E_ACCESSDENIED  The source buffer was an invalid address, or your code does


//                 not have sufficient privilege to access the memory.


// E_OUTOFMEMORY   The memory allocation failed.


// S_OK            The allocation (and duplication, if necessary) succeeded.


//


// It is strongly recommended to use the SUCCEEDED() / FAILED() macros to test


// the return value of this function.


//


HRESULT


CeOpenCallerBuffer(


    PVOID* ppDestMarshalled,        // Receives a pointer that the current


                                    // process can use to access the buffer


                                    // synchronously.


    PVOID  pSrcUnmarshalled,        // Pointer to the caller’s data,


                                    // to be access checked, marshalled, and


                                    // possibly duplicated.


    DWORD  cbSrc,                   // Size of the caller’s buffer, in bytes.


                                    // If the ArgumentDescriptor is a WSTR


                                    // or ASTR, then a size of 0 can be used.


                                    // If the size of a string is non-zero, then


                                    // it must include the terminating NULL.


    DWORD  ArgumentDescriptor,      // Descriptor explaining what kind of API


                                    // argument the buffer is, eg. ARG_I_WSTR,


                                    // ARG_O_PTR, etc. ARG_DW is NOT a valid


                                    // descriptor for marshalling!


    BOOL   ForceDuplicate           // Set to TRUE to require a temporary heap


                                    // buffer to be allocated in the current


                                    // process.  Set to FALSE to allow


                                    // CeOpenCallerBuffer to select the most


                                    // efficient marshalling method.


    )


 


 


//


// Frees any resources that were allocated by CeOpenCallerBuffer.


// Performs any required write-back to the caller buffer.  (Due to “out”


// descriptors ARG_IO* or ARG_O*)


//


// This function is protected by __try/__except so as not to throw an exception


// while accessing the input pointer pSrcUnmarshalled.


//


// Possible return values:


// E_INVALIDARG    pSrcUnmarshalled was NULL, the length was 0, or some other


//                 argument was invalid.


// E_ACCESSDENIED  Required write-back could not be performed.  If this error


//                 occurs, resources are still released and the marshalled


//                 pointer is no longer accessible.


// S_OK            The free succeeded.


//


// It is strongly recommended to use the SUCCEEDED() / FAILED() macros to test


// the return value of this function.


//


HRESULT


CeCloseCallerBuffer(


    PVOID  pDestMarshalled,     // Pointer to the buffer that was allocated by


                                // CeOpenCallerBuffer.


    PVOID  pSrcUnmarshalled,    // The source pointer that was passed to


                                // CeOpenCallerBuffer.


    DWORD  cbSrc,               // The buffer size that was passed to


                                // CeOpenCallerBuffer.


    DWORD  ArgumentDescriptor   // The descriptor that was passed to


                                // CeOpenCallerBuffer.


    )


 




 


//


// Re-marshals a buffer that was already marshalled by CeOpenCallerBuffer, so


// that the server can use it asynchronously after the API call has returned.


// Call this function synchronously before your API call returns.  You can not


// call this function asynchronously.  This function allocates resources which


// must be freed by a subsequent call to CeFreeAsynchronousBuffer.


//


// API parameter access (KERNEL MODE ONLY):


//   You can use CeAllocAsynchronousBuffer to get asynchronous access to an API


//   parameter (which would have already been marshalled by the kernel).


//   However if there is any chance that your code will run in user mode, then


//   don’t do this.  Instead follow the user mode instructions below.


// API parameter access (USER MODE):


//   To access an API parameter asynchronously, define the API function


//   signature so that the parameter is declared as an ARG_DW value, so that


//   the kernel does not automatically marshal the parameter for you.  Then call


//   CeOpenCallerBuffer to marshal the parameter.  The asynchronous buffer will


//   become inaccessible if you close the marshaled buffer by calling


//   CeCloseCallerBuffer, so you should call CeFreeAsynchronousBuffer before


//   calling CeCloseCallerBuffer.  In other words, do not call


//   CeCloseCallerBuffer until after you have called CeFreeAsynchronousBuffer.


//


// CeAllocAsynchronousBuffer is not required for buffers that have been


// duplicated by CeAllocDuplicateBuffer.  You do not need to do anything in


// order to use those buffers asynchronously.  Those buffers can be used until


// they are closed/freed.  But if you choose to call CeAllocAsynchronousBuffer


// on a duplicated buffer, it will work.  In that case you must not call


// CeFreeDuplicateBuffer until after you have called CeFreeAsynchronousBuffer.


//


// Does not allocate any resources if the call fails, or if the source buffer


// was NULL.  If duplication is required but no memory is allocated, the pointer


// returned by CeFreeAsynchronousBuffer is NULL.


//


// This function is protected by __try/__except so as not to throw an exception


// while accessing the input pointer pSrcSyncMarshalled.


//


// Possible return values:


// E_INVALIDARG    pSrcUnmarshalled was NULL, or the length was 0.


// E_ACCESSDENIED  The source buffer was an invalid address.


// E_OUTOFMEMORY   The memory allocation failed.


// S_OK            The allocation (and duplication, if necessary) succeeded.


//


// It is strongly recommended to use the SUCCEEDED() / FAILED() macros to test


// the return value of this function.


//


HRESULT


CeAllocAsynchronousBuffer(


    PVOID* ppDestAsyncMarshalled,   // Receives a pointer that the current


                                    // process can use to access the buffer


                                    // asynchronously.


    PVOID  pSrcSyncMarshalled,      // Pointer to the buffer that has already


                                    // been marshalled for synchronous access


                                    // by the current process.


    DWORD  cbSrc,                   // Size of the marshalled buffer, in bytes.


                                    // If the ArgumentDescriptor is a WSTR


                                    // or ASTR, then a size of 0 can be used.


                                    // If the size of a string is non-zero, then


                                    // it must include the terminating NULL.


    DWORD  ArgumentDescriptor       // Descriptor explaining what kind of API


                                    // argument the buffer is, eg. ARG_I_WSTR,


                                    // ARG_O_PTR, etc. ARG_DW is NOT a valid


                                    // descriptor for marshalling!


    )


 


 




//


// Frees any resources that were allocated by CeAllocAsynchronousBuffer.


// Performs any required write-back to the source buffer.  (Due to “out”


// descriptors ARG_IO* or ARG_O*)


//


// This function is protected by __try/__except so as not to throw an exception


// while accessing the input pointer pSrcSyncMarshalled.


//


// Possible return values:


// E_FAIL          Required write-back could not be performed.  If this error


//                 occurs, resources are still released and the marshalled


//                 pointer is no longer accessible.


// S_OK            The allocation (and duplication, if necessary) succeeded.


//


// It is strongly recommended to use the SUCCEEDED() / FAILED() macros to test


// the return value of this function.


//


HRESULT


CeFreeAsynchronousBuffer(


    PVOID  pDestAsyncMarshalled,// Pointer to the buffer that was allocated by


                                // CeAllocAsynchronousBuffer.


    PVOID  pSrcSyncMarshalled,  // The source pointer that was passed to


                                // CeAllocAsynchronousBuffer.


    DWORD  cbSrc,               // The buffer size that was passed to


                                // CeAllocAsynchronousBuffer.


    DWORD  ArgumentDescriptor   // The descriptor that was passed to


                                // CeAllocAsynchronousBuffer.


    )


 


 




//


// Flushes any changed data between source and destination buffer allocated by


// CeAllocAsynchronousBuffer.


// ARG_O_PTR:   Writes back data from asynchronous buffer into source buffer.


// ARG_IO_PTR:  Writes back data from asynchronous buffer into source buffer.


//              Does NOT read from source buffer.


// Others:      Fail with ERROR_NOT_SUPPORTED


//


// This function is protected by __try/__except so as not to throw an exception


// while accessing the input pointer pSrcSyncMarshalled.


//


// Possible return values:


// E_FAIL          Required read or write-back could not be performed.


// S_OK            The read or write succeeded.


//


// It is strongly recommended to use the SUCCEEDED() / FAILED() macros to test


// the return value of this function.


//


HRESULT


CeFlushAsynchronousBuffer(


    PVOID  pDestAsyncMarshalled,// Pointer to the buffer that was allocated by


                                // CeAllocAsynchronousBuffer.


    PVOID  pSrcSyncMarshalled,  // The source pointer that was passed to


                                // CeAllocAsynchronousBuffer.


    PVOID  pSrcUnmarshalled,    // The source pointer that was passed to


                                // CeOpenCallerBuffer, or NULL if the buffer


                                // was an API parameter than never came from


                                // CeOpenCallerBuffer (kernel mode only).


    DWORD  cbSrc,               // The buffer size that was passed to


                                // CeAllocAsynchronousBuffer.


    DWORD  ArgumentDescriptor   // The descriptor that was passed to


                                // CeAllocAsynchronousBuffer.


    )


 


 




//


// This function abstracts the work required to make secure-copies of API


// arguments.  Don’t use it for buffers other than API arguments.  Don’t


// expect the duplicated buffer to be accessible after the API call returns.


//


// Allocates a new heap buffer, copies data from the source buffer to the heap


// buffer [if necessary due to “in” descriptors ARG_I* or ARG_IO*], and returns


// the new heap buffer.  This function allocates resources which must be freed


// by a subsequent call to CeFreeDuplicateBuffer.  Any required write-back to


// the source buffer will be managed by CeFreeDuplicateBuffer [if necessary due


// to “out” descriptors ARG_IO* or ARG_O*].


//


// Duplication prevents asynchronous modification of the buffer by the caller.


// If duplication is not required for security purposes, don’t use it.  Just


// access the caller’s buffer as passed to your API.


//


// Does not allocate any memory if the call fails, or if the source buffer was


// NULL.  If no memory is allocated, the pointer returned in *ppDestDuplicate


// is NULL.


//


// Do not use CeAllocDuplicateBuffer with a buffer marshalled by


// CeOpenCallerBuffer.  Instead have CeOpenCallerBuffer do the duplication.


//


// You do not need to call CeAllocAsynchronousBuffer in order to use the buffer


// returned by CeAllocDuplicateBuffer asynchronously.  The duplicate buffer can


// be used until it is closed by CeCloseCallerBuffer.  CeAllocAsynchronousBuffer


// will not work on buffers that have been duplicated by CeAllocDuplicateBuffer.


//


// This function is protected by __try/__except so as not to throw an exception


// while accessing the input pointer pSrcMarshalled.


//


// Possible return values:


// E_INVALIDARG    pSrcMarshalled was NULL, or the length was 0.


// E_ACCESSDENIED  The source buffer was an invalid address, possibly a pointer


//                 that has not been marshalled.


// E_OUTOFMEMORY   The memory allocation failed.


// S_OK            The allocation and copy succeeded.


//


// It is strongly recommended to use the SUCCEEDED() / FAILED() macros to test


// the return value of this function.


//


HRESULT


CeAllocDuplicateBuffer(


    PVOID* ppDestDuplicate,     // Receives a pointer to a newly-allocated heap


                                // buffer.


    PVOID  pSrcMarshalled,      // Pointer to the caller’s data, that has


                                // already been marshalled.


    DWORD  cbSrc,               // Size of the caller’s buffer, in bytes.


                                // If the ArgumentDescriptor is a WSTR


                                // or ASTR, then a size of 0 can be used.


                                // If the size of a string is non-zero, then


                                // it must include the terminating NULL.


    DWORD  ArgumentDescriptor   // Descriptor explaining what kind of API


                                // argument the buffer is, eg. ARG_I_WSTR,


                                // ARG_O_PTR, etc. ARG_DW is NOT a valid


                                // descriptor for duplicating!


    )


 


 




//


// Frees a duplicate buffer that was allocated by CeAllocDuplicateBuffer.


// Performs any required write-back to the source buffer.  (Due to “out”


// descriptors ARG_IO* or ARG_O*)


//


// This function is protected by __try/__except so as not to throw an exception


// while accessing the input pointer pSrcMarshalled.


//


// Possible return values:


// E_FAIL          Required write-back could not be performed.  If this error


//                 occurs, resources are still released and the duplicated


//                 pointer is no longer accessible.


// S_OK            The free (and write-back, if necessary) succeeded.


//


// It is strongly recommended to use the SUCCEEDED() / FAILED() macros to test


// the return value of this function.


//


HRESULT


CeFreeDuplicateBuffer(


    PVOID  pDestDuplicate,      // Pointer to the buffer that was allocated by


                                // CeAllocDuplicateBuffer.


    PVOID  pSrcMarshalled,      // The source pointer that was passed to


                                // CeAllocDuplicateBuffer.


    DWORD  cbSrc,               // The buffer size that was passed to


                                // CeAllocDuplicateBuffer.


    DWORD  ArgumentDescriptor   // The descriptor that was passed to


                                // CeAllocDuplicateBuffer.


    )


 


And here are the C++ classes from %_WINCEROOT%\public\common\oak\inc\marshal.hpp:


// This class is a wrapper for CeOpenCallerBuffer / CeCloseCallerBuffer and


// CeAllocAsynchronousBuffer / CeFreeAsynchronousBuffer.


// You should only use it with embedded pointers that have NOT already been


// access-checked or marshalled by the kernel.  To duplicate a buffer that has


// already been marshalled, use DuplicatedBuffer_t.  To gain asynchronous


// access to a buffer that has already been marshalled, use AsynchronousBuffer_t.


class MarshalledBuffer_t {


public:


 


    //


    // Access-checks and marshals a buffer pointer from the source process, so


    // that it may be accessed by the current process.  Exposes the marshalled


    // pointer via the ptr() accessor.  Any allocated resources related to the


    // marshalling are freed only by a subsequent call to Unmarshal(), or by the


    // destructor.


    //


    // Typically, you would either use the default constructor plus Marshal()


    // to marshal the buffer, or you would use the marshalling constructor to


    // accomplish the same task.  Use the former method if you require an


    // HRESULT.  Similarly, you can allow the destructor to release marshalling


    // resources, or use Unmarshal().  If an HRESULT is required, use the


    // Unmarshal function.


    //


 


    MarshalledBuffer_t();


    ~MarshalledBuffer_t();


 


    // Please see the description of CeOpenCallerBuffer and


    // CeAllocAsynchronousBuffer for more information about the operation of


    // this function.


    //


    // If marshalling fails, ptr() will return NULL and size() will return zero.


    // Otherwise the marshalled buffer will be accessible via ptr() and size().


    MarshalledBuffer_t(


        PVOID pSrcUnmarshalled,


        DWORD cbSrc,


        DWORD ArgumentDescriptor,


        BOOL  ForceDuplicate = TRUE,


        BOOL  Asynchronous = FALSE


        );


 


    // Takes a const pSrcUnmarshalled, can only be used with ARG_I_* types


    MarshalledBuffer_t(


        PCVOID pSrcUnmarshalled,


        DWORD cbSrc,


        DWORD ArgumentDescriptor,


        BOOL  ForceDuplicate = TRUE,


        BOOL  Asynchronous = FALSE


        );


   


    // Please see the description of CeOpenCallerBuffer and


    // CeAllocAsynchronousBuffer for more information about the operation of


    // this function.


    //


    // Once a MarshalledBuffer is marshalled (using the marshalling constructor


    // or the Marshal() method, it cannot be re-used by calling Marshal(), until


    // after Unmarshal() is called.  An attempt to do so will return


    // ERROR_ALREADY_EXISTS.


    //


    // If Marshal() fails, ptr() will return NULL and size() will return zero.


    // Otherwise the marshalled buffer will be accessible via ptr() and size().


    HRESULT


    Marshal(


        PVOID pSrcUnmarshalled,


        DWORD cbSrc,


        DWORD ArgumentDescriptor,


        BOOL  ForceDuplicate = TRUE,


        BOOL  Asynchronous = FALSE


        );


 


    // Takes a const pSrcUnmarshalled, can only be used with ARG_I_* types


    HRESULT


    Marshal(


        PCVOID pSrcUnmarshalled,


        DWORD cbSrc,


        DWORD ArgumentDescriptor,


        BOOL  ForceDuplicate = TRUE,


        BOOL  Asynchronous = FALSE


        );


   


    // Please see the description of CeFlushAsynchronousBuffer for more


    // information about the operation of this function.


    //


    // If the buffer has already been unmarshalled, or if it is not an


    // asynchronous buffer, Flush will fail with ERROR_INVALID_PARAMETER.


    HRESULT Flush();


 


    // Please see the description of CeCloseCallerBuffer for more information


    // about the operation of this function.


    //


    // If the buffer has already been unmarshalled, Unmarshal will fail with


    // ERROR_ALREADY_EXISTS.


    HRESULT Unmarshal();


 


    // Returns a pointer to the marshalled buffer, or NULL if the buffer


    // has not been marshalled or has already been unmarshalled.


    LPVOID ptr() const;


 


    // Returns the size of the marshalled buffer, or zero if the buffer


    // has not been marshalled or has already been unmarshalled.


    DWORD size() const;


};


 


 


 


class DuplicatedBuffer_t;


// This class is a wrapper for CeAllocDuplicateBuffer / CeFreeDuplicateBuffer.


// It should only be called with API arguments that have already been


// access-checked and automatically marshalled (if necessary) by the kernel.


// All other duplication can be done by MarshalledBuffer_t.


//


// You can either call the constructor to do the duplication, or use


// the default constructor and then duplicate using the Allocate() method.


// If an HRESULT is required, use Allocate().  Similarly, you can allow the


// destructor to release the duplicate memory, or use the Free() method.  If


// an HRESULT is required, use Free().


//


// If Allocate() fails, ptr() will return NULL and size() will return zero.


// Otherwise the duplicated buffer will be accessible via ptr() and size().


//


// Once a DuplicatedBuffer is allocated (using the constructor or the


// Allocate() method), it cannot be re-used by calling Allocate(), until


// after Free() is called.  An attempt to do so will return


// ERROR_ALREADY_EXISTS.


//


// If the buffer is not currently allocated, Free() will fail with


// ERROR_ALREADY_EXISTS.


//


// Please see the description of CeAllocDuplicateBuffer and


// CeFreeDuplicateBuffer for more information about the operation of


// the Allocate() and Free() methods.


//


// Public methods are:


//      DuplicatedBuffer_t();


//      DuplicatedBuffer_t(PVOID  pSrcMarshalled, DWORD cbSrc, DWORD ArgumentDescriptor);


//      DuplicatedBuffer_t(PCVOID pSrcMarshalled, DWORD cbSrc, DWORD ArgumentDescriptor);


//      ~DuplicatedBuffer_t();


//


//      HRESULT Free();


//      HRESULT Allocate(PVOID  pSrcMarshalled, DWORD cbSrc, DWORD ArgumentDescriptor);


//      HRESULT Allocate(PCVOID pSrcMarshalled, DWORD cbSrc, DWORD ArgumentDescriptor);


//


//      LPVOID ptr() const;


//      DWORD size() const;


 


 


class AsynchronousBuffer_t;


// This class is a wrapper for CeAllocAsynchronousBuffer /


// CeFreeAsynchronousBuffer.  It is meant to be used with pointers


// already access-checked or marshalled by the kernel that require


// asynchronous buffer access.  It should ONLY be used in kernel mode!


// See the description of CeAllocAsynchronousBuffer for information on how to


// access a buffer asynchronously in user mode.


//


// You can either call the constructor to allocate the async buffer, or use


// the default constructor and then use the Allocate() method.


// If an HRESULT is required, use Allocate().  Similarly, you can allow the


// destructor to release the async buffer, or use the Free() method.  If


// an HRESULT is required, use Free().


//


// If Allocate() fails, ptr() will return NULL and size() will return zero.


// Otherwise the async buffer will be accessible via ptr() and size().


//


// Once an AsynchronousBuffer is allocated (using the constructor or the


// Allocate() method), it cannot be re-used by calling Allocate(), until


// after Free() is called.  An attempt to do so will return


// ERROR_ALREADY_EXISTS.


//


// If the buffer is not currently allocated, Free() will fail with


// ERROR_ALREADY_EXISTS.


//


// Please see the description of CeAllocAsynchronousBuffer and


// CeFreeAsynchronousBuffer for more information about the operation of


// the Allocate() and Free() methods.


//


// Public methods are:


//      AsynchronousBuffer_t();


//      AsynchronousBuffer_t(PVOID  pSrcMarshalled, DWORD cbSrc, DWORD ArgumentDescriptor);


//      AsynchronousBuffer_t(PCVOID pSrcMarshalled, DWORD cbSrc, DWORD ArgumentDescriptor);


//      ~AsynchronousBuffer_t();


//


//      HRESULT Free();


//      HRESULT Allocate(PVOID  pSrcMarshalled, DWORD cbSrc, DWORD ArgumentDescriptor);


//      HRESULT Allocate(PCVOID pSrcMarshalled, DWORD cbSrc, DWORD ArgumentDescriptor);


//      HRESULT Flush();


//


//      LPVOID ptr() const;


//      DWORD size() const;



Also see %_WINCEROOT%\public\common\oak\inc\pkfuncs.h for the ARG_* values to pass as an ArgumentDescriptor:


ARG_I_PTR           // input only pointer, size in the next argument


ARG_I_WSTR          // input only, unicode string


ARG_I_ASTR          // input only, ascii string


ARG_I_PDW           // input only, ptr to DWORD


 


ARG_O_PTR           // output only pointer, size in the next argument


ARG_O_PDW           // output only, pointer to DWORD


ARG_O_PI64          // output only, pointer to 64 bit value


 


ARG_IO_PTR          // I/O pointer, size in the next argument


ARG_IO_PDW          // I/O pointer to DWORD


ARG_IO_PI64         // I/O pointer to 64 bit value


 

Comments (10)

  1. Posted by: Sue Loh It appears that the MSDN online help was revised in the last day or so; my old shortcuts

  2. Matthew says:

    Sue, thank you for providing this information before the documentation updates were made. You are making up for the slacking of some of your coworkers. I want to clarify this is definitely no stab at you, but rather at Microsoft. I just can’t help but follow through.

    In your "CE6 Virtual Launch keynote: post 1" post a while back, you quoted from the announcement the bit about CE6 being the first / only real-time OS available in source form. I called BS on that, and your follow-up comment indicates you agree with me that there are other real-time OSes available in source form.

    In this post, you mention where these marshalling APIs exist in the kernel build tree, and also state that they are not part of the Shared Source program. Now, if we think about that real quick, that means this necessary part of the kernel is not available in source. This evidence finishes completely shooting down the statement made during the CE6 announcement. Not only are there other real-time kernels available in source, but the CE6 kernel itself is not completely available in source form.

    Essentially, the statement in the original announcement was a complete lie, and Microsoft, I’m calling you on it. Sue, sorry I have to do it on your posts. I think you deserve better than to be part of the face to developers of a company that consistently lies to developers, users, governments and basically anyone willing to lend them an ear.

  3. ce_base says:

    Well, the marshalling APIs aren’t part of the kernel.  They are part of coredll.  In fact, they are intentionally not part of the kernel, so that they can’t do anything that kernel drivers (k.coredll.dll) or user mode drivers (coredll.dll) can’t do already.  Because we didn’t want the marshalling APIs to accidentally expose access to memory that the caller of the marshalling APIs didn’t have rights to access.

    I actually agree with the statement that the kernel is 100% available in shared source.  Though the statement puzzles me a little, because I thought it was true of our shared source in CE5 too.  After all, I’ve seen people on our newsgroups asking how to rebuild the kernel source.  But I haven’t gone digging to see what might have been missing in CE5.

    Thanks for giving me credit for trying to be honest about things, but I don’t think the rest of Microsoft is trying to be dishonest.  I know a lot of people love to hate us, but really I think my co-workers, including our management and marketing team, do try to be honest about things.  They try to ‘spin’ things in the best possible light of course; that’s their job I guess.  But I don’t think they are intentionally lying.  You can call them ignorant, or lazy about doing the research before making statements like they did.  I don’t think they are dishonest.

    Of course, how much you end up believing that depends on how much honesty you give me credit for.  :-)  The best I can do is show by example that I’m willing to admit mistakes or acknowledge problems.

    Sue

  4. ce_base says:

    PS.  Some of the blame for the marshalling APIs not getting properly documented in the release version of CE6 is my own, too.  We get so busy writing code and fixing bugs that reviewing documentation gets delayed too long.  By the time we really did a good job reviewing documentation in CE6, the release version of the docs was already frozen.  So our fully reviewed docs will be the first update.

    So absolutely, some of the blame is my own.  IMO it is also partly a management problem, that our team in general needs to put higher priority on getting docs right; and partly a tools problem, that our team can’t properly track doc reviews and quality.  (And, probably, this is a problem with Microsoft in general, and in fact with many software companies, though that’s no excuse for us not doing our jobs right.)  I hope in the future we’ll get better about that too.

  5. ce_base says:

    (Hah you must have touched a nerve because I keep thinking of more things I want to say. :-) )

    Also please don’t say I deserve better than to work here.  I make no claims about the rest of Microsoft, but when it comes to Windows CE / Mobile, I work with a world class team of smart people who care very much about doing a good job.  Maybe we don’t always get everything right, but everyone is trying.  And quite competent in my opinion.  I’m proud to work here, don’t want to be anywhere else.

    Sue

  6. Matthew says:

    I guess I did, but its really easy for me to get worked up with such an overall negative view of Microsoft. The comment about you deserving better was basically saying that I put you in a higher class than the overall company. It may just be that I’ve had the chance, through meeting in person and reading your posts, to see that you know what you’re doing and aren’t afraid to talk about the negative side. I’ve run into plenty of Microsofties that simply cannot see any aspect of Microsoft in anything but a positive light. If I didn’t know better, I would assume the same of you, which is where the comment came from.

    I’m sure its not always intentional, but there are limits. A small detail being off, fine, everyone makes a mistake. But as you move up in the company to higher levels of management, these mistakes become more visible and tend to be of greater consequence. At a level as high as the person making the claim about the first real-time kernel available in source form, there should be at least some basic fact checking going on. Claiming ignorance seems like a pretty weak excuse for what looks a lot more like marketing spin. The uninformed may believe it, but anyone with a little knowledge is more likely to view it either as an outright lie or some perverse stab at the Linux, almost saying that it isn’t a real-time OS.

    The impression I get isn’t that the individual developers are trying to do anything wrong, but that the whole organization as a whole does a lot of bad things. Of course there are the excepts, such as one of the guys at MEDC telling a Russian programmer that he should encourage his countries IP laws to be brought up to ‘US standards’ so they could benefit from Shared Source. The IP laws current in effect here are quite screwed up and are one of the things that make me dislike this country right now. The idea of forcing those upon other countries is quite disturbing.

    I’ve heard some really messed up bits about the development environment at Microsoft that mostly boils down to mismanagement. Hierarchical source control that takes months to fully distribute changes, roaming groups that join other teams to push a particular technology (which might make no sense at all to use) which then move on as soon as their goal is accomplished, etc. I haven’t heard anything about documentation, and would be interested in hearing a little more on that now that you have alluding to writing it yourself.

    I have always assumed the documentation was written by a different group because it is so far distanced from the code. If I reported every documentation error I found, I’d spend at least a quarter of my time doing that. Grammatical errors are common, which makes me think the docs aren’t even being proofread. The real annoying part is when the docs are just plain wrong. There are a number of instances where a function behaves significantly different than what the docs claim, and figuring out which one and under what conditions wastes days at a time. There are also a number of instances where the documentation seems to have a fundamental disconnect with the portion of the system at hand. So I wonder, who writes to docs and who checks the docs? These are one of the worst aspects of development on the Windows platforms.

    Prior to working on Windows, I had heard a from a variety of sources of the wonderful developer support Microsoft provides. Now that I’ve been working as a full-time developer on a Microsoft platform for a significant time, I have to wonder what crack these people were smoking. Between documentation that is just plain wrong and a compiler riddled with bugs ranging from the obscure code mishandling to basic pieces of the Visual Studio GUI just plain not working, this is most strenuous platform on which I’ve ever had the displeasure of working. I’m sticking with my current employer due to a bit of loyalty, but I will never take another job where Windows is the primary platform for which I write. I don’t need the late nights and long weekends spent furiously trying to make deadlines that are always cut close due to some piece out of my control.

    I have contemplated starting my own blog dedicated to the completely screwed up Windows platforms from a dev point of view. It hasn’t happened yet only due to my lack of time since dealing with this mess consumes all my time. If I do start it sometime, I will have no shortage of material for the foreseeable future.

  7. Matthew says:

    I got off on a tangent in that last reply and forgot to touch a whole topic, so here’s the rest.

    I assumed the marshalling API would be part of the kernel. It maps memory between processes, some of which can be running in the kernel. If this isn’t in the kernel, then I really have to wonder what else is out of kernel, and thus not publicly visible, that would be a basic part of the system.

    With CE5, the Shared Source included little snippets that were useful, but had big chunks missing that are essential parts to have a useful OS. Pieces of particular interest to me were filesys.exe and gwes.exe. With CE6, we are told these things, along with device.exe and all drivers not living in the new udevice.exe, were being moved into the kernel. With the announcement about all the source being available, I thought great, I can finally see those parts and get some clear answers an a number of bugs. Not that I want to go digging through that mess, but if I could I could at least see exactly what is happening and thus know exactly how to work around the problems. Then I pulled down the CE6 wad and saw the Shared Source had barely doubled in size. I quickly glanced through the directories and saw that there are more pieces, but not really enough to be complete. I haven’t yet gone looking for stuff like the FSMgr, but I don’t have high hopes for it between teh small increase in size of the source tree and the knowledge that something as basic as the marshalling API isn’t in kernel.

    That raises the question of what in kernel means. I think its clear that depends on the context. All the aforementioned systems probably now execute in kernel space, which is important to know when interacting with those parts of the system. It sounds like the kernel is still a very small piece with only minimal functionality, still relying on external pieces to plug into to allow it to present a GUI, access files, or even shuffle data for API calls.

    Perhaps my stab at Microsoft will have to be limited to only contesting the claims of ‘first’ and ‘only’ while granting possible truth to the claim of the kernel actually being completely available in source form when speaking in the context of what exists in the kernel binary as opposed to what executes in kernel space.

  8. ce_base says:

    It is funny that a fairly "emotionally charged" comment thread like this is running on what is possibly the driest blog post I ever wrote…  :-)

    Well I guess all I can say to the quality issues is, while they definitely do happen, part of the thing about Microsoft being one of the largest software companies in the world is that we have one the largest sum totals of problems.  I am not sure that if you could somehow compare the ratio of good to bad that we are much worse than many companies in the world.  But everyone manages to remember more bad than good, it seems.  Perhaps I am being naive or optimistic when I say that.  But all around me I see smart people trying to make things better, and I have a hard time believing anyone else would end up with better products.

    As far as docs go, I won’t go deep into my opinions about what we need to do to fix our docs.  I’ll agree that docs can and have been badly neglected or incorrectly edited to the point of being totally, embarrassingly, wrong.  And that needs to be fixed.  All I can do is try my best to influence the process.  I believe fixing our documentation is a task big enough to warrant a whole new product team, and I’m not willing to quit Windows CE to create new doc tools.

    Sue

  9. ce_base says:

    Oops we crossed over.  OK I see, the dispute over kernel source is a problem of unclear terminology.  I mentioned this in other blog posts, but yeah, people really should be more careful about distinguishing between the kernel PROCESS (you could call it nk.exe but I generally don’t because in CE6 nk.exe is really a rename of oal.exe which the OEM writes) and the actual kernel module (kernel.dll).  The kernel module isn’t any bigger in CE6 than it was in CE5.  In fact it’s smaller because now the OAL is in a separate .exe.

    100% of the source code of kernel.dll is shared in CE6.

    But not all of the modules that load into the kernel process (there are many) are part of shared source.

    • filesys is still premium

    • fsdmgr is shared: privatewinceoscoreosstorage, and by the way most of the old file system load/unload functionality from filesys moved into fsdmgr I believe.

    • I don’t know gwes well but after a quick look it looks mostly premium

    • Some of coredll is shared: privatewinceoscoreoscore, but the part that contains the marshalling code, thunks, is not part of any source code sharing.  I don’t know why; I’m guessing there’s a reason some of it couldn’t be released, and someone didn’t take the time to sort out what was OK from what was bad.  I reported it and asked for at least the marshalling code to be exposed.

  10. This is one of those blogs where I hesitated to post it because it may be going too hard core into the