p/invoke gotcha: C++ bool is not Win32 BOOLEAN is not UnmanagedType.Bool


Welcome to CLR Week. I hope you enjoy your stay.

A customer reported that their p/invoke was not working.

We aren't getting the proper return codes from the Audit­Set­System­Policy. When the call succeeds, the return code is 1, as expected. But in our tests, when we force the call to fail (insufficient access), the return code is not zero. Instead, the return code is some value of the form 0xFFxxxxxx, where the x's vary, but the high byte is always 0xFF.

For reference, the DllImport declaration we are using is

[DllImport("advapi32.dll", SetLastError=true)]
public static extern UInt32 AuditSetSystemPolicy(
    IntPtr pAuditPolicy,
    UInt32 policyCount);

The corresponding Win32 declaration is

BOOLEAN WINAPI AuditSetSystemPolicy(
  _In_  PCAUDIT_POLICY_INFORMATION pAuditPolicy,
  _In_  ULONG PolicyCount
);

Alas, the customer fell into one of the common gotchas when writing p/invoke: They confused BOOLEAN and BOOL.

BOOL is a 32-bit integer, whereas BOOLEAN is an 8-bit integer.

Since they were marshaling the return code as a UInt32, they were getting the byte returned by the function, plus three bonus uninitialized garbage bytes. If they studied more closely, they would have found that the erroneous return codes were all of the form 0xFFxxxx00 where the bottom 8 bits are all zero. That's because the bottom 8 bits are the actual value; the rest are garbage.

The correct declaration is to use Unmanaged­Type.U1 aka byte rather than Unmanaged­Type.U4 aka UInt32.

[DllImport("advapi32.dll", SetLastError=true)]
public static extern byte AuditSetSystemPolicy(
    IntPtr pAuditPolicy,
    UInt32 policyCount);

The customer confirmed that switching to Unmanaged­Type.U1 fixed the problem.

Comments (25)
  1. skSdnW says:

    I tend to call it a NT BOOLEAN to differentiate it from a Win32 BOOL. It always seemed to me like they unintentionally leaked out from the NT API to the public Win32 API on the usermode side…

  2. Cesar says:

    > BOOL is a 32-bit integer, whereas BOOLEAN is an 8-bit integer.

    I'd expect 8-bit values to be zero-extended or sign-extended when returned in the EAX register. Where is the garbage coming from? Or is the Win32 ABI one which doesn't have the "small values passed in registers are zero-extended or sign-extended" rule?

  3. Antonio Rodríguez says:

    @Cesar: afaik, no calling convention in x86/x64 sign-extends values smaller than the register whre they are passed, so the garbage comes from whatever the function has left in EAX before storing the return value.

  4. skSdnW says:

    @Cesar: The garbage probably comes from instructions like SETZ AL where only the LSB of EAX is changed.

    @Raymond: Do you know why BOOL is INT sized and not BYTE sized? Alignment? Avoiding having to use "return !!ptrThatIsNullOnFailure;"? (Before the 64-bit era)

  5. Scott F says:

    Or you could set the return type as bool in your P/Invoke definition and set the MashalAsAttribute on the return as UnmanagedType.I1.  That would give you a more natural interface on the managed side.

    [DllImport("advapi32.dll", SetLastError=true)]

    [return: MarshalAs(UnmanagedType.I1)]

    public static extern bool AuditSetSystemPolicy(

       IntPtr pAuditPolicy,

       UInt32 policyCount);

  6. skSdnW says:

    @Myself: blogs.msdn.com/…/10146459.aspx so I guess MS did not really make the choice.

  7. Dan Bugglin says:

    I'm going to leave this here: http://pinvoke.net/

    Don't write your own error-filled DllImport declarations from scratch when someone else already figured it out!

    Of course some of those aren't optimal (for example, some enums are typed as ints) and might require some refactoring or cleanup, but it's a good starting point and can at least help you determine if your p/invoke call is wrong or the way you're calling the function is wrong.

  8. AC says:

    Couldn't Microsoft simply release ready-made p/invoke declarations for their win32 headers?

    (That's a question, not a hidden request. I have no idea about .NET stuff)

  9. Cesar says:

    So I went and took a look at the latest version of the AMD64 ABI. It has no mention of zero or sign extension on function call/return, other than _Bool (C++'s bool) being zero-extended… to 8 bits (and says in a footnote that the rest of the register is garbage). I'd guess Microsoft's x86 ABIs are similar.

    I probably got confused by a RISC ABI I've been reading recently, which does require zero/sign extension. The difference is probably because x86 processors have a lot of instructions which operate directly in the lower byte of a register, while in RISC processors most instructions can only operate in the whole register.

  10. Wear says:

    msdn.microsoft.com/…/ms182206.aspx

    Huh, I wonder if they originally had it returning a bool and then changed it to UInt32 to remove the warning.

  11. jnm236 says:

    @The MAZZTer: I can't count the number of times the p/invoke definitions on pinvoke.net were inconsistent or subtly incorrect. As someone who has uses p/invoke heavily, I've found it usually more efficient to type it out from the MSDN documentation by hand.

  12. CarlD says:

    @jnm236 – I completely agree.  I do check pinvoke.net when I need something, but I've found errors in nearly every pinvoke signature that I've used from that site – usually errors that only show up in 64 bit apps.  I just fixed a bug that had lain dormant in the code since 2006 – only to manifest on Windows 8.1.  This was a pinvoke.net signature for a win32 structure that was wrong in so many ways, I'm truly shocked that it "worked" for 9 years before I tracked it down.

  13. Paul Parks says:

    @CarlD: I tell junior developers, "Borrowed code is like borrowed underwear: it should be thoroughly examined before use, and even then should be assumed to harbor viruses."

  14. John says:

    I too echo AC's question (and I realize this isn't the correct place for it and don't mean to thread-crap).

    Often times if Microsoft had released an officially sanctioned P/Invoke Signature many bugs would have been avoided (or at least had a place to point at to show why they were doing it wrong). However to play devils advocate there are plenty of reasons why they wouldn't do it:

    * Programmers should be trusted to do the right thing and understand their libraries (as avid readers of this blog know this was asking for too much)

    * Who should maintain it? (The .NET Group? "Why show favoritism towards the Win32API and play favorites?" I can hear the Slashdot screams now!)

    * Microsoft's resources are not limitless, the time is probably better spent writing a "Native" managed library that better conforms to the "DotNet-ness look and feel"

    I do know at one time there was a little utility released on Microsoft Downloads that automated the generation of such signatures, but I suspect it was much like TweakUI in the sense that it was "without warranty" and probably got canned in the last great cleansing of tools that did not have full support.

  15. Scott Brickey says:

    @John / @AC

    VB4 through VB6 included a nice little utility that would write the import statements for the Win32 APIs… it was wonderful.

    As far as .Net goes, I would imagine that the header files could generate the first 80% of the code… and a little static analysis could fill in the rest (which I assume would be primarily bit flags and constants)… perhaps a table for mapping internal names to more "publically understandable" field names would be necessary… but I doubt it'd be too terrible

  16. dmex says:

    @John @AC

    Funny enough, I personally added over 3000 pinvoke signatures on the msdn pages after meticulously creating the signatures… Then received a letter signed by two teams (who shall remain nameless per Raymonds Ground Rules) stating that these contributions were spam and would be deleted – after some conversations I was told to stop adding pinvoke signatures on msdn.

    Every single one of those community contributions where removed from my profile but are still visible on the documentation:

    msdn.microsoft.com/…/aa363639.aspx

    msdn.microsoft.com/…/ms646303.aspx

    msdn.microsoft.com/…/ms646303.aspx

    Google also shows these pvinoke contributions:

    http://www.google.com/search

    Just a shame they don't appear on my profile :(

  17. dmex says:

    Edit: It might not be obvious but the google search link opens the msdn site results correctly :P

  18. Joshua says:

    The number of bugs that can be traced to incorrect P/Invokes for Win32 API functions is ridiculous.

  19. Brian says:

    PInvoke.net is a wonderful place to get started, but, people who use it really need to know that the code there is often incorrect, or often what I consider to be inelegant code (the .NET API just "feels" like a literal translation from Win32, rather than something that "feels" like .NET).

    When you do start with PInvoke.Net and come up with a better solution, make sure to add it back (so that I can use your better solution).

  20. cheong00 says:

    For people who wonder whether Microsoft should release their own P/Invoke header equivalents for C#, they have done a huge part of it in form of wrapper libraries. (For example, the System.Drawing namespace is wrapper of GDI+ functions) Just that not all functions are included.

  21. Mark Sowul says:

    I too have found pinvoke.net too unreliable; MSDN magazine published an article with a tool that creates the signatures (you can paste in the unmanaged signature and it will generate the pinvoke signature).  

    blogs.msdn.com/…/making-pinvoke-easy.aspx

  22. cheong00 says:

    @Mark Sowul: Nice to know that. I'm still using the "API Viewer" that primary designed for VB6 and added VB.NET/C# later.

  23. Jerome says:

    Microsoft released something called the WindowsAPICodePack years ago, and I used it for a while but it was never updated. I used it for Win32 shell thumbnails but found it better in the end to do the pinvoke myself.

    @Mark Sowul: I had no idea. Will check it out. What I've done up until now is use pinvoke.net and then run Visual Studio code analysis on the code. It generally picks up the platform-related issues of code that will not work on 64-bit apps. And once you've written a fair amount of pinvoke it gets easier, but it can be time-consuming and laborious.

  24. Chris Crowther @ Work says:

    I find PInvoke.Net a great place to crib off…you just have to check the actual docs for the function, just to make sure it's not being wildly wrong again.

  25. John says:

    @dmex

    Haha yes, I recognize your profile picture from many MSDN pages (shame about their redesign, makes MSDN practically unusable but I digress).

    @Mark Sowul

    Yes that is the tool I remember

Comments are closed.

Skip to main content