ML64 bug to watch out for

If you’re porting your application to x64, and you use much in the way of __asm in your x86 code, you’re likely to start looking at ml64 – the 64 bit version of Masm.  The reason you’re likely to do this is that the x64 compiler doesn’t support __asm blocks in C code.  So you can either use the compiler intrinsic functions [and there are a lot of ’em, and they’re all documented relatively well], or you have to use ml64.  For folks that aren’t new to Masm, you’re also likely to try to use some of the nifty little time-saver features of Masm to automatically generate prologue’s and epilogues for your functions.  For ML64, DO NOT DO THIS!
Here’s an example:
testfunc PROC uses rbx
xor rbx, rbx
testfunc ENDP
There are 2 problems with the code that ML64 generates.  #1:  There is no .xdata of any sort.  No unwind directives are emitted, despite the fact that you’re allocating stack, and saving a nonvolatile register.  #2:  The epilogue is invalid – it uses the ‘leave’ instruction, which is ineffecient, but also just plain illegal in an x64 function epilogue.  The stack unwind routines will not properly recognize the instruction sequence as an epilogue, so the debugger and the EH routines will all fail [the EH routines will just terminate your process if an exception occurs while testfunc is on the stack].

BTW – Here’s the link to the customer bug.  It came in about 3 months too soon for serious consideration for VC8, so we punted it to the next version, but now it’s pushed out until we get some more resources for investment in MASM.

Comments (0)