Some tips for P/invoke
Here are some tips I've learned the hard way after wrapping some nasty things into managed code (including wrappers for the native debugging API and reading minidumps):
- Use resources like Pinvoke.net and Fxcop (which may have some useful marshalling checking rules).
- Beware of 32-bit vs. 64-bit. One thing I note is that https://pinvoke.net has a general error of porting SIZE_T as uint; where as it really should be IntPtr. Fortunately, FxCop has a rule to catch this!
- Have good examples on hand. You can see some of the BCLs imports in the Rotor source at $\sscli20\clr\src\bcl\microsoft\win32\win32native.cs.
- For really bad stuff, prototype the APIs in unmanaged code first. In worst case, you can spin up a managed and native debugger and simultaneously debug the algorithm in both, using the native case as a baseline.
- Get friendly with the Memory window in your debugger and structure offset.
- Use an unmanaged debugger to verify field offsets of the native structure. Double check with the raw unmanaged headers. Use other documentation (such as MSDN) cautiously, since it may be wrong.
- Never compute SizeOf of a non-trivial structure manually from inspecting the docs in MSDN. There's too much room for error. Even if you get through all the padding, alignment, and #define issues, the docs may be be wrong. When in doubt, just check the actual values in the native debugger.
- Pay close attention to data being shifted. Eg, if all the bytes are 4 bytes off from where there should be.
- If things are going bad, try replacing more elaborate structure definitions with a bunch of uint, just to verify that you're getting the correct bytes into managed code.
- For ugly structures, you may not need to wrap the entire structure. For example, if you just need the first 3 fields; you just need to wrap the first 3. This is especially handy if later fields are hard to do (eg, ugly unions). Or if you just need the last field, you can use an ExplicitLayout and just pick that up.