Part 1 is here.
At the end of Part 1, I’ve decided it’s time to try this COM Message Filter thing out. Implementing the IMessageFilter interface and registering the message filter turns out to be not hard at all, because really, in our case the message filter doesn’t need to do anything complicated, it just needs to return RETRY a lot.
Yes, not hard at all, except… suddenly my application starts crashing during random calls to DTE. Arg! What has gone wrong?
The exception is Access Violation which serves as a clue. The program counter is pointing at some crazy memory address. It seems like a good guess that something has gone wrong with the interop between managed and native code. But why?
Backtrack 24 hours to yesterday, when I decided it would be either easy or educational to try to write my own C# interface definition for IMessageFilter from scratch based on the C++ interface docs.
So here’s, for example, the signature for IMessageFilter::RetryRejectedCall
DWORD RetryRejectedCall( [in] HTASK htaskCallee, [in] DWORD dwTickCount, [in] DWORD dwRejectType );
What could possibly be wrong with this managed equivalent?
int RetryRejectedCall(IntPtr hTaskCallee, uint dwTickCount, SERVERCALL rejectType);
Yes, I had decided to make it fancy, and defined a SERVERCALL enumeration…
internal enum SERVERCALL : uint
IsHandled = 0,
Rejected = 1,
RetryLater = 2,
And no, logically that should not be the cause of the mystery problem…
Internet to the rescue! It turns out that if you use tlbimp.exe which is the tool for automating what I had just done manually (defining managed interfaces for interoperating with COM), it will generate something a little bit different from what I wrote.
[PreserveSig, MethodImpl(MethodImplOptions.InternalCall, MethodCodeType = MethodCodeType.Runtime)]
int RetryRejectedCall(IntPtr hTaskCallee, uint dwTickCount, uint rejectType);
It puts a whole bunch of attributes there. Apparently the only important one is PreserveSigAttribute, the rest are JIT compiler optimizations or something. In trying to understand this, the first place I looked was PreserveSigAttribute docs.
‘By default, the Tlbexp.exe (Type Library Exporter) ensures that a call that returns an HRESULT of S_OK is transformed such that the [out, retval] parameter is used as the function return value. The S_OK HRESULT is discarded. For HRESULTs other than S_OK, the runtime throws an exception and discards the [out, retval] parameter. When you apply the PreserveSigAttribute to a managed method signature, the managed and unmanaged signatures of the attributed method are identical.’
How cool! If you have a method signature with an out param, and it returns HRESULT, you can use PreserveSig, or actually NOT use PreserveSig, and you can call it as if it returned the out param directly. But what happens if you have an HRESULT return value but no out param? And why is not preserving signatures the default?
It turns out you can find a much fuller description of what PreserveSigAttribute by looking up DllImportAttribute.PreserveSig field, which I suppose came first or something, and I suppose does the same thing.
And in these docs its fairly clear that when there isn’t any out param, then the return type of the method signature gets converted to void. Problem solved. I have just rediscovered what a thousand other people discovered about PreserveSig before me. Implementing your own COM interfaces turns out to be more educational than easy, and I learned what PreserveSig can do. I wonder if I’ll get to use it again some day …