Reverse P/Invoke and exception

P/Invoke is managed code executing a native function pointer. And Reverse P/Invoke is native code executing a managed delegate as a function pointer, mostly as a callback.

When the native code executes the managed delegate function pointer, the call really calls into a CLR thunk. The thunk will do an AppDomain transition to the correct AppDomain, then invoke the delegate. If the managed delegate throws an unhandled exception in the callback, the exception will be raised as cross context exception at the base of the AppDomain transition, and will likely cause a Dr.Watson error report.

If we don't want the managed exception to trigger Watson report, there are two things we could do.

1) have a top level try/catch block in the managed delegate and catch all exceptions. Yes we need some mechanism to convey the error information to the native code. We can either have the managed delegate return an HRESULT, or have it as one of the out parameter.

2) use ICLRRuntimeHost::ExecuteInAppDomain. CLR will catch any unhandled exception and convert it to an HRESULT to return to the caller.