Both Word and Excel OMs have APIs that allow closing documents programmatically. I suppose there are some scenarios when people would like to call ThisWorkbook.Close() from within their document customizations. This has some interesting effects when such call is made from within managed VSTO customizations.
First of all closing the document will also cause the current AppDomain be unloaded. This is the basic feature of VSTO loader to ensure managed code does not hold references to the document that has just been closed. Note that AppDomain unloading happens while there is code from this AppDomain on the managed stack. This is kind of an interesting situation but CLR folks have found a way to deal with it. They try to unwind the stack as aggressively as possible to the frame where first method from the unloading AppDomain has been entered. To accomplish that CLR throws System.Threading.ThreadAbortException. In the documentation you will find that when this exception is thrown you can run but you cannot hide: ThreadAbortException is a special exception that can be caught, but it will automatically be raised again at the end of the catch block. Chris Brumme explains the whole process in much more details so you might want to read his excelleny blog entry.
When ThreadAbortException reaches the base-most frame the exception turns into the DomainUnloaded exception and, in our case, the execution transitions into the native code. And now the fun begins.
First scenario: the document was closed from within an event handler for Excel or Word event such as Worksheet.SelectionChange. In this case CLR realizes the host application calls the managed code as a regular COM call and the failure should be communicated according to COM conventions. The convention is that failure code is returned. In this case it is COR_E_APPDOMAINUNLOADED. Excel/Word ignore the return value and execution continues normally.
Second scenario: the document was closed from within a WinForms event handler. For example a button on a modeless from was pressed and Button.Click event was raised. In this case the point of entry into the managed code is a windows procedure (WndProc) implemented in the depths of System.Windows.Forms assembly – all managed code. Such invocation is called “Reverse P/Invoke” as it is the reverse situation from when managed code calls native API through mechanism known as Platform Invoke. Contrary to the COM scenario CLR does not know about any convention to communicate errors to the native callers. So, it just throws an exception, particularly Windows SEH. There is another long but outstanding post on this from Chris Brumme. So, now this SEH is thrown right into Excel/Word. But these application are designed to pop up a Dr. Watson dialog (informing that the application has crashed and suggesting to send a report to Microsoft) if they see an unhandled and unrecognized exception come their way. And they do! So, if you were seeing that Document.Close causes host application to crash this is why.
Another variation of “unhandled exception in reverse P/Invoke” crash is when your are debugging Word/Excel and there is an unhandled exception thrown from Button.Click event handler. Normally WinForms would internally catch unhandled exceptions but debugging mode is special cased. When WinForms detect managed debugger is attached they would let the exception go through to benefit from the Visual Studio debugger behavior which always breaks on unhandled exceptions. But when WinForms are hosted inside native application this trick causes crashes. In order to force WinForms into always catching the exception just install an event handler for System.Windows.Forms.Application.ThreadException event.
Those crashes were a big bummer for VSTO. Starting with VSTO v2 Beta2 you would not (usually :)) see those crashes. What is the trick? We decided that we want to be in control. If managed WndProc misbehaves we are going to deal with it right away without letting the SEH propagate and destabilize the entire application. Our VSTO loader sub-classes every WinForms “managed” window and redirects the WndProc to our own native WndProc. The latter calls into original managed WndProc and if the SEH is thrown we just swallow it.
Some internal implementation details (that are not guaranteed to stay this way in the future but this is the way things are now). We sub-class the window inside WH_CALLWNDPROCRET windows hook after WM_CREATE message is processed by WinForms. We check whether an HWND belongs to our AppDomain by calling System.Windows.Forms.NativeWindow.FromHandle(hwnd) from inside the AppDomain. The original WndProc is stuffed into GWL_USERDATA slot in the window.