ICorDebug (ICD) is a com-classic interface. In terms of COM threading models, ICorDebug is technically free-threaded (aka, should reside in the "neutral apartment"), which means that it manages its own threading. We go through great pains in the ICD to managed our own synchronization and takes internal locks accordingly. Also, calls to ICorDebug never block (pending CLR bugs and some interop-debugging cases*), which gives clients some additional flexibility. (See Chris Brumme's blog for details on the CLR and COM threading).
Unfortunately, ICD does not aggregate the free-threaded marshaler (this was on our radar bug got cut due to time constraints), which is bad because things like com-interop can't view it as free-threaded. ICD has a callback thread, so it can't be STA either. Thus COM-interop must import it as a set of MTA objects. This is significant because MDbg uses COM-interop to import it into managed code.
WinProcs have a single thread that owns the window and pumps messages, and thus UI controls are STA. The GUI thread really should be STA in order to use certain GUI controls, such as the common file dialogs.
However, STA and MTA objects don't play well on the same thread. So the MDbg Gui specifies the GUI thread is MTA so that the gui thread can directly call MDbg objects to populate the GUI controls. This greatly simplifies the programming model for letting GUI elements access MDbg.
But now the Gui thread can't call the file dialogs! It looks like post beta2 builds start enforcing this, and this is why the Mdbg GUI is currently broken in post Beta 2 builds. I didn't realize this constraint when I wrote about the threading model for the MDbg Gui.
Specifically, once the GUI thread tries to show a file dialog (by calling OpenFileDialog.ShowDialog) you get an exception like:
Current thread must set to single thread apartment (STA) mode before OLE calls can be made.
Ensure that your Main function has STAThreadAttribute marked on it.
If you go make the main thread STA, then everything deadlocks when MDbg tries to dispatch a callback on the callback thread.
It turns out there's not a good solution for the COM-interop case here without actually fixing the product.
1) Ideally, we'd let ICD be free-threaded in managed code. However, this requires that we make a product change to ICorDebug (in mscordbi) to aggregate the Free Threaded Marshaler.
2) Another idea is to make the managed ICorDebug wrappers STA.
2a) One way to do this is to have the callback thread immediately forward the callback to the UI thread and so the UI thread is effectively dispatching the callback. Thus only the UI thread is using the ICorDebug wrappers. This is my favorite solution, but it requires some plumbing changes within MDbg.
2b) Another way is to leave the callback plumbing as is, but have all calls to ICorDebug get marshaled back to the owning thread. I believe this requires we setup some additional plumbing and I'm personally not com-savy enough to know how to do this.
3) UI Purists say the UI thread should never call exterior functions (such as anything on MDbg or ICorDebug) because if those functions block, the UI will hang. That's technically correct, but it's significantly harder to program against. What should be a simple 3 line imperative function becomes literally a 300 line hassle with passing messages back and forth across threads.
I'd like to employ one of these solutions and fix the MDbg GUI. I'll certainly post back here when we have a solution.