Implementing ICorDebug to reuse existing debuggers?

I usually talk about people writing debuggers to consume the ICorDebug API and debug CLR applications.

Nathanael Presson asked me the exact inverse question:

… my concern is not to write a debugger for managed code, but to make my own self cooked .net execution engine (I work in video game industry) to be debugable by say, c# express. I don’t ask any sample code, I just want your opinion, do you think its even possible ?. For the moment I use my own little debugger wrote in c#, via auto generated rpc's to communicate to the ee (can be on an other pc, an Xbox, etc...

 

In other words, can I write my own implementation of ICorDebug to get existing managed debuggers to debug my own virtual machine (instead of the CLR) . Let me look at if from a purely technical level:

 

In theory, it should work:

- Existing managed debuggers won’t know what’s on the other side of the interface and would just use it naively.

- The ICorDebug API is also abstracted at a good level for this. It was designed to allow us to wildly change the CLR’s implementation. For example, we could switch the CLR from being jitted to being interpretted and ICorDebug would still work. It has high level constructs like ICorDebugBreakpoint instead of low-level things like SEH exceptions which then force the debugger to build breakpoints on top of it. (Unfortunately, this same quality also makes interop-debugging very difficult, as I lament here)

- This is actually what the .NET compact frameworks did, so it can be done.

 

In practice, there are some hurdles to overcome:

- ICorDebug is a large API. V1.1 was 250 methods, and v2.0 has 300 methods. Most of these are pay-as-you-go. For example, you don’t need to implement ICorDebugEval if you don’t expose func-eval.

- The API is not very well specced. That will make it difficult to get the proper behavior.

- ICorDebug has callbacks that the debugger implements. You’ll need to understand when to fire these callbacks.

- You’ll need a pretty decent understanding of ICorDebug. Some random resources links are here.

- You’ll need to get the debugger to create your new implementation of ICorDebug. The addition of CreateDebuggerInterfaceFromVersion in v2.0 make this easier because you can just trick whidbey debuggers into thinking your new EE is just another version (eg, “vMyPrivateImpl”) of the CLR. Put your implementation in mscordbi.dll under %windir%\microsoft.net\framework\vMyPrivateImpl, and then put  a config file on your exe pointing it at that version.

- You’ll also have to implement some metadata APIs in addition to just ICorDebug. ICorDebug just provides metadata tokens, and then the debugger uses the metadata APIs to resolve those tokens to names. ICorDebug provides the debugger a metadata object via ICorDebugModule::GetMetaDataInterface, so you'll have a chance to provide the debugger your fake metadata implementation too.

 

 

For those adventurous folks who want to try this, I strongly advise you to download the MDbg sample (or Cordbg sample from v1.1 sdk) and use that to test it. If your ICD implementation won’t work with MDbg, it probably won’t work with VS either. And since you have all the source for MDbg, you can understand what’s going on when things go wrong.

 

If anybody ends up tackling this, I’d be very interested in knowing how it turns out.