IFEO and Managed-debugging

Be careful  of Image File Execution Options (IFEO) with managed debugging - it won't work like you expect.  

IFEO lets you set some registry goo such that when you launch a target app (specified by a registry key name), a debugger (specified by a string named "debugger" under that registry key) is executed instead. The debugger then launches the target app under its control.  (For more about IFEO: see MSDN for details, GreggM talks about debugger details. Junfeng talks about other IFEO tips; MSDN has some tips here; and Raymond Chen has more.)

MSDN warns this only works for native and interop-debugging. It does not work for managed-only debugging. Here's why...

First, look at a step-by-step walkthrough:

  1. You setup a registry key for "MyApp.exe" with a string "debugger"="MyDebugger.exe".
  2. You attempt to run MyApp.exe via Explorer.
  3. Explorer makes a call to CreateProcess("MyApp.exe", flags=NotDebugging)
  4. IFEO intercepts that CreateProcess call because flags=NotDebugging. It launches a debugger by concatenating the value in the "debugger" registry string with the parameters from CreateProcess. So it will actually launch: "MyDebugger.exe MyApp.exe" instead of launching MyApp.exe.  
  5. It is expected that MyDebugger.exe will then use the command line args to launch a debuggee. Specifically, it will call CreateProcess("MyApp.exe", flags=Debugging).
  6. Since that CreateProcess call specifies it is debugging, the call is not intercepted by IFEO and MyApp.exe is created under the debugger as normal.

GreggM discusses a lot of interesting ramifications of this.

So what's the problem for managed-debugging?
Managed-debugging is not built on native-debugging. Managed-debugging has its own debugging channel that built on its own interprocess-communication protocol, which is completely separate from the OS facilities used by native-debugging. That means that launching the debuggee under the managed-debugger will do CreateProcess("MyApp,exe", flags=NotDebugging).  This introduces infinite recursion with IFEO, because that will get intercepted by IFEO and relaunch the debugger. In other words, we'd loop forever between step 4 and step 5.

Interop-debugging is built on OS-facilities, and so looks like a native-debugger to the OS. This is why MSDN tells you to use interop-debugging with IFEO.  Another option may be to disable IFEO after the debugger is launched, but before it lauches the debuggee.