IFEO and Managed-debugging

Be careful  of Image File Execution Options (IFEO) with managed debugging – it won’t work like you expect.  

IFEO lets you set some registry goo such that when you launch a target app (specified by a registry key name), a debugger (specified by a string named “debugger” under that registry key) is executed instead. The debugger then launches the target app under its control.  (For more about IFEO: see MSDN for details, GreggM talks about debugger details. Junfeng talks about other IFEO tips; MSDN has some tips here; and Raymond Chen has more.)

MSDN warns this only works for native and interop-debugging. It does not work for managed-only debugging. Here’s why…

First, look at a step-by-step walkthrough:

  1. You setup a registry key for “MyApp.exe” with a string “debugger”=”MyDebugger.exe”.

  2. You attempt to run MyApp.exe via Explorer.

  3. Explorer makes a call to CreateProcess(“MyApp.exe”, flags=NotDebugging)

  4. IFEO intercepts that CreateProcess call because flags=NotDebugging. It launches a debugger by concatenating the value in the “debugger” registry string with the parameters from CreateProcess. So it will actually launch: “MyDebugger.exe MyApp.exe” instead of launching MyApp.exe.  

  5. It is expected that MyDebugger.exe will then use the command line args to launch a debuggee. Specifically, it will call CreateProcess(“MyApp.exe”, flags=Debugging).

  6. Since that CreateProcess call specifies it is debugging, the call is not intercepted by IFEO and MyApp.exe is created under the debugger as normal.

GreggM discusses a lot of interesting ramifications of this.

So what’s the problem for managed-debugging?
Managed-debugging is not built on native-debugging. Managed-debugging has its own debugging channel that built on its own interprocess-communication protocol, which is completely separate from the OS facilities used by native-debugging. That means that launching the debuggee under the managed-debugger will do CreateProcess(“MyApp,exe”, flags=NotDebugging).  This introduces infinite recursion with IFEO, because that will get intercepted by IFEO and relaunch the debugger. In other words, we’d loop forever between step 4 and step 5.

Interop-debugging is built on OS-facilities, and so looks like a native-debugger to the OS. This is why MSDN tells you to use interop-debugging with IFEO.  Another option may be to disable IFEO after the debugger is launched, but before it lauches the debuggee.

Comments (7)

  1. You say to be careful with "managed debugging".  So, if I want to debug managed code using WinDbg/SOS, will it work as expected?

  2. jmstall says:

    Yes, windbg/sos is native-debugging and so everything should work as intended.

    I was focusing more on Visual Studio and ICorDebug-based debugging.

  3. Adrian Eidelman says:

    So isn´t it possible to launch an exe file and debug it with Visual Studio debugger from the beginning as you can do with others debuggers ?

  4. jmstall says:

    Adrian –

    1. IFEO works great for native + interop-debugging; this is just an issue for managed-only debugging.

    2. This has nothign to do with VS. Any other managed debugger (eg, MDbg) would have the same problem.

  5. Tiho's blog says:

    I’m putting this here so that I don’t forget where to look for it later when I need it as it has…