External Process Debugging in VSTA

VSTA was faced with some interesting problems when it came to the add-in developer’s debugging experience. VSTA is built on Visual Studio 2005 and .Net Framework 2.0. The debugging model in these products is a cycle of Build-Start-Debug and eventually Stop. Each time the developer wants to debug the application, the IDE builds output if needed, launches a new instance of it under the debugger and allows the user to take control with the debugger. When debugging ends via the user hitting Stop in the IDE, the debugger terminates the application forcefully. No finalizers run, no cleanup code executes. The application simply ceases to be. This model is supported by VSTA, but many host applications are not designed to be terminated at arbitrary execution points. The host app might be in the middle of writing out to a file, or performing a normally atomic operation. Terminating the app at these locations may result in data corruption and / or very angry add-in developers. Luckily, VSTA has a solution we call External Process Debugging.  There are two External Debugging scenarios in VSTA. Internally, we call these non-destructive debugging (NDD) and Alt-F11 (named after the shortcut key for launching the VBA IDE in Office).

With External Process Debugging, the add-in being debugged is executed in a process separate from the host application. It is almost entirely transparent to the add-in that it is not running within the host’s process. Furthermore, the work required for the host to support this model is fairly small and transparent.  When the add-in developer wants to debug, a new instance of the external process is created. This debugging process either launches a new instance of the host application and the two are connected (NDD), or it attaches to an already existing instance of the host (Alt-F11). The add-in is then loaded in the external process and objects from the object model in the host process. When debugging ends, the IDE forcefully terminates the external process. The host application receives a notification that debugging has ended. In response, it cleans up its state and can either gracefully exit (NDD), or reload the in-process (Alt-F11).

Why two models of external execution? Because some hosts want to provide add-in developers with a more integrated experience similar to that of VBA which their users are accustomed to. Alt-F11 is a much more interactive developer experience. The host application is created first. While the user is working on the host application, he/she decides a macro needs to be written or recorded. The host application opens an instance of the IDE and automates it externally. When the add-in developer decides its time to debug, they start debugging in the IDE and it appears that the addin runs in-process. When debugging ends, the host reloads the add-in in-process.

In the NDD scenario, the user model is very different than that of VBA. The user starts developing in the VSTA IDE. There is not interactive experience between the add-in developer and the host application during development. Only when a debugging session begins, is an instance of the host created.

Why an external process? The VSTA team’s first experience with this problem was in Visual Studio Tools for Office which is built on VSTA technology. In VSTO, there is a communication channel between the debugger and the host application running the add-in When debugging stops, VSTO is able to notify the app (in this case Word or Excel) and have it shutdown gracefully, even if the user has the application suspended at a breakpoint or stuck in an infinite loop. This technology was very complex, hard to implement and would only work with a few types of host applications. This led the VSTA team to search for a more general solution and this led to the external process.

Jackson Davis,

Comments (2)

  1. Guy Godin says:

    Would it be possible to do "In Process Debugging" with VSTA? I’d like the debugger to debug my application’s process directly when the user hits F5 in the VSTA IDE.

  2. I have a couple of questions that I hope you could answer. 1) It seems that any time the VSTA documentation