Adding Stepping Support to a Debug Engine
One feature that is clearly missing from the current debug engine sample is stepping. The reason it is missing is it only requires one AD7 method to initiate the step and one AD7 event when the step is complete. However, the actual implementation of stepping is very complex and requires support from the underlying platform to implement. Since the purpose of the sample is to supplement the existing documentation of AD7, I decided stepping was one thing that could be left out. Real world debug engines will require stepping support. So, the question becomes, how do you add it. First, let's talk a little bit about AD7's support for stepping, and then I'll give a high-level descriptoin of what is required to implement a stepping engine in x86.
When a user initiates a step in Visual Studio, the Debugger UI (and thus the SDM) make a call to the current IDebugProgram2's Step method. Step's signature looks like this:
HRESULT Step(
IDebugThread2* pThread,
STEPKIND sk,
STEPUNIT step
);
The pThread parameter is the thread to step. The STEPKIND parameter describes what type of step to do (step-in, step-over, step-out, step-backwards). I've never actually seen an engine that supports step-backwards, but the other three are pretty much required for any engine that supports stepping.
STEPUNIT describes what "unit" to step. This can be statement, line, or instruction. Statement is used for languages such as VB or C# that allow the user to step through sub-expressions in a statement (such as a for loop initializer and its condition). Line is for languages that don't support that (such as C++). Instruction is used for stepping in the disassembly window.
When the engine receives the call to step, it should do the required setup in the debuggee to make the step occur, and then continue the last stopping event. This will allow the debugee to run. Once the engine detects that the step has completed successfully, it should send an instance of IDebugStepCompleteEvent2 to the SDM. IDebugStepCompleteEvent2 is a stopping event. Like all stopping events, the debuggee should remain in break state until the UI makes a call to Continue or to Execute.
So, what hardware support is required on a native x86 machine to enable stepping? The debug engine will make use of the trap-flag and breakpoints. The trap-flag is used to execute a single-instruction in the debuggee and then stop in break-mode. It is bit 8 in EFlags (one of the x86 control registers). When the next instruction completes executing in the debuggee with the trap flag enabled, a single-step exception (code 0x80000004) is thrown to the debugger if one is present.
Breakpoints are used to avoid stopping on every instruction in cases that will be too expensive or for instructions that do no honor the trap-flag. For instance, when an instruction has the repeat prefix on it, a debug engine may choose to set a breakpoint at the next instruction after the repeated instruction and then continue. This is to avoid single-stepping over the same instruction repeatedly.
So, the most-basic stepping algorithm to perform a step on x86 looks like this:
1) Look at the source-line information in the symbol file. This can map the current instruction pointer to a line in source and tell the debugger how many instructions make up the source line.
2) Enable single stepping unless current instruction is one where the trap-flag cannot or should-not be used.
3) Continue the debuggee to allow it to run
4) Once the current instruction executes, receive the single-step exception. Check to see if the step is complete. If not, repeat from step 2 continuing the single-step exception handed.
5) Once the step completes, send an instance of IDebugStepComplete2.
Obviously, I'm missing a lot of details. For instance, how do you implement Step over if the stepping range contains a call instruction? How do you implement step-out? etc... Perhaps I'll get to those in a later post. Hopefully, this is enough to get some people started on the execution control portion of their debug engine.