Advanced Debugging Part Two – TracePoint Macros that continue execution

TracePoints are a new feature in Visual Studio 2005. One of the things it allows you to do is to cause traces appear in the output window w/o adding code to your app. This is an incredibly useful feature and has been discussed in various blogs.

The other part of this feature is the ability to run VS Macros as part of a trace point. I think that when the VS2005 guys put in this feature, the main functionality they were interested in adding was the ability to trace more elaborate stuff to the output window. This, again, is a very useful feature. However.. Another very useful feature would be the ability to actually decide in a macro whether or not you want to actually cause the break. This is an extended version of a Conditional BreakPoint where you actually want to break only if something relatively complex happens.

As an example, say I have code that attempts to open a file for writing, and lets say that I want the ability to break only if the file is in use. How would you do that? Conditional Breakpoints will only take you so far – you can’t write the elaborate code needed for trying to open a file and seeing if it’s in use. So, hot shot, what do you do?

When looking at the trace-point functionality, you may smile to yourself and say “What’s the big problem? I will just write a macro that checks if the file is in use and tell the debugger to break if it is!” After saying that, you would smugly bang on your keyboard and may come up with a macro similar to this one:

    Sub BreakIfFileIsInUse()

        If Not (File.Exists("c:\file.ext")) Then Exit Sub

        Dim f As System.IO.FileStream

        Try

            f = New FileStream("c:\file.ext", FileMode.Open)

        Catch ex As Exception

            ' An exception happened - this means we want to cause a break.

            DTE.Debugger.Break()

        Finally

            If Not (f Is Nothing) Then

                f.Close()

            End If

        End Try

    End Sub

You are then going to place a tracepoint in the function and set it up so that it runs the BreakIfFileIsInUse() macro.

Well. That’s close, but it will not get you a cigar. When the call to DTE.Debugger.Break() will occur, VS2005 will show you an error saying:

A macro called a debugger action which is not allowed while responding to an event or while being run because a breakpoint was hit.

Apparently, this portion of the debugger code is not reentrant. I asked around in our internal DLs and the answer I got was that there’s no workaround and that they may consider putting it in Orcas. When I heard this, it made me into a very Sad Panda. It blocked too many cool debugging schemes. So I tried to find a workaround. And a partial workaround I did find.

Instead of causing a break, you can cause a continuation of the run – but in a round-about way. So in the aforementioned example, we will uncheck the Continue Execution checkbox and modify our macro to execute the workaround:

Sub BreakIfFileIsInUse()

      If Not (File.Exists("c:\file.ext")) Then Exit Sub

        Dim f As System.IO.FileStream

        Try

            f = New FileStream("c:\file.ext", FileMode.Open)

        Catch ex As Exception

            ' An exception happened - this means we do nothing

            ' (Since our tracepoint does not continue execution)

        Finally

            If Not (f Is Nothing) Then

                f.Close()

            End If

            ' No exception happened - make sure we continue execution

DeferredDebuggerContinue()

End Try

End Sub

The way we continue execution is to circumvent the reentrancy check that VS2005 does when you try to change execution mode – by scheduling a thread that will do it after the current debugger call is done:

Sub DeferredDebuggerContinue()

System.Threading.ThreadPool.QueueUserWorkItem(New System.Threading.WaitCallback(AddressOf DebuggerContinue))

End Sub

Sub DebuggerContinue(ByVal O As Object)

DTE.Debugger.Go()

End Sub

That’s it! A bit of a headache, but it works.