Why debugging breaks in optimized (production) builds.

You may have noticed that debugging optimized builds (eg, what you commonly get when you attach to a end-user instance of your app that wasn’t launched from your debugging environment) is usually a degraded experience.

At an architectural level, there’s a fundamental tension between being debuggable and optimizing. The debugger generally requires the code to behave to a certain contract established between the compiler and the debugger (and recorded in the PDB). That contract mainly requires the code to maintain a close relationship to the original source code. The Optimizer makes aggressive transforms to the code for the sole purpose of making it execute more efficiently, with no regards to the original source code. 

Here are some of the things that break and why. This is not a complete list. This is general, so isn’t managed or native code specific:

  1. Callstack frames may be missing.  This is often Inlining: (This is one reason that you shouldn’t programmaticaly rely on the results on System.Diagnostics.StackTrace). The CLR’s inliner in Whidbey is pretty wimpy, so this is actually less of an issue for managed code compared to native code.
  2. You callstack may show you functions that you’re not actually executing. Code folding. If function_1() and function_2() both compile to the same code, the compiler/linker could share that code and just use 1 copy. So calls to function_2() would appear to have called function_1().  But the code all executes the same. The CLR does code-folding for generics.
  3. Local variables may be missing: Locals may get optimized out. The local may not be used, or maybe the compiler can store it in a more efficient fashion than the current PDB contract can describe to a debugger.
  4. Locals may be reported, but have the wrong value. This is bad. This often happens when the debugger doesn’t realize the code has breached the PDB contract. IMO, I would rather a debugger just tell you it can’t find a value, or tell you that it’s guessing.
  5. You may not be able to set breakpoints on certain lines. The lines may be optimized out. This may be code folding.
  6. Stepping through a function may get random: The optimizer may rearrange the ordering of basic blocks. This could be to eliminate jump instructions or to rearrange the code for better hot/cold locality.
  7. Breakpoints you do set may not be hit even when you think the code executes.  This is another manifestation of the previous issues. In this case, the optimizer has changed the wrong of source to code executed, without properly informing the debugger (perhaps the existing PDB contract can’t describe the optimizer’s actual mapping).

In theory, the you can compensate for many of these problems by making the PDB contracts richer so that the optimizer can describe to the debugger what it’s actually doing.

Comments (7)

  1. Nice post.

    Many of the reasons that you cite have convinced me that in the long run it is better to debug optimized (production) builds at the disassembly level.  I would also suggest that much of the disorientation arises in those whose debugging experience is almost solely at the source code level.  Finally, the occasions when you think you are using the correct PDB’s, but in fact they do not match the binary are almost worse than having no PDB’s in the first place — you are lulled into a false sense of security.

  2. jmstall says:

    Hi Russ. I guess my posts are finally starting to get technical enough again 🙂

    Re using wrong PDBs: yeah, that’s disasterous. Fortunately there’s pretty good matching support for a debugger to use (asame with source files).

  3. GP says:

    What about the /debug:pdbonly /optimize csc.exe switches (Default "Release" Configuration von VS 2005 for C# Projects)? How deals the C# Compiler with the above issues when building the Module and the Symbol-File?

    When a Application is lauched from a Debugger it may be possible to disable Optimization that would be made by the JIT-Compiler.

    But what happens when a Debugger is attached to an already running Application? Methods may be already JITted (and optimized) and the CLR may be in a (internal) state that makes it impossible to start all over without loosing "public" state. So the Symbols will not match in some cases (maybe inlined functions). Or am I wrong?

    This would be a very interesting Subject for another Blog-Entry: "/debug:pdbonly and /debug:full Symbols compared"

  4. Don’t have your non-debugger app use the debugging services just to get some cool functionality. The

  5. jmstall says:

    GP – that’s some good blog fodder. Stay tuned…