Sucking the exception pointers out of a stack trace


Often, you find yourself staring at a stack trace for a caught exception and want to see the original exception.

ChildEBP RetAddr  Args to Child
030c21d0 76df3448 00000000 030c6138 76db6b0d ntdll!DbgBreakPoint
030c21dc 76db6b0d 030c2204 77b8d585 030c220c ole32!PeekMessageExceptionFilter+0x42
030c21e4 77b8d585 030c220c 00000000 030c220c ole32!CCliModalLoop::MyPeekMessage+0x41
030c220c 77f36992 030c25d0 030c6128 030c22e8 msvcrt!_except_handler3+0x61
030c2230 77f36964 030c25d0 030c6128 030c22e8 ntdll!ExecuteHandler2+0x26
030c22d8 77f36884 030c1000 030c22e8 00010007 ntdll!ExecuteHandler+0x24
030c25b8 77f6e0dd 030c25d0 00000000 00000000 ntdll!RtlRaiseException+0x3d
030c262c 77d3c239 77d4a4b6 77d3e2c5 030c3767 ntdll!RtlDeactivateActivationContextUnsafeFast+0x233
030c2630 77d4a4b6 77d3e2c5 030c3767 030c26a0 USER32!UserCallWinProcCheckWow+0x167
030c2634 77d3e2c5 030c3767 030c26a0 77d4a46f USER32!_NLG_Return2
030c265c 77d3e288 030c57b4 ffffffff 030c2688 USER32!__local_unwind2+0x70
030c2688 77f36992 030c26f8 030c57b4 030c27a4 USER32!_except_handler3+0xd5
030c26ac 77f36964 030c26f8 030c57b4 030c27a4 ntdll!ExecuteHandler2+0x26
030c2a74 77b8d36d 030c6128 77b8d36d 00000000 ntdll!ExecuteHandler+0x24
030c2a9c 77b8d59d 030c6128 030c2ac0 00000000 msvcrt!__global_unwind2+0x18
030c2ac0 77f36992 030c2ba4 030c6128 030c2bc0 msvcrt!_except_handler3+0x75
030c2ae4 77f36964 030c2ba4 030c6128 030c2bc0 ntdll!ExecuteHandler2+0x26
030c2b8c 77f36796 030c1000 030c2bc0 030c2ba4 ntdll!ExecuteHandler+0x24
030c2b8c 77b7aa54 030c1000 030c2bc0 030c2ba4 ntdll!KiUserExceptionDispatcher+0xe
030c3300 77b7b4dc 030c3324 7715b1b4 00000000 msvcrt!_woutput_l+0x18

(You too can get symbols for operating system binaries, either by using the symbol server to get the symbols on-demand or, if you have a gigabyte of disk space, you can download symbol packages to get them all at one go. Even if you go for the symbol package, you still need the symbol server, since it gets updated with symbols for binaries that have been updated since the most recent service pack.)

Here, we caught an exception in the PeekMessageExceptionFilter. What was the exception? Well, an exception filter receives a pointer to an EXCEPTION_POINTERS structure as its argument.

typedef struct _EXCEPTION_POINTERS {
    PEXCEPTION_RECORD ExceptionRecord;
    PCONTEXT ContextRecord;
} EXCEPTION_POINTERS, *PEXCEPTION_POINTERS;

Here, we see that the parameter to PeekMessageExceptionFilter is

030c21dc 76db6b0d 030c2204 77b8d585 030c220c ole32!PeekMessageExceptionFilter+0x42

0:0000> dd 030c2204 l2
030c2204 030c25d0 030c22e8
         -------- --------
         .exr     .cxr

The first value points to the exception record and the second points to the context record. You can view the exception by typing .exr 030c25d0 and view the context for the exception (i.e., what code was executing when the exception occurred) by typing .cxr 030c22e8. Those two values also appear as the first and (go figure) third parameters to ExecuteHandler2.

It so happens that doing the .exr on this particular exception record reports that the exception was c015000f which happens to be STATUS_SXS_EARLY_DEACTIVATION, and after setting the context to the exception context record, the stack goes

ChildEBP RetAddr
030c262c 77d3c239 77d4a4b6 77d3e2c5 030c3767 ntdll!RtlDeactivateActivationContextUnsafeFast+0x233
030c2630 77d4a4b6 77d3e2c5 030c3767 030c26a0 USER32!UserCallWinProcCheckWow+0x167
030c2634 77d3e2c5 030c3767 030c26a0 77d4a46f USER32!_NLG_Return2
030c265c 77d3e288 030c57b4 ffffffff 030c2688 USER32!__local_unwind2+0x70
030c2688 77f36992 030c26f8 030c57b4 030c27a4 USER32!_except_handler3+0xd5
030c26ac 77f36964 030c26f8 030c57b4 030c27a4 ntdll!ExecuteHandler2+0x26
030c2a74 77b8d36d 030c6128 77b8d36d 00000000 ntdll!ExecuteHandler+0x24
030c2a9c 77b8d59d 030c6128 030c2ac0 00000000 msvcrt!__global_unwind2+0x18
030c2ac0 77f36992 030c2ba4 030c6128 030c2bc0 msvcrt!_except_handler3+0x75
030c2ae4 77f36964 030c2ba4 030c6128 030c2bc0 ntdll!ExecuteHandler2+0x26
030c2b8c 77f36796 030c1000 030c2bc0 030c2ba4 ntdll!ExecuteHandler+0x24
030c2b8c 77b7aa54 030c1000 030c2bc0 030c2ba4 ntdll!KiUserExceptionDispatcher+0xe
030c3300 77b7b4dc 030c3324 7715b1b4 00000000 msvcrt!_woutput_l+0x18

Wow, we took an exception while trying to handle another exception! (It so happens this was easy to spot in the original stack trace, but in the general case, the next outer exception may require digging.)

Repeat the exercise with this next exception:

0:000> .exr 030c2ba4
ExceptionAddress: 77b7aa54 (msvcrt!_woutput_l+0x00000018)
   ExceptionCode: c00000fd (Stack overflow)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000001
   Parameter[1]: 030c2e88

0:000> .cxr 030c2bc0
eax=030c33b0 ebx=00000000 ecx=0000005c edx=00000000 esi=030c33c4 edi=030c33c4
eip=77b7aa54 esp=030c2e8c ebp=030c3300 iopl=0         nv up ei pl nz na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010202
msvcrt!_woutput_l+0x18:
001b:77b7aa54 53          push    ebx

Aha, so the SXS exception was triggered by a stack overflow. At this new context, you can use the "k" command to see how we got into this state.

It so happens that this particular bug was, as predicted, a stack overflow caused by unintended recursion when a call to an off-thread COM object forced the calling thread to wait for the reply, during which time a new request came in. The precise details of the problem aren't important; the goal today was to show how you can suck the exception pointers out of the stack to see what Win32 exception was the source of the problem.

Comments (20)
  1. nikos says:

    this symbol server, does it work with the visual studio debugger or only windbg? Where can we find info on the supported debuggers?

    thanks

  2. richard says:

    This is why I love your blog because I pick up so many useful tidbits of information.

    What would be better is a nice book or other document that spells out all this information for us. Yes, I am sure it is all available, deeply buried in the mountain of documentation MS makes available, but somehow it would be nicer it it was more easily accessible.

    Any books in your future, perhaps an "A Real Insider’s Guide to Windows Development and Debugging"

  3. brianw says:

    I second richard’s comment regarding how nice it would be if information such as this were collected in a more accessible format.

    I would definitely go for a book by Raymond!

  4. Skywing says:

    You can use the `.ecxr’ command to automate this if you are working with a dump file that had the EXCEPTION_POINTERS* passed to MiniDumpWriteDump.

  5. Carl F. says:

    nikos:  The symbol server works fine with Visual Studio.  I use it that way all the time.

  6. jmstall says:

    One technique that may be useful is actually searching the stack for the context flags (1003f on x86). It’s quick, dirty, and doesn’t require symbols, and works 99% of the time on x86.

    > s -d esp L1000 1003f

    0535ef48  0001003f 00000000 00000000 00000000  ?……………

    > .cxr 0535ef48

         

    More verbose details are here:

    http://blogs.msdn.com/jmstall/archive/2005/01/18/355697.aspx

  7. Tom says:

    @ Nikos and @Carl F.

    You might have to check what version of Visual Studio you’re using.  The symbol server functionality may only be available in Professional and Enterprise.  It may not be available in the standalone Visual C++ (usually retails for $99USD)

  8. mikeb says:

    To setup Visual Studio to use the symbol server, see Microsoft’s KB article 319037 (http://support.microsoft.com/default.aspx?scid=kb;en-us;319037)

  9. Mike Dimmick says:

    As far as I’m aware, any edition of Visual Studio from 2002 onwards works with the symbol server. You might need to replace dbghelp.dll with the version from the Debugging Tools for Windows, which you can download from http://www.microsoft.com/whdc/DevTools/Debugging/default.mspx. Raymond doesn’t explicitly say so but I assume he’s using WinDbg or one of the other debuggers from this package.

    A good book on debugging is ‘Debugging Applications for Microsoft .NET and Microsoft Windows’ by John Robbins (MS Press), one of the few good books on Win32 development that MS Press have not let fall out of print. It would help Raymond’s cause a huge amount if they would reprint ‘Programming Applications for Microsoft Windows, Fourth Edition’ by Jeffrey Richter.

  10. Myria says:

    Releasing PDB files is like releasing the source code for an expert reverse engineer.

    Still, go ahead and keep releasing them, I love it. =)

    Melissa

  11. silkio says:

    Brian and Richard: Search the blog for a discussion of why Raymond will not write a book.

  12. KJK::Hyperion says:

    Visual Studio 2005 Express does support Symbol Server. I use it nearly every day

  13. Mike Hearn says:

    Gotta love that unexpected re-entrancy. Yay for COM!  (and CORBA…)

  14. richard says:

    Raymond wrote back in 2003 why he wouldn’t write a book:

    http://blogs.msdn.com/oldnewthing/archive/2003/10/08/55204.aspx

    ———

    I’m doing this instead of writing a book

    Some commenters mentioned that I should write a book. It turns out that writing a book is hard.

    A few years ago, MS Press actually approached me about writing a book for them. But I declined because the fashion for technical books is to take maybe fifty pages of information and pad it to a 700-page book, and I can’t write that way. None of my topics would ever make it to a 100-page chapter. They’re just little page-and-a-half vignettes. And it’s not like the world needs yet another book on Win32 programming.

    So I’ll just continue to babble here. It’s easier.

    ———

    I agree with him on not wanting a 1200+ page tome on my desk. I hate those fat books. Why can’t the publishers understand that we prefer thinner, straight to the point books instead of bloated monstrosities that are filled with padding but not information.

    I disagree on the world not needing another Win32 programming book (maybe Vista will make Win32 obsolete), but I still think a useful thin book is better than a fat bloated book any day of the week.

    Wads of words do not make me more productive.

  15. Jamie Gordon says:

    WinDbg rules! But now I know how to use it, why doesn’t Windows XP let you access the crash dump file, that it creates when it catches unhandled exceptions, so you can analyze those problems? It’s locked even against read only access and gets deleted as soon as you close the exception dialog.

    Come back Dr Watson, I miss you, at least you let me have my minidumps!

    It’s fine being able to debug an exception when it happens easily and reliably in front of you and you can fire up a debugger, armed with your symbols. But when it happens in a piece of software that only exhibits the problem very occasionally, it’s very frustrating.

    I am forced to make the minidump read only before the Windows unhandled exception filter deletes it.

    And that is easier said then done if, for example, the exception occurs in an Explorer context menu and the user who has the problem is not very computer literate. "Now I want you to open a command prompt, CD to your temp folder and attrib the minidump file" … "Wha?!"

    Or I have to add my own exception handler into my programs which is slightly overkill in my opinion for most of the things I end up writing.

    Still, it’s very annoying that Windows creates the file you need to analyse the problems but makes it so difficult to use it.

    Thanks for the tip Raymond.

  16. Mark Steward says:

    Jamie – I’ve never found the dump files to be locked.  However, you can get around it by running

    "%ProgramFiles%Debugging Tools for Windowswindbg.exe" -I

    and then changing the Auto value in HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersionAeDebug to 2.  When it dumps, it’ll also give you a button to allow JIT debugging.

    Myria brings up an interesting point about the role of reverse engineering in Windows, but I’ll leave it out here, because I don’t suppose anyone has the answer.

    Mark

  17. Mark Steward says:

    Whoops – sorry Jamie, didn’t realise you were distributing your software.  In that case, I guess you could sign up for error reports (which like everything, requires a damn VeriSign ID at $499) ;-)

  18. Simon Cooke says:

    Jamie – it’s a moot thing anything. With VS2005 they removed the ability to handle unhandled exceptions yourself. Which sucks, because I architected and wrote a system which has evolved to be used across the whole of Midway Games to do automated post mortem debugging of crashes across our toolset and games, and it won’t work when we move to VS2005.

    Unless we hack the CRT and provide our own version, that is.

    It’s worth the effort. Should take you less than a week to get something very basic together (you don’t need a flashy UI, or network storage of the dumps, or human readable stack traces or anything like that). Just try to create the minidump from an EXE outside of the context of the one that’s crashing (by spawning a child process when you get the exception) otherwise you’ll get flaky results.

    Si

  19. In-depth Articles- Matt Peitrek on the internals of SEH . Matt Pietrek on Vectored Exception Handling

  20. Look for the KiDispatchException.

Comments are closed.