Real life usage of loading a driver as a dump file

Today I got a callstack via email (from the NTDEV list), but no dump file. I needed to determine if the bugcheck was due to a KMDF or a USB core bug. Since I had no dump file, I had to work purely on what the message contained. The callstack did have symbols and offsets though, so I had a decent place to start. All in all, it took me less then 10 minutes from start to finish to fully debug the problem (which is funny because it took me more time to write this entry up 😉 ).

The callstack looked like this:

eax=00000107 ebx=82d60104 ecx=82e69f5c edx=82e69f5c esi=82e69ce8 edi=8293c3e8
eip=f7813371 esp=f7b36a94 ebp=f7b36aac iopl=0         nv up ei pl nz ac pe cy
cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000 efl=00010213

f7813371 8908             mov     [eax],ecx 

f7b36aac f78134b2 82962cc0 00000100 8293c3e8 usbhub!USBH_SetPowerD0+0xd3
f7b36ac8 f7813727 8293c3e8 82962cc0 82962cc0 usbhub!USBH_PdoSetPower+0x80
f7b36ae8 f780b97b 82962d78 82962cc0 00000002 usbhub!USBH_PdoPower+0x201
f7b36b08 f78091d8 8293c3e8 82962cc0 f7b36b3c usbhub!USBH_PdoDispatch+0x83
f7b36b18 804e37f7 8293c330 82962cc0 82962d78 usbhub!USBH_HubDispatch+0x48
f7b36b5c ed4da316 8293c330 8293c518 82962d9c nt!PoCallDriver+0x195
f7b36b7c ed4da3b9 f7b36bb8 8293ca00 ed4ec730 Wdf01000!FxPkgFdo::RaiseDevicePower+0x50

Not much to go on. What I really wanted to do was map usbhub!USBH_SetPowerD0+0xd3 to a line in the source file, but how could I do that? Then it occurred to me that I could load usbhub.sys as a dump file, unassemble at that offset and let the debugger do all the hard work for me. So, here is what I did:

  1. Copied the XP SP2 (the email included the OS version, otherwise I would have had to iterate across multiple releases) usbhub.sys off the build server.
  2. Loaded the image as a dump file.
    C:\debuggers\windbg.exe -z usbhub.sys
  3. Ran !symfix to fix up the symbols
  4. Unassembled usbhub!USBH_SetPowerD0+0xd3 and make sure that the instructions at that offset in my debugger matched the callstack.
    :000> u USBH_SetPowerD0+0xd3
    usbhub!USBH_SetPowerD0+0xd3 [[...] @ 596]:
    0001c371 8908             mov     [eax],ecx

    Note that not only did I confirm that the source was correct, the debugger also gave me a source file (name removed) and line number!

  5. Loaded up the source file and goto line 596 which had this pseudo code
    InsertTailList(&fdoExt->ListHead, &PDO_EXT(pdo)->link);

    PDO_EXT() returns the device extension if the pdo is != NULL while -1 if pdo == NULL. From the email, I had the values of the registers and eax was 0x107, which looks suspiciously like -1 + 0x108. To verify this last step, I did the following:

    Registers from the email
    eax=00000107 ebx=82d60104 ecx=82e69f5c edx=82e69f5c esi=82e69ce8 edi=8293c3e8
    eip=f7813371 esp=f7b36a94 ebp=f7b36aac iopl=0         nv up ei pl nz ac pe cy
    cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000 efl=00010213
    0:000> dt [PDO DEVICE EXTENSION TYPE] link
       +0x108 link: _LIST_ENTRY
  6. So I have confirmed that this is the line that was executed and that -1 was returned, but I really wanted to make sure. To verify that eax was what i thought it was, I disassembled right before 0xD3 to see how eax was initialized.
    0:000> u usbhub!USBH_SetPowerD0+0xc0 usbhub!USBH_SetPowerD0+0xd3+1
    usbhub!USBH_SetPowerD0+0xc0 [[...] @ 596]:
    0001c35e e8834dffff       call    usbhub!PDO_EXT (000110e6)
    0001c363 8d8e74020000     lea     ecx,[esi+0x274]
    0001c369 8b5104           mov     edx,[ecx+0x4]
    0001c36c 0508010000       add     eax,0x108
    0001c371 8908             mov     [eax],ecx

    eax is the value retuned from the call to usbhub!PDO_EXT whic we then add 0x108 to 2 instructions later. This confirms my theory. To put the final nail in the coffin, I wanted to see the offset of fdoExt->ListHead and see if the initialization of ecx right after the call was the FDO list head.

    0:000> dt [FDO DEVICE EXTENSION TYPE] ListHead
       +0x274 ListHead : _LIST_ENTRY

I am yet again surprised by the utility of windbg. It really helped me identify and quantify a bug with very little information to go by. Certainly not all debugging sessions are this easy, but it very gratifying when they are this easy and quick to diagnose.

Comments (6)

  1. diegocg says:

    This is pretty much the way lots (most?) of the linux bugs with a kernel backtrace are "debugged" remotely: take the backtrace, look up that info inside the source code file, try to figure out where the bug is (the official linux kernel does not have a "debugger" at all, for religious reasons)

  2. doronh says:

    Typically we have mini dumps which have 64K of data & callstack in them which is sent when you agree to send the info to MSFT.  The debug sessions rarely happen over email though w/only the stack and no other info.  Call me spoiled, but the minidumps are a great offline utility.

    I know why linux has no default kernel debugger, never really agreed with it though.

  3. Alex Ionescu says:

    Nice writeup… I wouldn’t have thought of this myself… doesn’t Linux use gdb as a debugger? Not sure if it’s "official" though.

    Also, I know you were just being careful, but using public PDBs and dia2dump on my Win2K3 symbols, I got this:

    d:srvrtmdriverswdmusbdriverhub.pnphubobji386hubpwr.obj : [00029244], len = 00000559 _USBH_SetPowerD0@8

    So the name of the source file is hubpwr.c :) Just wanted to point out the number of information that is still there in "stripped" symbols.

  4. Alexander Lyakas says:

    What reading would you recommend to get basic understanding of the assembly code?

  5. rakibul2 says:

    Hello, I like your blog. This is a great site and I wanted to post a comment to let you know,

    provides crappy gifts wrapping and delivery services from coast to coast everyday.

    Pounds of poop that are beautifully wrapped and sent to your friend with a sense of humor,

    and we guarantee the delivery to remain 100% anonymous. Our services are strictly for

    <a href="">gags pranks</a> only.