Signs that the symbols in your stack trace are wrong


One of the things programmers send to each other when they are trying to collaborate on a debugging problem is stack traces. Usually something along the lines of "My program does X, then Y, then Z, and then it crashes. Here is a stack trace. Can you tell me what's wrong?"

It helps if you at least glance at the stack trace before you send it, because there are often signs that the stack trace you're about to send is completely useless because the symbols are wrong. Here's an example:

We are testing our program and it gradually grinds to a halt. When we connect a debugger, we find that all of our threads, no matter what they are doing, eventually wind up hung in kernel32!EnumResourceLanguagesA. Can someone explain why that function is hanging, and why it seems all roads lead to it?

   0  Id: 12a4.1468 Suspend: 1 Teb: 000006fb`fffdc000 Unfrozen
kernel32!EnumResourceLanguagesA+0xbea00
kernel32!EnumResourceLanguagesA+0x2b480
bogosoft!CObjMarker::RequestBlockForFetch+0xf0
...

   1  Id: 12a4.1370 Suspend: 1 Teb: 000006fb`fffda000 Unfrozen
kernel32!EnumResourceLanguagesA+0xbea00
kernel32!EnumResourceLanguagesA+0x2b480
bsnetlib!CSubsystem::CancelMain+0x90

   2  Id: 12a4.1230 Suspend: 1 Teb: 000006fb`fffd8000 Unfrozen
NETAPI32!I_NetGetDCList+0x117e0
kernel32!EnumResourceLanguagesA+0x393a0
ntdll!LdrResFindResource+0x58b20
...

   3  Id: 12a4.cc0 Suspend: 1 Teb: 000006fb`fffd6000 Unfrozen
kernel32!EnumResourceLanguagesA+0xa80
bsnetlib!BSFAsyncWait+0x190
...

  4  Id: 12a4.1208 Suspend: 1 Teb: 000006fb`fffd4000 Unfrozen
kernel32!EnumResourceLanguagesA+0xbea00
kernel32!EnumResourceLanguagesA+0x2b480
bogosoft!TObjList<DistObj>::Get+0xb0

  5  Id: 12a4.1538 Suspend: 1 Teb: 000006fb`fffae000 Unfrozen
kernel32!EnumResourceLanguagesA+0xbf3d0
kernel32!EnumResourceLanguagesA+0x2c800
bsnetlib!Tcp::ReadSync+0x340
...

   6  Id: 12a4.16e0 Suspend: 1 Teb: 000006fb`fffac000 Unfrozen
ntdll!LdrResFindResource+0x61808
ntdll!LdrResFindResource+0x1822a0
kernel32!EnumResourceLanguagesA+0x393a0
ntdll!LdrResFindResource+0x58b20 
...

This stack trace looks suspicious for a variety of reasons.

First of all, look at that offset EnumResourceLanguagesA+0xbea00. It's unlikely that the EnumResourceLanguagesA function (or any other function) is over 750KB in size, as this offset suggests.

Second, it's unlikely that the EnumResourceLanguagesA function (or any other function, aside from obvious cases like tree walking) is recursive. And it's certainly unlikely that a huge function will also be recursive.

Third, it seems unlikely that the EnumResourceLanguagesA function would call, NETAPI32!I_NetGetDCList. What does enumerating resource languages have to do with getting a DC list?

Fourth, look at those functions that are allegedly callers of EnumResourceLanguagesA: bogosoft!CObjMarker::RequestBlockForFetch, bsnetlib!CSubsystem::CancelMain, bsnetlib!Tcp::ReadSync. Why would any of these functions want to enumerate resource languages?

These symbols are obvious wrong. The huge offsets are present because the debugger has access only to exported functions, and it's merely showing you the name of the nearest symbol, even though it has nothing to do with the actual function. It's just using the nearest signpost it can come up with. It's like if somebody gave you directions to the movie theater like this: "Go to city hall downtown and then go north for 35 miles." This doesn't mean that the movie theater is in the downtown district or that the downtown district is 35 miles long. It's just that the person who's giving you directions can't come up with a better landmark than city hall.

This is just another case of the principle that you have to know what's right before you can see what's wrong. If you have no experience with good stack traces, you don't know how to recognize a bad one.

Oh, and even though the functions in question are in kernel32, you can still get symbols for that DLL with the help of the Microsoft Symbol Server.

Comments (19)
  1. KT says:

    Oh yes, of course. How could anyone make such silly mistakes.

  2. DXdude says:

    Yes, the symbol server is a great resource, when the symbols you’re looking for are available.

    However, I have not found how to know for sure which symbols are in there, and how to ask for missing symbols to be added to it. The linked page merely explains how to use it, not what it contains.

    See, e.g.:http://forums.xna.com/forums/t/39102.aspx

    The worst part ? that missing file makes the "break on D3D error" useless, as visual studio fails to walk the stack then.

  3. Wojciech Gebczyk says:

    Hmm… It could be possible if this magic function will contain "compatibility hacks/patches" for all applications starting from Windows 95 AND has obfuscated name [because original name ApplyUglyHacksForApplication had compatilibity issues it self ;-)]?

    ;-)

    I would state that such thinking has same insanity level like making hacks in OS for broken applications… :P

  4. Henke37 says:

    You have to be insane to do some kinds of programing. Nobody sane would even try it. Yet, someone did it, successfully.

  5. ERock says:

    That movie theater analogy was brilliant and will be shamelessly stolen next time I have to explain distant offsets around seemingly normal symbols. Bravo!

  6. Teo says:

    One very nice side effect of the MS public symbols existence is that you can glimpse inside MS stuff. Example: see how much C++ is used in Windows :) Hint: not much. I know about WMI and that’s all. Another interesting observation is that Visual Studio uses ATL and no MFC.

    Aside from useless trivia, public symbols greatly help debugging our software, so thanks for everyone responsible for keeping them in sync.

  7. asf says:

    @Teo: what the hell are you talking about, there is plenty of c++ used in the shell & explorer, and older versions of VS are MFC

  8. Teo says:

    !asf, Right now Process Explorer cannot download the symbols from the MS server so I cannot check. I haven’t checked every dll in the System32 folder, so I may missed some files. But even if the shell uses C++ it’s just 2 or 3 dlls out of the thousands shipped with Windows. I do not care for old versions of VS, I believe that on the eve of 2010 it’s obvious I talked about VS 2008.

  9. porter says:

    > I believe that on the eve of 2010 it’s obvious I talked about VS 2008.

    On this site nothing is obvious, also we’ll have to wait till December 31st to find out what you talked about on the eve of 2010.

  10. Dan says:

    Actually VS 2010 Beta 2 is out too, so it’s even less obvious. :)

  11. Dean Harding says:

    Teo: I’m pretty sure the symbol server only shows the public symbols from most of the files you’ll find in system32. Since the public API of Win32 is C, of course the public symbols are also C. That says nothing about the internal implementation details of those DLLs, though.

  12. Medinoc says:

    @Dean: If by public symbols you mean exported ones, no, the symbol server gives much more than that.

  13. Gabe says:

    Considering the difficulty of using COM from C, I would bet that any part of Windows that uses COM would be written in C++. Explorer would be one significant example of this.

  14. DWalker says:

    I have looked at stack traces before, mostly in IBM mainframe code (!) and when I glanced at this one, I immediately thought that "EnumResourceLanguagesA+0xbea00" looked suspicious.  That offset just looks too big.

  15. Yuhong Bao says:

    BTW, VC disable loading of export symbols by default, to not show misleading symbol names.

    "how to ask for missing symbols to be added to it. "

    Yep, I was asking the same questions too:

    http://groups.google.com/group/microsoft.public.windbg/browse_frm/thread/11832ced96accccb/be56a50120dad61d

    http://groups.google.com/group/microsoft.public.windbg/browse_frm/thread/d4b3a48f0d2d712c/b3e576aee2a853e6

    I did get help from the MSXML team after I posted via the Contact form of their blog that a security update to MSXML4 was missing symbols, now it is fixed.

  16. Yuhong Bao says:

    "@Dean: If by public symbols you mean exported ones, no, the symbol server gives much more than that."

    Yep, for example symbols can be used to reveal the real name of ordinal-only exports.

  17. Teo says:

    Gabe, if you look at the thread stacks of Explorer in Process Explorer, you’ll see an awful little C++ functions. Don’t forget that C++ compiler mangle names, so they are quite easy to spot.

  18. Mike Dimmick says:

    One of the first things I do when setting up a new system is to download the Debugging Tools (http://www.microsoft.com/whdc/DevTools/Debugging/default.mspx) and set _NT_SYMBOL_PATH to

    C:Windows;C:WindowsSystem32;SRV*C:WebSymbols*http://msdl.microsoft.com/download/symbols

    Process Explorer needs to be told where you installed the debugging tools to, so it can use the appropriate DbgHelp.dll. See Options menu, Configure Symbols. If you have a stuck process, you can then look at the Threads tab and see which thread might be stuck. From there you can hit Stack and find out what the thread might be doing.

    If it’s a .NET program, though, you’ll normally see some huge offsets from somewhere as the native images for any .NET-supplied libraries (e.g. mscorlib.dll) don’t have associated symbols when the runtime generated the native image, which was done when it was installed on your computer. Everyone’s copy can be a little different. The CLR knows what the function is, but it has to be asked. The debugger extension !clrstack (in SOS.dll) can show you.

  19. @Teo: The non-mangling of names has nothing at all to do with the underlying language. It’s quite common to export undecorated names from a DLL written in C++ (or some other language) to make it easier to call. I write C++ DLLs like this all the time.

Comments are closed.

Skip to main content