Why do I get weird function names on my stack? (a discussion on symbols)

Symbols can contain information about global variables, local variables, function names, parameters, structures and source line numbers.

There are 3 types of symbols, export symbols, pdb symbols (public symbols) and private pdb symbols (private symbols).

The export symbols are part of the dll itself. For example ntdll.dll and kernel32.dll expose a big part of their functions as export symbols so that they can be called as API’s, but most dll’s that you find in a process have a very small set of exported symbols. Generally export symbols don’t contain parameter info for the functions and since very few functions are exposed this way you can’t really relay on the validity of the stacks when you have only export symbols.

Public symbols contain some basic symbols such as function names and global variables, but again, not all function names are exposed in public symbols. The developer of the dll chooses what to expose as public symbols and thus he/she can hide anything that they feel would give away too much information about the implementation.

Private symbols contain pretty much everything listed in the first paragraph

When debugging, symbols are matched up with the respective dlls or exe’s by a GUID linking the dll/exe to a symbol file (thanks Pavan for the correction). This means that if you have multiple ntdll.pdb’s in your symbol search path, the debugger will know which one corresponds to your particular version of ntdll.dll. The search path is given by .sympath, and apart from what is listed in your sympath the debugger will also look in the directory where the dll is loaded from as well as anything in the paths given in the environment variable _NT_SYMBOL_PATH.

So… what happens if you have the wrong symbols?

Let’s look at this stack with public symbols for mscorsvr.dll: (sorry about the formatting)

  54  Id: 62c.1590 Suspend: 1 Teb: 7ffa2000 Unfrozen
ChildEBP RetAddr Args to Child
1212ef44 7c59a030 00000090 00000000 1212ef64 ntdll!NtWaitForSingleObject+0xb [i386\usrstubs.asm @ 2004]
1212ef6c 7c57b3db 00000090 00009c40 00000000 kernel32!WaitForSingleObjectEx+0x71 [D:\nt\private\windows\base\client\synch.c @ 1309]
1212ef7c 791b578b 00000090 00009c40 00000000 kernel32!WaitForSingleObject+0xf [D:\nt\private\windows\base\client\synch.c @ 1217]
1212efa0 791dbe6e 00000000 00000000 00000000 mscorsvr!ThreadpoolMgr::WorkerThreadStart+0x3a
1212ffb4 7c57b388 0d406838 00000002 00000000 mscorsvr!ThreadpoolMgr::intermediateThreadProc+0x44
1212ffec 00000000 791dbe2d 0d406838 00000000 kernel32!BaseThreadStart+0x52 [D:\nt\private\windows\base\client\support.c @ 460]

We are creating a worker thread and then sit and wait for work

Let’s look at the same stack with export symbols for mscorsvr:

0:054> kb
ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
1212ef6c 7c57b3db 00000090 00009c40 00000000 ntdll!NtWaitForSingleObject+0xb
*** ERROR: Symbol file could not be found. Defaulted to export symbols for mscorsvr.dll -
1212efa0 791dbe6e 00000000 00000000 00000000 kernel32!WaitForSingleObject+0xf
1212ffb4 7c57b388 0d406838 00000002 00000000 mscorsvr!GetCompileInfo+0x8e99
1212ffec 00000000 791dbe2d 0d406838 00000000 kernel32!lstrcmpiW+0xb7

The debugger is nice and tells us that we couldn’t find the symbol file for mscorsvr, but yet it gives us a function name (from the export symbols) so it appears that we’re calling some function called GetCompileInfo, hmm… very strange, why did it pick that name?

If we were to list the symbols (export symbols for mscorsvr) we would get a list that looks like this

0:054> x mscorsvr!*
791b0000 mscorsvr!Ordinal73 =
791b0000 mscorsvr!Ordinal76 =
791b0000 mscorsvr!Ordinal77 =
791b0000 mscorsvr!Ordinal75 =
791b0000 mscorsvr!Ordinal78 =
791b0000 mscorsvr!Ordinal71 =
791b0000 mscorsvr!Ordinal79 =
791b0000 mscorsvr!Ordinal74 =
791b0000 mscorsvr!Ordinal72 =
791d2fd5 mscorsvr!GetCompileInfo =
791e0920 mscorsvr!GetAssemblyMDImport ()

So we are executing something at address 791dbe6e (791d2fd5+0x8e99), if we take a peak at the symbols with public symbols loaded, we would see that that address landed inside the intermediateThreadProc function (+0x44 to be exact)

791bbb78 mscorsvr!SimpleRWLock::LeaveRead = 
793eaf50 mscorsvr!ExecutionManager::m_dwReaderCount =
791dbe2d mscorsvr!ThreadpoolMgr::intermediateThreadProc =
793035c5 mscorsvr!VtBoolMarshaler::UnmarshalComToNativeIn =
791f8e0d mscorsvr!LayoutClassPtrMarshaler::ClearNativeTemp =

So in essence, the reason it picked the name GetCompileInfo was that that was the last symbol it could find, preceding the address that we were executing at.

You can see how this can easily get confusing. Oh and btw, not only did it give us a completely wrong function name, but if you take a closer look at the two stacks, we lost a whole function call on the stack with the export symbols (The call to WorkerThreadStart). The reason for this is that the function we were supposed to look at was at address 791b578b and this address is located before our first exported function so it can’t even resolve it to something fake.

How do you know if your symbols are good?

For starters, just by looking at the stacks we can see that the function names seem pretty weird. It doesn’t make sense that a thread would start with lstrcmpiW, they usually start with BaseThreadStart or something to that effect. The second thing that gives it away just by looking at the stack is that we were supposed to be 0x8e99 instructions in to the function GetCompileInfo, that is 36505 instructions, wow, that would be a mighty long function.

The more definite way of knowing would be to run lmv on the executable. (notice the extra m before mscorsvr for match pattern)

0:054> lmv mmscorsvr
start end module name
791b0000 79418000 mscorsvr (export symbols) mscorsvr.dll
Loaded symbol image file: mscorsvr.dll
Image path: C:\WINNT\Microsoft.NET\Framework\v1.1.4322\mscorsvr.dll
Image name: mscorsvr.dll
Timestamp: Thu Jul 15 09:26:32 2004 (40F631A8)
CheckSum: 002664D4
ImageSize: 00268000
File version: 1.1.4322.2032
Product version: 1.1.4322.2032
File flags: 20 (Mask 3F) Special
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04e4
CompanyName: Microsoft Corporation
ProductName: Microsoft .NET Framework
InternalName: MSCORSVR.DLL
OriginalFilename: mscorsvr.dll
ProductVersion: 1.1.4322.2032
FileVersion: 1.1.4322.2032
FileDescription: Microsoft .NET Runtime Common Language Runtime - Server
LegalCopyright: Copyright © Microsoft Corporation 1998-2002. All rights reserved.
LegalTrademarks: Microsoft® is a registered trademark of Microsoft Corporation. Windows(TM) is a trademark of Microsoft Corporation
Comments: Microsoft .NET Runtime Common Language Runtime - Server

Lmv will give you a lot of good information about the dll, but in this particular case what we are interested is what is printed after the module name, i.e export symbols in this case. (You can also get this for all modules by running lm)

One last comment on this: As you may have noticed, going to the public Microsoft symbols server you sometimes get public symbols and sometimes end up with export symbols for mscorsvr.dll for example. This is because some cases like certain hotfixes and private builds have not been uploaded to the public symbol server.

Lazy symbol loading and downloading symbols for offline debugging

If you run lm you will notice that a lot of the dll’s and exe’s will have their symbols listed as deferred. This is because the debugger doesn’t actually load all the symbols when the debugger starts (as this would take a looooooooong time especially if you have a remote symbol store). Symbols are loaded on an as needed bases, i.e. when you run kb (to list the stacks) or run x to examine the symbols for a particular executable, or any other command that requires symbol lookup.

Sometimes though, it is useful to download all the symbols for a process, a common example would be if your application is running on a machine with no internet access and you want to be able to live debug it.

In this case what you can do is this:

  1. Take a dump of the process when all modules are loaded and open it on a machine with internet access.
  2. Set the sympathy to srv*c:\myappsymbols*http://msdl.microsoft.com/download/symbols;
  3. run .reload /i /f to force load all the symbols for the dump

This will download all the symbols available on Microsofts public symbol server that match your dlls into the c:\myappsymbols and then you can simply copy this folder over to the machine without internet access and set the symbolpath there to srv*c:\myappsymbols*c:\myappsymbols;c:\folderwithanyadditionalsymbols.

But… some of the functions on my stack only show up as addresses, what is up with that?

This really doesn’t have anything to do with symbols but I thought it was worth mentioning anyways. In the callstack below we can see one function call only listed by it’s address… 0xd44334a, the reason for this is that it is a managed function which doesn’t have a native translation, and after all windbg is merely a native debugger. You can run !clrstack or !ip2md on the address to get the .net function name.

0:045> kb
ChildEBP RetAddr Args to Child
1182ecd0 7c59c5b7 7920d6a7 00000010 0f39c800 ntdll!NtYieldExecution+0xb [i386\usrstubs.asm @ 2108]
1182ecd4 7920d6a7 00000010 0f39c800 00000010 kernel32!SwitchToThread+0x6 [D:\nt\private\windows\base\client\thread.c @ 2611]
1182ed00 791c0283 0f39c800 00000010 00000000 mscorsvr!gc_heap::allocate_more_space+0xe3
1182ef28 791b6930 0f39c800 00000010 00000000 mscorsvr!GCHeap::Alloc+0x7b
1182ef3c 791c0359 00000010 00000000 00000000 mscorsvr!Alloc+0x3a
1182ef5c 791c03a7 0d421928 1182efec 062eff1c mscorsvr!FastAllocateObject+0x25
1182efc8 0d44334a 00000000 1182f01c 1182f040 mscorsvr!JIT_NewFast+0x2c
WARNING: Frame IP not in any known module. Following frames may be wrong.
1182effc 791b33ec 1182f114 791b3c96 1182f050 0xd44334a
1182f004 791b3c96 1182f050 00000000 1182f028 mscorsvr!CallDescrWorker+0x30
1182f114 791b5612 00bf1bdb 79b7a000 00000004 mscorsvr!MethodDesc::CallDescr+0x1b8
1182f1d0 791b56c6 79bf1bdb 79b7a000 79bd1d16 mscorsvr!MethodDesc::CallDescr+0x4f
1182f1f8 791b5c40 1182f210 0ba57968 0f39c7c8 mscorsvr!MethodDesc::Call+0x97
1182f2ac 791b5b97 0f3a6480 00000001 00000000 mscorsvr!AddTimerCallbackEx+0x165
1182f2c0 791b4de8 1182f374 791b3cf4 ffffffff mscorsvr!AddTimerCallback_Wrapper+0x13
1182f308 791b5ae5 0ba57968 791b5b84 1182f374 mscorsvr!Thread::DoADCallBack+0x5c
1182f3c4 791b59ce 0f3a6480 00000001 00000001 mscorsvr!AddTimerCallbackEx+0x165
1182f3d8 791b59a5 0f3a6480 00000001 791b5996 mscorsvr!AddTimerCallback+0x10
1182f3ec 791b5802 0f3a2230 0f3b6bf0 793ea988 mscorsvr!ThreadpoolMgr::AsyncTimerCallbackCompletion+0xf
1182f400 791b57c1 0f3b6bf0 00000000 791b5751 mscorsvr!ThreadpoolMgr::ExecuteWorkRequest+0x19
1182f420 791dbe6e 00000000 00000000 00000000 mscorsvr!ThreadpoolMgr::WorkerThreadStart+0x129

0:045> !clrstack
Loaded Son of Strike data table version 5 from "C:\WINNT\Microsoft.NET\Framework\v1.1.4322\mscorsvr.dll"
Thread 45
0x1182efa4 0x77f88fe3 [FRAME: HelperMethodFrame]
0x1182efd0 0x0d44334a [DEFAULT] [hasThis] Void System.Timers.Timer.MyTimerCallback(Object)
0x1182f2cc 0x791b33ec [FRAME: ContextTransitionFrame]

How about symbols for managed dlls? Do I need them?

Technically for debugging with windbg the answer is that you would need them very seldom or never. If you do have managed symbols and the dlls are compiled in debug mode you can get line numbers and source files. You can also get some parameter info when using !clrstack –p. But all in all, the extra “stuff” you get from managed symbols is very little when using a native debugger like windbg.

Over and out…  


Comments (20)
  1. Wow thank you! This stuff is just mind blowing. How did you learn all that?

  2. jdelinger says:

    this blog is one (if not) the best debugging blogs I’ever visited. — John – Texas

  3. I have talked about a number of different hang/performance issues in my posts. This post is a generic

  4. Coderjoe says:

    Wow. I was trying to figure out how I was going to load symbols for Windows DLLs that don’t seem to match the symbol packs while remote debugging without being able to access the internet to get to the symbol server (stupid VPN software redirects all traffic except DHCP and the VPN connection over the VPN, and the customer site does not allow internet access.) I’m debugging using Visual Studio, and was not aware that I could create a dump file using it. I’m currently waiting for the dump file to transfer across the VPN so I can load it while I have internet access. I don’t see why this wouldn’t work, though. Thanks!

    (found this blog entry via a google search, fwiw)

  5. Coderjoe says:

    Apparently, I can create the dump using VS, but in order to download the symbols, I have to load the dump into WinDbg and do the force reload command given above. Once I do that, everything works. Thanks again!

  6. OnereeAcron says:

    Sorry pelase :

    Wrong cdteagory…

    wippbe careful

  7. .NET Debugging Demos This is a series of debugging demos aimed to help you get some hands on experience

  8. When you debug .net applications you will sometimes get error messages in windbg. Here are a few of the

  9. If you have read any of my posts you have probably noticed that I am very partial to windbg and the debugging

  10. Justin says:



  11. pavan says:

    The debugger matches the symbols not by the timestamp and checksum but by a GUID that links the dll and pdb. This GUID is generated by the compiler and is inserted into the dll and pdb headers.


  12. yanglei says:

    0:054> lmv mmscorsvr

    shoulde be

    0:054> lmv m mscorsvr

  13. Tess says:

    Yanglei,  you can do it either way

  14. JR says:

    I was wondering what steps are need for offline dump analysis for .net code. I always end up xcopying the tools to the prod server because when I take a crash dump and copy it local the managed functions are gone.

    Example !clrstack from server

    0:031> !clrstack

    OS Thread Id: 0x754 (31)

    ESP       EIP    

    1ed6ed94 7939f908 System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].FindEntry(System.__Canon)

    1ed6edb0 7939f510 System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].get_Item(System.__Canon)

    1ed6edbc 1ee44048 MarketingBusinessLayer.DefaultSettingsConfigHelper.BuildDefaultPathFromConfigWithSiteURL(System.String)

    1ed6edcc 1ee43f79 MarketingBusinessLayer.DefaultSettingsConfigHelper.BuildFolderPathForSelectedFile(System.String, System.String, System.String)

    1ed6ede4 1fcc16b9 ControlFactory.BuildUserControl(System.Xml.XPath.XPathNavigator)

    1ed6edf4 1fcc1525 _Default.Page_Load(System.Object, System.EventArgs)

    1ed6ee30 698a1928 System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)

    1ed6ee40 6891d73f System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)

    1ed6ee54 687ca5a4 System.Web.UI.Control.OnLoad(System.EventArgs)

    1ed6ee64 687ca5f0 System.Web.UI.Control.LoadRecursive()

    1ed6ee78 687dbbf6 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)

    1ed6f030 687db521 System.Web.UI.Page.ProcessRequest(Boolean, Boolean)

    1ed6f060 687db487 System.Web.UI.Page.ProcessRequest()

    1ed6f098 687db3a7 System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)

    1ed6f0a0 687db33a System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)

    1ed6f0b4 1fcc0a95 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)

    1ed6f0b8 686888df System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()

    1ed6f0ec 6865a071 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)

    1ed6f12c 6865a39b System.Web.HttpApplication.ResumeSteps(System.Exception)

    1ed6f174 6865912d System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object)

    1ed6f190 6865e0bd System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest)

    1ed6f1c4 6865dd72 System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest)

    1ed6f1d0 6865c447 System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr, Int32)

    1ed6f380 79f1ef33 [ContextTransitionFrame: 1ed6f380]

    1ed6f3d0 79f1ef33 [GCFrame: 1ed6f3d0]

    1ed6f528 79f1ef33 [ComMethodFrame: 1ed6f528]

    then when I copy the dump file local

    0:031> !clrstack

    OS Thread Id: 0x754 (31)

    ESP       EIP    

    1ed6ed94 7939f908

    1ed6edb0 7939f510

    1ed6edbc 1ee44048

    1ed6edcc 1ee43f79

    1ed6ede4 1fcc16b9

    1ed6edf4 1fcc1525

    1ed6ee30 698a1928

    1ed6ee40 6891d73f

    1ed6ee54 687ca5a4

    1ed6ee64 687ca5f0

    1ed6ee78 687dbbf6

    1ed6f030 687db521

    1ed6f060 687db487

    1ed6f098 687db3a7

    1ed6f0a0 687db33a

    1ed6f0b4 1fcc0a95

    1ed6f0b8 686888df

    1ed6f0ec 6865a071

    1ed6f12c 6865a39b

    1ed6f174 6865912d

    1ed6f190 6865e0bd

    1ed6f1c4 6865dd72

    1ed6f1d0 6865c447

    1ed6f380 79f1ef33 [ContextTransitionFrame: 1ed6f380]

    1ed6f3d0 79f1ef33 [GCFrame: 1ed6f3d0]

    1ed6f528 79f1ef33 [ComMethodFrame: 1ed6f528]

    it seems as though windbg needs the actual .net exe and dll files to get the info from because none of the .net code was built debug. Is offline dump analysis of sites that use the gac and temp dlls possible / practical without copying a bunch of files along with the dump file?

  15. Tess says:

    it looks like a problem with sos where you dont have the proper symbols for mscordacwks on the client.  Do you have the same framework version on the client and server?

  16. Tim says:

    Tess, I’m getting some problems with loading the symbols for the apps I need to debug. They are a mish-mash of "co-operative" ("promiscuous" is more accurate really) Smalltalk, Win32 C API and .Net C++/C# exes and dlls. I think part of the problem is because the .Net bits are using a point-release/hotfixed version of the .Net Frameworks (v1.0.3705) for which windbg can’t find the appropriate symbols (or not at http://msdl.microsoft.com/download/symbols, anyway). First, does this sound like a plausible cause of ‘problems’ for windbg? If so how badly will it affect what I can reliably do in windbg with these kind of apps if I can’t resolve it? And do you know if its possible to get symbols for this version of the .Net Frameworks somehow?



    PS this blog is really useful – tvm.

  17. Tess says:

    Tim, for the actual .net functions you shouldnt need the symbols but are you talking about native components like mscorsrv?  Typically hotfix symbols also make it on the public symbol servers unless its an unpublished one.  

    There is also the issue of 1.0 being out of support and symbols for unsupported products get removed from the symbol servers after a while if i recall correctly.

  18. Tim says:

    What I’m trying to do is get windbg to download an appropriate symbol cache we can then deploy for use against the live system to try to get some diagnostic info when the system hangs/crashes. The problem is there’s an awful lot of disparate technology involved in the system and some of it is getting a bit past its sell-by date.

    So anyway, in windbg I’ve setup the sympath as "SRV*C:symbols*http://msdl.microsoft.com/download/symbols;C:mysystempdbdebug", fired up a typical app from the system, attached windbg to the process and then tried the ".reload /i /f" command to populate the local symbol cache directory. At this point I get the following errors/warnings (amongst some others related to the system itself):

    0:008> .reload /i /f

    Reloading current modules

    ..*** ERROR: Module load completed but symbols could not be loaded for C:WINDOWSsystem32xpsp2res.dll

    .*** ERROR: Module load completed but symbols could not be loaded for c:windowsassemblygacmicrosoft.visualc7.0.3300.0__b03f5f7f11d50a3amicrosoft.visualc.dll

    ….*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:PROGRA~1SophosSOPHOS~1SOPHOS~1.DLL –

    ……………………..*** WARNING: Unable to verify checksum for c:windowsassemblynativeimages1_v1.0.3705mscorlib1.0.3300.0__b77a5c561934e089_a2440ec1mscorlib.dll

    *** ERROR: Module load completed but symbols could not be loaded for c:windowsassemblynativeimages1_v1.0.3705mscorlib1.0.3300.0__b77a5c561934e089_a2440ec1mscorlib.dll

    ..*** WARNING: Unable to verify checksum for c:windowsassemblynativeimages1_v1.0.3705system1.0.3300.0__b77a5c561934e089_f297e772system.dll

    *** ERROR: Module load completed but symbols could not be loaded for c:windowsassemblynativeimages1_v1.0.3705system1.0.3300.0__b77a5c561934e089_f297e772system.dll

    ..*** WARNING: Unable to verify checksum for c:windowsassemblynativeimages1_v1.0.3705system.drawing1.0.3300.0__b03f5f7f11d50a3a_666a8c8csystem.drawing.dll

    *** ERROR: Module load completed but symbols could not be loaded for c:windowsassemblynativeimages1_v1.0.3705system.drawing1.0.3300.0__b03f5f7f11d50a3a_666a8c8csystem.drawing.dll

    ..*** WARNING: Unable to verify checksum for c:windowsassemblynativeimages1_v1.0.3705system.windows.forms1.0.3300.0__b77a5c561934e089_ad8c3507system.windows.forms.dll

    *** ERROR: Module load completed but symbols could not be loaded for c:windowsassemblynativeimages1_v1.0.3705system.windows.forms1.0.3300.0__b77a5c561934e089_ad8c3507system.windows.forms.dll


    I’m just paddling in the shallow end with windbg so I’m not sure how much these type of problems are going to compromise the diagnostic info if I can’t solve them and (generally) how useful windbg will be in this situation (the situation is broadly any combination of .Net 1.0 Managed C++ and native Win32 C user components you can imagine).



  19. Tess says:


    If I would venture a guess, I think that the 1.0 symbols might have been removed from the public symbol server to save some space, as 1.0 hasn’t been supported for quite some time now.

  20. Artjom says:

    Hi Tess, what is going wrong if I see the following entry:

    ? <Unloaded_tle.dll>+4dffc8

    Best regards,


Comments are closed.

Skip to main content