If you can’t find the function, find the caller and see what the caller jumps to


You’re debugging a program and you want to set a breakpoint on some function, say, netapi32!Ds­Address­To­Site­NameW, but when you execute the bp netapi32!Ds­Address­To­Site­NameW command in the debugger, the debugger says that there is no such function.

The Advanced Windows Debugging book says that the bp command should set a breakpoint on the function, but the debugger says that the symbol cannot be found. I used the x netapi32!* command to see that the debugger did find a whole bunch of symbols, and it says that the symbols were loaded (from the public symbol store), but netapi32!Ds­Address­To­Site­NameW isn’t among them. The MSDN documentation says that Ds­Address­To­Site­NameW is in the netapi32.dll, but it’s not there! I can’t believe you guys stripped that function out of the symbol file, since it’s a function that people will want to set a breakpoint on.

Okay, first let’s rule out the conspiracy theory. The symbols were not stripped from the public symbols. And even if they were, that shouldn’t stop you, because after all, the loader has to be able to find the function when it loads your program, so it’s gotta be obtainable even without symbols.

Don’t be helpless. You already have the tools to figure out where the function is.

Just write a program that calls the function, then load it into the debugger and see what the destination of the call instruction is. You don’t even have to pass valid parameters to the function call, since you’re never actually executing the code; you’re just looking at it.

And hey looky-here, you already have a program that calls the function: The program you’re trying to debug! So let’s see where it goes.

0:001>u contoso!AwesomeFunction
...
00407352 call [contoso!__imp__DsAddressToSiteNameW (0040f104)]
...
0:001>u poi 0040f104
logoncli!DsAddressToSiteNameW:
7f014710 push ebp
7f014711 mov esp, ebp
...

There you go. The code for the function is in logoncli.dll.

What happened? How did you end up in logoncli.dll?

What you saw was the effect of a DLL forwarder. The code for the function Ds­Address­To­Site­NameW doesn’t live in netapi32.dll. Instead, netapi32.dll has an export table entry that says “If anybody comes to me asking for Ds­Address­To­Site­NameW, send them to logoncli!Ds­Address­To­Site­NameW instead.”

Officially, the function is in netapi32.dll for linkage purposes, but internally the function has been forwarded to another DLL for implementation. It’s like a telephone call-forwarding service for DLL functions, except that instead of forwarding telephone calls, it forwards function calls. You publish a phone number in all your marketing materials, and behind the scenes, you set up the number to forward to the phone of the person responsible for sales. That way, if that person quits, or the responsibility for selling the product changes, you can just update the call-forwarding table, and all the calls get routed to the new person.

That’s what happenned here. The MSDN phone book lists the function as being in netapi32.dll, and whenever a call comes in, it gets forwarded to wherever the implementation happens to be. And the implementation has moved around over time, so you should continue calling netapi32!Ds­Address­To­Site­NameW and let the call-forwarding do the work of getting you to the implementation.

Don’t start calling logoncli directly, thinking that you’re cutting out the middle man, or in a future version of Windows, your program may start failing with a “This number is no longer in service” error, like calling the direct office number for the previous sales representative, only to find that he left the company last month.

Comments (17)
  1. Jonathan says:

    My first thought was DLL forwarding. Got that a alot when KernelBase.dll was introduced. However, I would run "x *!Ds­Address­To­Site­NameW", grab a coffee, and return to find the relocated function.

  2. AC says:

    Couldn't the debugger be smart enought to recognize the DLL forwarding just like the loader does and correct your bp for you?

    [It could. The question is how smart you want your debugger to be. Sometimes it can be too smart for its own good. "Of course my DLL has that function. Look, see, I'm setting a breakpoint on it right now." And each time you add another step to symbol resolution, you make symbol resolution a tiny bit slower. -Raymond]
  3. lefty says:

    I still can't find Ds­Address­To­Site­NameW(), but I have no problem with Ds­Address­To­Site­NamesW().

    Is this a nitpick, a conspiracy theory, or a helpful pointer of a spelling error? Or am I just clueless?

  4. MItaly says:

    @lefty: the government made DsAddressToSiteNameW "disappear" since it started to know too much about chemtrails. Since now you know, they'll probably have to kill you.

  5. Antonio Rodríguez says:

    @Matteo: now you have to kill me, too. And thousands or readers. Of course.

  6. jon says:

    "And each time you add another step to symbol resolution, you make symbol resolution a tiny bit slower."

    I know you tend you have pretty old computers at Microsoft, but I would expect it to take a lot less time for my computer to do the forwarding lookup automatically than I would doing it manually…

    [It's not whether the computer can do it faster than you. It clearly can. It's whether the additional cost of doing this work when it ultimately fails outweighs the benefit of the rare cases where it actually succeeds. (Because approximately 99.99999% of all symbol lookups are for non-forwarded functions.) -Raymond]
  7. The cat says:

    I never saw this before, the comment thing.

  8. Killer{R} says:

    There're also other funny cases wih symbols names mismatched with corresponding exports. Start windbg without symbols path set and attach to process (w2k3 in this particular case):

    0:001> u USER32!SetActiveWindow

    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for E:WINsystem32USER32.dll –

    USER32!SetActiveWindow:

    7738a91f b8f5110000      mov     eax,offset <Unloaded_elp.dll>+0x11f4 (000011f5)

    7738a924 ba0003fe7f      mov     edx,offset SharedUserData!SystemCallStub (7ffe0300)

    7738a929 ff12            call    dword ptr [edx]

    7738a92b c20400          ret     4

    7738a92e 33c0            xor     eax,eax

    7738a930 40              inc     eax

    7738a931 e9c0380100      jmp     USER32!CharUpperA+0xfe (7739e1f6)

    7738a936 90              nop

    yep, we can see SetActiveWindow. There'is such API and there is such export. While windbg has no .pdb for user32.dll it deals with exports. Now set symbols path and see what happen:

    0:001> .symfix

    0:001> .reload

    Reloading current modules

    …………..

    0:001> u USER32!SetActiveWindow

    Couldn't resolve error at 'USER32!SetActiveWindow'

    oops, SetActiveWindow went away :( lets see what is at its address (7738a91f) now

    0:001> u 7738a91f

    USER32!NtUserSetActiveWindow:

    7738a91f b8f5110000      mov     eax,offset <Unloaded_elp.dll>+0x11f4 (000011f5)

    7738a924 ba0003fe7f      mov     edx,offset SharedUserData!SystemCallStub (7ffe0300)

    7738a929 ff12            call    dword ptr [edx]

    7738a92b c20400          ret     4

    7738a92e 33c0            xor     eax,eax

    7738a930 40              inc     eax

    7738a931 e9c0380100      jmp     USER32!CategoryMaskFromEvent+0x67 (7739e1f6)

    7738a936 90              nop

    thats the reason – now there is no SetActiveWindow, but NtUserSetActiveWindow – its actual in-code name as reflected in pdb..

    BTW I noticed that in latest Windows many system API moved from usual dlls to new ones.. Looks like MS hired special people to invent new dll names…

  9. Weng Fu says:

    It is easy to lookup the line number when debugging the Visual Basic 6. Not necessary to use the contosoAwasomFunction technique

    [AwesomeFunction was not a technique. It was just the name of the function that was trying to call DsAddressToSiteNameW in the first place. -Raymond]
  10. Neil says:

    So, if Killer's use case was fixed (search both symbols and public exports) would netapi32!DSAddressToSiteNameW be found in the public exports (as presumably GetProcAddress(GetModuleHandle("netapi32"), "DSAddressToSiteNameW) has to work)?

  11. "Looks like MS hired special people to invent new dll names"

    I thought it was called refactoring…  it takes really special people to do that. :)

  12. Leo Davidson says:

    "(Because approximately 99.99999% of all symbol lookups are for non-forwarded functions.)"

    Those 99.99999% of cases would incur no cost because the extra work is only needed when the direct lookup fails.

    Given the abysmal amount of time debuggers spend getting symbols in the first place (at least when you're not on the same LAN* as the symbol server), I do not think checking the loaded DLLs' import tables for forwards and acting accordingly would take a noticeable amount of time at all, and it would be rather useful.

    (*May contain hyperbole. :))

    [The extra work kicks in when you misspell a symbol because "Oh wait, maybe it's not really a typo but a fill in the blank. Let me go check that possibility." (Hint: Set up a symbol cache. Then it's slow only the first time.) -Raymond]
  13. Joshua says:

    [The extra work kicks in when you misspell a symbol because "Oh wait, maybe it's not really a typo but a fill in the blank. Let me go check that possibility." (Hint: Set up a symbol cache. Then it's slow only the first time.) -Raymond]

    Gee, and I would have had the debugger use the equivalent of RemoteGetProcedureAddress (yes I know this function doesn't exist) as a first pass so it could (1) function in the absence of symbols and (2) actually detect the forwarded case.

    [RemoteGetProcedureAddress would be horrible, because that might trigger a LoadLibrary if the forwarded-to DLL hasn't been loaded yet! -Raymond]
  14. Joshua says:

    [RemoteGetProcedureAddress would be horrible, because that might trigger a LoadLibrary if the forwarded-to DLL hasn't been loaded yet! -Raymond]

    It would have returned -ENOSYS because I wouldn't have dared let it load a DLL. Your intuition was spot on about can't load it.

  15. AC says:

    Of course computers should do humans job as much as possible. Do do boring manual work is stupid.

    [Doing unwanted work automatically is also stupid. -Raymond]
  16. Why does it feel that having to compile a program just to figure out the address of a given function is an overkill?

    I would encourage people to look at DUMPBIN or Dependency Walker to discover forwarded APIs and figure out where and how the function in question gets resolved.

    [Um, the address of the function isn't know until you run a program. (ASLR moves the function around.) And you didn't have to compile a new program. You already have a program compiled and ready to go: The program you are currently debugging! -Raymond]
  17. Thanks for your reply Raymond.

    I must have been confused by this line

    "Just write a program that calls the function, then load it into the debugger and see what the destination of the call instruction is." and skipped the following line about the program you're already debugging! :)

    But still, what if in the program I am debugging, I don't know Awesomefunction or other functions that calls what I thought to be netapi32!Ds­Address­To­Site­NameW ?

    For example, in many cases when I am debugging a program, I know (by the program's nature) that it will be calling a certain API. For instance, say that I expect a program to talk to a driver, I know it must be calling DeviceIoControl but I don't know Awesomefunction that calls it, so I won't be able to follow your method.

    And sorry, I did not really mean to say that one has to discover the addresses of the functions with dumpbin or DepWalker but instead discover how they are forwarded so one can directly put the correct breakpoint (modname!ApiName) in the debugger.

    [You can set the breakpoint via the import table. bp poi Contoso!__imp__DsAddressToSiteNameW. -Raymond]

Comments are closed.