Don’t be helpless: You can put things together, it doesn’t have to be a single command

Humans are distinguished among all animal species by their advanced development of and heavy reliance on tools. Don’t betray your ancestors. Use those tools you have.

For example, during the debugging of a thread pool problem, it looked like somebody did a PostThreadMessage to a thread pool thread and left the message unprocessed after the thread pool function returned. Who could it have been? Well, one idea was to see if there were any DLLs in the system which called both QueueUserWorkItem and PostThreadMessage.

I did a little legwork and contributed the following analysis to the mail thread:

Of all the DLLs loaded into the process, the following call PostThreadMessage:

SHLWAPI.dll 77D72436 221 PostThreadMessageA
SHELL32.dll 77D78596 222 PostThreadMessageW
ole32.dll 77D78596 222 PostThreadMessageW
… (list trimmed; you get the idea) …

Of those DLLs, these also call QueueUserWorkItem:

… (list trimmed; you get the idea) …

Astounded, somebody wanted to know how I came up with that list.

Nothing magic. You have the tools, you have a brain, so connect the dots.

The lm debugger command lists all the DLLs loaded into the process. Copy the output from the debugger window and paste it into a text file. Now write a little script that takes each line of the text file and does a link /dump /imports on the corresponding DLL. I happen to prefer perl for this sort of thing, but you can use a boring batch file if you like.

for /f %i in (dlls.txt) do ^
@echo %i & link /dump /imports %i | findstr PostThreadMessage

Scrape the results off the screen, prune out the misses, and there you have it.

“I tried that, but the result wasn’t in the same format as what you posted.”

Well, yeah. There’s no law that says that I can’t manually reformat the data before presenting it in an email message. Since there were only a dozen hits, it’s not worth writing a script to do that type of data munging. Typing “backspace, home, up-arrow” twelve times is a lot faster than writing a script to take the output of the above batch file and turn it into the output I used in the email message.

Another boring batch file filters the list to those DLLs that also call QueueUserWorkItem. Writing it (or a script in your favorite language) is left as an exercise.

No rocket science here. Just taking a bunch of tools and putting them together to solve a problem. That’s what your brain is for, after all.

Comments (29)
  1. BryanK says:

    Well, "lm" was the part that I would have been missing, had anyone asked me.  (Not that anyone did.)  But then, I do very little with WinDbg, and nothing with kd/ntsd (or any other debugger on that level), so that’s not entirely surprising.  ;-)

  2. mikeb says:

    The first question, "how did you come up with the list?" is a legitimate question – the person may not be aware of the tools available (debuggers are complex, after all).  I think that even just pointing out the primary tools used (the debugger’s lm command to get the full DLL list and the "link /dump /imports" command to figure out which ones had the interesting imports) would be an adequate answer – the scripting details are nice, but shouldn’t be necessary.

    The seconds question is a bit more deserving of the condescension.

  3. Wojciech Gebczyk says:

    mikeb: It’s true! it’s even ‘truer’, not everyone knows that there is something like compiler behind click-drag-drop designer in VS.


  4. Maurits says:

    taking a bunch of tools and putting them together to solve a problem

    So true.  Humans are the dominant species – not because we’re the fastest, the strongest, or the most fecund – but because we’re the most adept and creating and using tools.

    (Which is why I think the Tour de France should just let people use whatever drugs or doping techniques they can come up with.)

  5. Christian says:

    Typing "backspace, home, up-arrow" twelve times is a lot faster

    Oh that is so a great topic!

    I always try to get a rhytm in it so that I don’t split a key when I have to insert 100 parentheses or so…

    What are your favourite tools to do this?

    Besides pressing keys again and again?

    I use ALT to select blocks in file, and sometimes (e.g. to insert data like country-codes into sql-server some Excel. Put it in columns, add ","-column, expand it by dragging it down, the same for "INSERT INTO …." and =")" or ‘ and then copy it to notepad2 and replace t with nothing.

    Or something like this…

    I ALWAYS wonder whether I should learn vi, emacs, more keyboard combinations or write a C#-programm or script, but it’s too much work. Especially learning any cool keyboard shortcuts in vim etc., if these even exist, I don’t know!

    Somebody should write a little "text studio" that helps you very quickly to do such things….

  6. KenW says:

    And for those of you using Borland/CodeGear tools instead of Microsoft’s, you can accomplish the equivalent of all of these steps using tdump.exe – no need for WinDbg at all.

    tdump -em yourapplication.exe temp.txt

    produces a file containing content like this:

    IMPORT:     oleaut32.dll=’SysFreeString’

    IMPORT:     advapi32.dll=’RegQueryValueExA’

    IMPORT:     advapi32.dll=’RegOpenKeyExA’

    IMPORT:     advapi32.dll=’RegCloseKey’

    IMPORT:       user32.dll=’GetKeyboardType’

    IMPORT:       user32.dll=’DestroyWindow’

    IMPORT:       user32.dll=’LoadStringA’

    IMPORT:       user32.dll=’MessageBoxA’

    IMPORT:       user32.dll=’CharNextA’

    Which can be parsed using a similar batch file for loop.

    [This misses DLLs loaded via LoadLibrary however. -Raymond]
  7. Koro says:

    KenW: I doubt tdump would list dynamically-loaded modules too (unless it actually runs the app)

    Raymond: I wholeheartedly agree with you. I’d have done the same thing instinctively, I can’t figure how people can fail to connect the dots sometimes…

    (As for the comment "the person may not be aware of the tools available", well, Google is your friend)

  8. Miles Archer says:

    Like programming, troubleshooting is a skill you either have or don’t have.

  9. Gene says:

    Koro: "(As for the comment "the person may not be aware of the tools available", well, Google is your friend)"

    So when a live person who just solved your specific problem and is clearly inclined to be of assistance is available, it is the proper action to shove them away and start pawing guesses at what they did into Google to find out?  Instead of just asking?

    Alrighty then.

  10. M Hotchin says:

    I’ve been carrying around a program called ‘tlist’ for years.  It lists all your tasks (hence the name), but it will also dump out a list of modules loaded in process.

    c:>tlist -?

    Microsoft (R) Windows NT (TM) Version 5.1 TLIST

    Copyright (C) Microsoft Corp. 1981-1999

    c:>tlist cmd.exe

    416 cmd.exe           CMD – tlist cmd.exe

      CWD:     c:

      CmdLine: "C:WINDOWSsystem32cmd.exe"

      VirtualSize:    30312 KB   PeakVirtualSize:    36644 KB

      WorkingSetSize:  2596 KB   PeakWorkingSetSize:  2612 KB

      NumberOfThreads: 1

       420 Win32StartAddr:0x4ad05046 LastErr:0x000000cb State:Waiting

     5.1.2600.5512 shp  0x4ad00000  cmd.exe

     5.1.2600.5512 shp  0x7c900000  ntdll.dll

     5.1.2600.5512 shp  0x7c800000  kernel32.dll

     7.0.2600.5512 shp  0x77c10000  msvcrt.dll

     5.1.2600.5512 shp  0x7e410000  USER32.dll

    (etc etc etc).  

    From here it’s just a hop, skip and a jump to the above info…

    Apparently XP ships with ‘tasklist.exe’ which can do similar things.

  11. DrkMatter says:

    "(As for the comment "the person may not be aware of the tools available", well, Google is your friend)"

    Google can be a dumb, annoying friend when you do not know where to start looking. The best answers are still provided by directly asking experts in the field, especially if it can save you a few hours of walking in the dark.

    Of course this is no excuse for intellectual laziness: said experts have limited time, energy and interest. Optimal resource usage requires for you to at least research a bit.

  12. Tom says:

    @Christian: You can use a scriptable text editor.  Or, use a text-editor with lots of built in functionality.  For example, PSPad lets you insert data at specific column positions — doesn’t always do the job, but can sometimes get you 90% of the way there.

  13. Ian says:

    Mark Russinovich’s Process Explorer is absolutely brilliant for this sort of thing. I use it as a replacement for my Windows Task Manager.

    As for munging text files, I tend to use a combination of regular expression search and replace in Visual Studio, quite a lot of "backspace, home, up-arrow", and the occasional adventurous foray into sed.

  14. mikeb says:

    > (As for the comment "the person may not be aware of the tools available", well, Google is your friend)

    I think it’s entirely appropriate for someone in an existing discussion to ask what was used to come up with the list.

    I also think it’s entirely appropriate for the answer to be something as simple as:  "I used the debugger’s lm command to find out the DLLs loaded, then used "link /dump /imports" to find the ones that used the APIs of interest."

    If at that point the inquisitive one wants to know how you got the list formatted the way you did, then it’s time to start rolling your eyes and maybe do a dressing-down.

    But then, there aren’t any hard and fast rules about this – it’s certainly within ones rights to start questioning the guy’s intelligence right off the bat.  But then you might have to live with people comparing your social skills to an atom bomb.  There are certainly worse things things in life.

  15. ChrisR says:

    For text munging of the kind here I tend to use the search-replace regexp in eclipse, since that’s our IDE at work and I usually have a copy open.  If I’ve not got one open, I might just do the same thing in a perl one-liner.

    Eclipse can do pretty much everything you would expect – pull out parts of matching lines, use them in the replacement expression etc.

    In the past I’ve used the text importer in excel for similar jobs, including those where you want to build a batch file where each line only differs in one or two columns.

    For just 12 lines, just doing it manually would be faster than almost anything else.

  16. Mark says:

    Although it can be one command:

    .foreach (module {lm 1m}) {x ${module}!*postthread*}

  17. Ulric says:

    Us Visual C++ users can type this command in the VC++ View->Other Windows->Command Window, while debugging:

    Debug.ListModules /Address:no /SymbolStatus:no /Path:no

  18. DM says:

    You can also complement Mark’s Process Explorer with Dependency Walker.


  19. Alexandre Grigoriev says:

    Posting messages to a thread you don’t own is extremely dumb idea. In a workitem thread you just process your workitems, you don’t mess with your thread state (which the message queue is part of). You can’t do any window messages-related actions, because it messes the thread state.

  20. AC says:

    Posting messages to a thread you don’t own is extremely dumb idea.

    Whoa, for a sec I thought you were talking about posting messages to a message forum thread!

  21. Anon says:

    Raymond, what you did is clever and would be non obvious to ~99% of humanity. And, sadly, about 80% of Windows developers. Though maybe 10% of them could solve ‘most’ of the problem, like missing the dynamically loaded DLLs.

  22. early/late binding says:

    [This misses DLLs loaded via LoadLibrary however. -Raymond]

    Does the link command display this? How can a program display late binded dependencies without executing the program? Do the link command execute the program, ie code which calls LoadLibrary?

    Either you display:

    1. Dependencies (early+some/all late binded) of a running process, or:
    2. Dependencies (early binded) of exe/dll files without starting the process.

    Does the link command do 1 or 2?

  23. Spike says:

    @Christian, and Raymond for that matter!

    If you use Visual Studio then Ctrl-Shift-R is your friend.

  24. KenW says:

    Raymond & Koro:

    Good catch on the dynamically loaded modules and tdump. I didn’t think it through that far. :-( Thanks for the correction.

    However, if I had I would have added Dependency Walker and Process Explorer, either of which will handle dynamic loading (Dependency Walker’s profiling works extremely well for this).

  25. carl says:

    Does the link command do 1 or 2?

    neither. raymond used the output of the debugger to decide which files to examine.

  26. RF says:

    Awesome that you’re using Perl; hooray eclecticism.  

    Incidentally, Perl goes to special effort to handle signals.  When it receives most signals, it sets a flag to make the interpreter deliver the signal after the current Perl opcode finishes, so it should be safe to run any Perl builtin in your signal handler.

  27. BryanK says:

    Which files to examine, yes.  But that’s only one side of the problem.  ;-)

    There’s an issue here (I think), where if one of the loaded DLLs calls PostThreadMessage or QueueUserWorkItem dynamically, it may not show up in the list.  The DLL will show up in the initial list if *it* gets loaded dynamically, but if it calls one of those functions via GetProcAddress, it could still get missed, because it won’t have an import listed for the function, and “link /dump /imports” won’t show PostThreadMessage (or QueueUserWorkItem) for it, when it does call those functions.

    Yes, this is probably a pathological case, but I believe it’s still possible.

    [True, but that’s a risk I was willing to take for this exercise. -Raymond]
  28. Keith Hill says:

    For those interesed in how to find DllEntry points used by executables in PowerShell check out this post:!5A8D2641E0963A97!6280.entry

Comments are closed.