What is this extra thread in my process?


A customer liaison asked:

After applying Service Pack 2 to Windows Server 2003, my customer found that a simple MFC application (just using the template, no customization) has two threads when it is supposed to have only one. After five minutes, one of the threads exits. This doesn’t happen on Windows Server 2003 RTM or Windows Server 2003 Service Pack 1.

Here is a stack trace of the extra thread:

0:001> kP
ntdll!KiFastSystemCallRet
ntdll!NtWaitForMultipleObjects+0xc
ntdll!EtwpWaitForMultipleObjectsEx+0xf7
ntdll!EtwpEventPump+0x27f
kernel32!BaseThreadStart+0x34

The parameters to Etwp­Wait­For­Multiple­Objects­Ex seem to be consistent with Wait­For­Multiple­Objects­Ex. Assuming that’s the case, the parameters are

nCount  = 0
lpHandles  = 00x004f4470
bWaitAll  = 0
dwMilliseconds  = 300000 = 5 minutes
bAlertable  = TRUE

Can you explain what the purpose of this thread is? Did this behavior change as a result of the update? It is important for the customer to know the purpose of this thread.

We asked, “Why does the customer need to know the purpose of this thread?”

We never heard back from the customer liaison. I guess it wasn’t that important after all.

In case you cared: From the names of the functions, it looks like this is the ETW event pump thread.

Comments (29)
  1. Adam Rosenfield says:

    I believe the answer is "I have no idea what it's doing, but whatever it's doing, it's normal."

    blogs.msdn.com/…/661389.aspx

  2. Joshua says:

    Let's just say I'm surprised that didn't screw up cygwin processes.

  3. JJJ says:

    I so enjoy when people sabotage their own questions, especially after putting a lot of effort in them.  The internet form of this phenomenon is "THIS IS URGENT" or "Respond immediately to <some email>" or some variation on the plea letting you know how desperately important it is to them.  And KABOOM, the question is pulverized under the ensuing downvote bombardment.

  4. Krishty says:

    Strange threads can be a pretty bad problem if you raised the stack reserve size of your application. No driver or system service is ever created with a reasonable stack size; they all blindly use the root module's default stack size instead. If you raise it to 100 MiB for your high-performance 3D application, and launch it on a hyper-threaded four-core CPU, then Direct3D and the graphics driver alone will create 16 worker threads you have no control over. Now suddenly there's about 2 GiB of address space wasted for nothing. Add more APIs and strange things will happen, like XAudio taking half an hour for DLL detach.

  5. 640k says:

    "It is important that we know the purpose of this thread." <– not a question?

    Short version:

    Q: Is rogue threads in my process normal?

    A: Nothing to see here, move along!

  6. shut up already says:

    @Karellen What a bunch of whiny crybaby nonsense. How many classic, head-smackingly obvious XY problems has Raymond posted about over the years? It's important to know why the customer wants to know because it's vital to solve the actual problem (whatever probably non-existent issue is actually being caused by this thread) rather than the fake problem (the simple fact that this thread exists). As a Windows developer it is categorically NOT important for you to know every little detail going on under the hood. You should be taking a higher-level view of the system. Knowing how and why the OS is doing everything it does behind your back is how we get software tied to specific OS releases that requires two tons of compatibility shims to function properly a year after its been released.

  7. Danny says:

    @shut up already – GTFO nub.

    @Ray – I wanna know why when I launch Task Manager in Win7 and switch to "Show processes from all users" I have 2 dllhost.exe processes in there that disappear after 2-3 seconds. This happens each time when I launch TM, and is doing this in all flavors for Win7. All of you can check this.

  8. parkrrrr says:

    @Danny – at a guess, those dllhost.exe processes are doing some behind-the-scenes work to get Task Manager elevated. See msdn.microsoft.com/…/ms679687.aspx

  9. Adam Rosenfield says:

    The "nCount = 0" parameter must be a typo, since it makes no sense to wait on 0 handles, and that's explicitly forbidden by the WaitForMultipleObjectsEx documentation.

  10. Tringi says:

    @Krishty – And I do the opposite. For really small apps I sometimes reduce the stack size, just for the sake of it. While I've never gone below 512 kB, some of those injected foreign threads could take my program down :-/

  11. @mh says:

    @parkrrrr, @Danny – and now we're gonna have someone building a program that relies on this behaviour, so when Windows 27 releases and the behaviour changes we'll have yet another broken legacy app.  Sigh.

  12. Fleet Command says:

    "It is important for the customer to know the purpose of this thread."

    Well, I'll be; this actually looks like a piece of translationese. (And since there is a liaison in the middle, no surprise.) But evidently, he was either curious or alarmed. Just put his concern to rest by giving a short explanation that does not betray Microsoft trade secrets. Nitpicking like this, however, has the risk of evolving alarm into paranoia; e.g. "NSA has lodged a backdoor into SP2."

  13. Karellen says:

    @shut up already: "As a Windows developer it is categorically NOT important for you to know every little detail going on under the hood."

    Wow.

    So, because some bad developers can't tell the difference between an API contract and an implementation detail, the answer ought to be "You don't need to know about that, so you are not permitted to know about that. Nothing to see here. Please move along."?

    I guess you didn't get into software development for the joy of it, for the sheer pleasure of figuring out how to build something and building it, for the satisfying "eureka" enlightenment moments you get from discovering how someone else did something, from just learning about how all this complexity goes together.

    "It's important to know why the customer wants to know because it's vital to solve the actual problem"

    The customer didn't appear to have a problem they were trying to solve. They weren't asking "How do I do X" for a strange-sounding value of X. They seemed (to me) to be saying "Our system used to do X, now it does Y for no apparent reason. What's up with that?" Looks like a case of curiosity to me.

    When it comes to curiosity, I prefer the approach that our blog host himself espouses: "we were encouraged not to run away from information, but to run toward it."[0]. That's why I found it strange that the answer to the question "how/why does this happen?" was met with "why do you want to know?", as if simply wanting to know for the sake of knowing isn't a good enough reason.

    [0] blogs.msdn.com/…/10492287.aspx

    [If they were merely curious, then they should have said so. Instead they said that it was vitally important information. (Note also that it is important to know what kind of information you are running toward. Some information will help you solve a problem. Others may cause you to make bad decisions.) -Raymond]
  14. Maurits says:

    Social skills of a thermonuclear device, indeed.

  15. Karellen says:

    "Why does the customer need to know the purpose of this thread?"

    Because the customer is a developer, is curious by nature, and wants to know exactly what's going on in their program. So that when (not if) it breaks, they know what's normal, and why the normal things are the way they are, so they have an idea of things they can initially ignore when hunting the source of their problem.

    If you thought you only had a single-threaded program, and on previous OS releases it was always single-threaded, but the first time you run across a new problem is on this other OS and suddenly there's a whole extra thread in it, well, that to me would be a good indication of where to start looking for problems. How did this extra thread get into my program? What data is it accessing? Is it modifying state that I thought didn't need protecting with a lock, because I thought my program was single-threaded? Has the thread been injected externally? By malware?

    Anyway, having figured out that this is the new normal, and checked the interwebs for any more info and not finding any, I thought I'd ask the people who wrote the OS what the heck is going on. But they wouldn't just tell me, oh no, they first wanted me to justify to them *why* they should tell me what my own program is doing.

    Suddenly, I can't be bothered any more. If MS don't automatically grok "because I just wanted to know" coming from a developer, then I can't be bothered to write the multi-thousand word essay it would take to explain it to them from scratch. If they'd prefer that I live in ignorance, maybe it will be easier that way. And when my client next sends me a bug report, I'll just reply with "I don't know mate – Windows started doing weird things to my app behind my back, and even though I'm a Windows dev, I've no idea what or why, and MS won't tell me. Windows is just unstable like that, I guess."

    [The customer's question wasn't "Is this normal?" It was "It is important that we know the purpose of this thread." Threads can be created for any number of reasons. An API might change to include a cache, or to be asynchronous, or to support a new scenario. This is just something you accept as part of the API contract. (And that thread can't run any application code because that would be a violation of the contract.) Now, if the extra thread was causing a problem, then the question would be "Is there a way to avoid this thread?" -Raymond]
  16. Anon says:

    @Raymond

    I've found, when dealing with MS support, that you are unlikely to actually get anything close to a decent response unless you claim to have an actual problem to solve. That's likely why they were alarmist in asking the question.

    And the response their liaison received after asking why they needed to know was, likely, unprintable.

  17. Antonio &#39;Grijan&#39; says:

    Well, I can understand the customer needing to know what's going on with the new thread, or even being curious. I would Google it, but somebody with full pockets could spend a support ticket in this matter. TRWTF is that the customer dissipated as soon as s/he was asked about his/hers motivations. Thermonuclear, as Maurits said; the customer, of course.

  18. Fleet Command says:

    @Antonio: I am afraid I must say your TRWTF part is just one of the possibilities. An alternative could be that the customer discovered it by himself or met a deadline and ceased investigation. Or, maybe he failed to receive the communication.

    en.wikipedia.org/…/Warnock%27s_dilemma

  19. alegr1 says:

    "I don't understand what those threads are doing, so I will go all TerminateThread on them". Oops, why am I crashing?

  20. alegr1 says:

    >If you raise it to 100 MiB for your high-performance 3D application

    That preposition is so stupid, I don't even know where to begin…

  21. Cheong says:

    @JJJ: That's because [If it's really urgent, you should go for paid support instead of wandering around hope someone happens to know your problem suddenly have interest in playing "ask 20 questions to see what you really want" game.]

  22. Kramii says:

    "Some information will… cause you to make bad decisions." Seriously, Raymond? Sure, a little knowledge is a dangerous thing. But is solution ignorance, or more knowledge?

    [More often than not, the people who ask these sorts of questions are likely to misuse it. Because if you knew how to use the information, you would have been able to figure out the answer on your own: (1) It doesn't matter. (2) It's right there in the function names.) -Raymond]
  23. Neil says:

    Just trying to guess wildly as to why the developer was so interested in the thread, perhaps he was concerned that it was changing the process's exit code, although if the new thread is clever, the main thread is one of the handles it's waiting on, so that if it exits first the new thread can itself exit with the same exit code.

  24. Chris Crowther says:

    I can't help but wonder if this was actually a case of "the customer wants to know if this new thread is normal and, if it is, what is it for?" being passed on as "it is important for the customer to know this" by the customer liaison.  The first part of the original question being the nub of the matter, the second being the first thing I'd ask after being told it was normal, just because I'd like to know.

    I've certainly had things come via support or sales people as "this is really important" only to talk to the customer and find out it was actually just "I was curious".  Customer facing people seem to have a tendency to categorize any customer request as "important".

  25. jas88 says:

    I can imagine a developer being baffled and concerned by this. They've written a single-threaded application – now, suddenly, it seems to have turned into a multi-threaded one! Do they need to switch to multi-threaded libraries and use locking or re-entrant functions everywhere now? (No, because the thread only interacts with its own code not the user's, but I can imagine someone worrying about that.) Or they're debugging some issue in their code, trying to figure out what's going on in the problem case which doesn't apply in their test environment, and the glaringly obvious difference is that the failing case has an unexplained thread floating around in it. If so, they probably found the actual culprit in between asking MS "what's this new thread doing in there?" and getting the non-answer.

  26. rob says:

    Not to mention that any random third party can use CreateRemoteThread to do something in your context such as one case of launching a thread to call ExitProcess.  Or to get a stack trace of a thread (such as SysInternals Process explorer)

  27. Ian Boyd says:

    I'm curious to know that the thread is doing. And we still don't know.

    I assume an updated version of MFC added some telemetry and registered themselves as an ETW provider, and starting generating some events about things interesting to the MFC team. It's unfortunate that the guy who added it isn't talking. And Microsoft, as an entity doesn't want to say ouy loud.

    This is when Charles from Channel 9 goes to visit MFC land, and the developer mentions the interesting or useful stat they included in MFC. Then they mention that anyone can turn on ETW events using the Control Panel, watch the generated events, and see the same cool information. The ETW was useful enough for debugging that it was pushed down to everyone using MFC – obviously it isn't useless.

    I could have figured out the ETW from the stack trace. What is the event provider guid? What event number is being generated. Is the MOF in WMI, or was it a quick addition that only has raw data? Was it ever updated to manifest based tracing after Vista?

  28. Danny says:

    @Miscrosoft Support/Ray/whoever is involved in answering questions – Just answer the God damn questions with an actual response instead of a "Why" question. Wasn't W8 an enough flop that alienated plenty developers that fled to Android/iOS, you really want more to flee due to your poor replies? Keep replying idiotic "Why" back, cause all the dev's you gained in the 90's will go to Android and the new ones that are forming right I bet they are already Android ones. In 10 years you'll have exactly what Apple had in the 90's as developers to do apps – those working inside the company. Stop digging your own grave!!

    [Knowing why they are concerned about the thread will focus the investigation. Otherwise, they are pulling a "Please tell me everything". -Raymond]
  29. Anon says:

    @Danny

    Raymond's "Why do you need to know" was fully justified in context.

    A more Functional response should have been provided — "ETW functionality was extended. Why do you need to know?", but the "Why do you need to know" isn't superfluous.

    Especially when you consider all the ridiculously idiotic requests received by MS, like the "How can we prevent the user from closing our application in any way?"

Comments are closed.