Quick overview of how processes exit on Windows XP


Exiting is one of the scariest moments in the lifetime of a process. (Sort of how landing is one of the scariest moments of air travel.)

Many of the details of how processes exit are left unspecified in Win32, so different Win32 implementations can follow different mechanisms. For example, Win32s, Windows 95, and Windows NT all shut down processes differently. (I wouldn’t be surprised if Windows CE uses yet another different mechanism.) Therefore, bear in mind that what I write in this mini-series is implementation detail and can change at any time without warning. I’m writing about it because these details can highlight bugs lurking in your code. In particular, I’m going to discuss the way processes exit on Windows XP.

I should say up front that I do not agree with many steps in the way processes exit on Windows XP. The purpose of this mini-series is not to justify the way processes exit but merely to fill you in on some of the behind-the-scenes activities so you are better-armed when you have to investigate into a mysterious crash or hang during exit. (Note that I just refer to it as the way processes exit on Windows XP rather than saying that it is how process exit is designed. As one of my colleagues put it, “Using the word design to describe this is like using the term swimming pool to refer to a puddle in your garden.”)

When your program calls ExitProcess a whole lot of machinery springs into action. First, all the threads in the process (except the one calling ExitProcess) are forcibly terminated. This dates back to the old-fashioned theory on how processes should exit: Under the old-fashioned theory, when your process decides that it’s time to exit, it should already have cleaned up all its threads. The termination of threads, therefore, is just a safety net to catch the stuff you may have missed. It doesn’t even wait two seconds first.

Now, we’re not talking happy termination like ExitThread; that’s not possible since the thread could be in the middle of doing something. Injecting a call to ExitThread would result in DLL_THREAD_DETACH notifications being sent at times the thread was not prepared for. Nope, these threads are terminated in the style of TerminateThread: Just yank the rug out from under it. Buh-bye. This is an ex-thread.

Well, that was a pretty drastic move, now, wasn’t it. And all this after the scary warnings in MSDN that TerminateThread is a bad function that should be avoided!

Wait, it gets worse.

Some of those threads that got forcibly terminated may have owned critical sections, mutexes, home-grown synchronization primitives (such as spin-locks), all those things that the one remaining thread might need access to during its DLL_PROCESS_DETACH handling. Well, mutexes are sort of covered; if you try to enter that mutex, you’ll get the mysterious WAIT_ABANDONED return code which tells you that “Uh-oh, things are kind of messed up.”

What about critical sections? There is no “Uh-oh” return value for critical sections; EnterCriticalSection doesn’t have a return value. Instead, the kernel just says “Open season on critical sections!” I get the mental image of all the gates in a parking garage just opening up and letting anybody in and out. [See correction.]

As for the home-grown stuff, well, you’re on your own.

This means that if your code happened to have owned a critical section at the time somebody called ExitProcess, the data structure the critical section is protecting has a good chance of being in an inconsistent state. (Afer all, if it were consistent, you probably would have exited the critical section! Well, assuming you entered the critical section because you were updating the structure as opposed to reading it.) Your DLL_PROCESS_DETACH code runs, enters the critical section, and it succeeds because “all the gates are up”. Now your DLL_PROCESS_DETACH code starts behaving erratically because the values in that data structure are inconsistent.

Oh dear, now you have a pretty ugly mess on your hands.

And if your thread was terminated while it owned a spin-lock or some other home-grown synchronization object, your DLL_PROCESS_DETACH will most likely simply hang indefinitely waiting patiently for that terminated thread to release the spin-lock (which it never will do).

But wait, it gets worse. That critical section might have been the one that protects the process heap! If one of the threads that got terminated happened to be in the middle of a heap function like HeapAllocate or LocalFree, then the process heap may very well be inconsistent. If your DLL_PROCESS_DETACH tries to allocate or free memory, it may crash due to a corrupted heap.

Moral of the story: If you’re getting a DLL_PROCESS_DETACH due to process termination,† don’t try anything clever. Just return without doing anything and let the normal process clean-up happen. The kernel will close all your open handles to kernel objects. Any memory you allocated will be freed automatically when the process’s address space is torn down. Just let the process die a quiet death.

Note that if you were a good boy and cleaned up all the threads in the process before calling ExitThread, then you’ve escaped all this craziness, since there is nothing to clean up.

Note also that if you’re getting a DLL_PROCESS_DETACH due to dynamic unloading, then you do need to clean up your kernel objects and allocated memory because the process is going to continue running. But on the other hand, in the case of dynamic unloading, no other threads should be executing code in your DLL anyway (since you’re about to be unloaded), so—assuming you coded up your DLL correctly—none of your critical sections should be held and your data structures should be consistent.

Hang on, this disaster isn’t over yet. Even though the kernel went around terminating all but one thread in the process, that doesn’t mean that the creation of new threads is blocked. If somebody calls CreateThread in their DLL_PROCESS_DETACH (as crazy as it sounds), the thread will indeed be created and start running! But remember, “all the gates are up”, so your critical sections are just window dressing to make you feel good.

(The ability to create threads after process termination has begun is not a mistake; it’s intentional and necessary. Thread injection is how the debugger breaks into a process. If thread injection were not permitted, you wouldn’t be able to debug process termination!)

Next time, we’ll see how the way process termination takes place on Windows XP caused not one but two problems.

Footnotes

†Everybody reading this article should already know how to determine whether this is the case. I’m assuming you’re smart. Don’t disappoint me.

Comments (37)
  1. Rob H says:

    Of course I already knew how to determine whether DLL_PROCESS_DETACH was due to process termination…

    But a "friend" who sits "near" my desk wasn’t. He was pretty worried about it at first but he told me that the first Google hit for DLL_PROCESS_DETACH was helpful.

  2. Sohail says:

    At some point will you explain specifically what you don’t like and what you would do differently? It might be instructive.

    [I don’t need that kind of grief. Besides, I don’t have a time machine. -Raymond]
  3. Adam says:

    OK, so thread injection is how you debug processes.

    But in this world of not expecting DLLs to clear up after themselves properly, a process should just terminate all the threads it finds in its own process before it shuts down. Even if it doesn’t, the call to ExitProcess() will forcibly terminate all threads apart from the caller.

    So, if you’re trying to debug process termination you start the debugger, attach it to the process, and shut down the program.

    At which point, the thread the debugger is using to debug the process gets terminated. Um, so how does that debugging situation work?

    Or are you supposed to attach the debugger to the process between the point where ExitProcess() has terminated stray threads and the point where things start to go wrong? On a multi-GHz machine?

    More confused than ever….

  4. S says:

    What improvements or changes have been made to process termination on Vista?

  5. Skywing says:

    ws2_32.dll is a good example of a DLL that detects a detach due to process exit and skips the usual cleanup.

    In order to do this, you should check the lpReserved parameter to DllMain; it’ll be nonzero for DLL_PROCESS_DETACH if you are exiting as part of process rundown, otherwise, the DLL is being unloaded "on-the-fly" a-la FreeLibrary (and in that case, you -should- still perform the usual cleanup, as an "on-the-fly" unload doesn’t imply the weird state of a process that is in the middle of "clean" termination).

  6. Pierre B. says:

    One of the problem you are likely to encounter in the "do nothing" case is that

    of global variables in C++. Those have their destructor called automatically and

    those destructors are called after DllMain() returns (unless you implemented your

    own "true" DllMain, but then you have the reverse problem that the global constructors

    might not be called… (Yes, the DllMain() is not the "true" DLL main, the C/C++ runtime

    has the real entry point for C/C++ DLLs.)

    Moral (which you should obey anyway in all situations for many reasons): never

    use global variables that have a constructor or destructor. Only use simply

    types like int or pointers.

  7. ksurvell says:

    I laughed when I read:

    Instead, the kernel just says "Open season on critical sections!"

    I had a mental image of an Elmer-fudd type figure with a shotgun inside my computer, firing away.

    Still, a fascinating article, nonetheless. I eagerly await part 2…

  8. bramster says:

    Okay, I’m not very smart, but I’ll certainly agree that "Exiting is one of the scariest moments in the lifetime "

  9. Disco Stu says:

    Hey Raymond,

    I’ve been reading your blog for years, despite never writing a lick of windows code in my life. I’ve never bothered to commment before ’cause I never have anything constructive to add.  With that out of the way…

    I just wanted to thank you for continuing to post interesting stuff like this in spite of all the flak you get from griefers, nitpickers, etc about every… little… miniscule detail.

    so, Thanks!

  10. SirCut says:

    Sort of how landing is one of the scariest moments of air travel.

    Actually according to pilots the take-off is when you should be scared.  Flight being one of the safest forms of travel, you should not be scared.  Still if you want to be scared do it on take-off.  Now the OS may be scared when staring a process, but that is a whole other ball of wax.

  11. Pavel Lebedinsky says:

    Behavior with regard to orphaned critical sections during shutdown has changed in Vista:

    http://www.bluebytesoftware.com/blog/CommentView,guid,2838487a-be93-4f6e-afe0-a8acd9de2e11.aspx

  12. In order to do this, you should check the lpReserved parameter to DllMain; it’ll be nonzero for DLL_PROCESS_DETACH

    What part of "reserved" don’t you understand?

  13. Jim Lyon says:

    My favorite way to shut down my process is TerminateProcess(). The kernel handles get closed, but you get absolutely none of this other mess.

    This is part of the pattern "every shutdown is a crash; every startup is a recovery". Assuming your crash recovery code works properly, you don’t have anything to lose (and you don’t have to rely on the author of every DLL in your process having understood the shutdown weirdness).

  14. I remember when writing DirectX/DirectShow apps before, everything was fine and dandy until exit time. That was the scary part, no matter how much you fiddled with the IGraphBuilder, IMediaControl etc. COM-pointers, e.g. calling Release() or not, changing the order, calling CoUninitialize() or not, you couldn’t make the app exit cleanly on all required OS platforms.

    So you’re right, calling ExitProcess() is very much like an airplane landing!

  15. Tom_ says:

    The story behind lpvReserved is surely very simple: it’s doing what it’s reserved for :)

  16. Wang-Lo says:

    "It doesn’t even wait two seconds first."

    Thanks for making me laugh today.

    -Wang-Lo.

  17. Dustin Long says:

    This post reads like the zombie film version of process exiting.

    Awesome.

  18. Brian says:

    The nice thing about landing is if there is a problem, you can throttle up and try again (assuming the engines are still working).  The problem with take-off is when there’s a problem, you’re normally out of runway (ie the disaster in Lexington, KY last year).

  19. Anony Moose says:

    > I had a mental image of an Elmer-fudd type figure with a shotgun inside my computer, firing away.

    I love the “Doom” sys admin “tool” on Linux. Every player runs their own instance of the game. Every running process on the system is represented as a monster in each game. When you shoot a monster, the process gets killed. If it’s another player’s instance they get dumped out. If it’s your own instance, you get dumped out. If it’s a random system process, well, you’ld better not be running on an important server.

    > My favorite way to shut down my process is TerminateProcess(). The kernel handles get closed, but you get absolutely none of this other mess.

    Complete and total inane pathetic nonsense. If you have a thread that’s in a tight loop allocating and releasing memory, there’s a high probability that the thread will be holding a lock while manipulating the heap. Therefore if you terminate that thread by absolutely any means then you will not cause that thread to complete its cleanup process. If the thread has locked the heap and adjusted a pointer and then porcessing switches back to the thread that calls TerminateProcess() then the thread that was manipulating the heap has caused corruption.

    If a thread holds a lock then any application shutdown absolutely and without question must wait until that thread releases the lock. (Modifying shared data without suitable locking is, I assume, so obviously wrong that even you wouldn’t do it.)

    You’re the type of person who writes code that virtually always works, but every now and again some random customer has their entire machine crash after they exit your application. But of course you couldn’t possibly be at fault, could you?

    [Um, you can’t “switch back to the thread that calls TerminateProcess” since the process has been terminated. There is no thread to switch back to. It’s really not that crazy an idea. -Raymond]
  20. Jim Lyon says:

    Anony Moose’s comment about terminated threads leaving things corrupt is a very good demonstration of why TerminateThread() is an abomination.

    But the concern doesn’t apply to TerminateProcess(). It doesn’t matter how corrupt you’ve left the process heap (or other structures) if there’s a guarantee that these structures will never again be used. Having stopped your threads, TerminateProcess() then throws away your entire address space, including your process heap and everything else.

    If Anony Moose’s arguement were correct, consider what would happen if you accidently had a thread that took a lock and then entered an infinite loop. You would not be able to safely kill it using Ctrl-C or TaskManager. (These guys effectively inject a thread into your process that calls TerminateProcess().)

  21. Mark Steward says:

    Really fantastic post, IMHO.

    If TerminateProcess doesn’t work, try tskill (WinStationTerminateProcess).  Somehow, it reaches places other TerminateProcesses can’t.

  22. Leo Davidson says:

    In my opinion, calls to ExitProcess, TerminateThread, etc. are like calls to Sleep: In almost every case they indicate you’re doing something extremely wrong.

    I would not argue for the APIs to be removed, because there are exceptions where they are the least-worst option, but I would be in favour of a tax on calls to the functions in question or a change to the compiler so that it requires an extra argument, consisting of a 500 word essay justifying the call, for each function call.

    A bit like Chris Rock’s idea of making bullets cost $5000 each.

  23. DriverDude says:

    If DirectInput will not be able to re-enable hot-tracking if the process isn’t exited cleanly, isn’t that a DirectInput bug?

    After all, games and apps will exit or crash for any number of reasons. Why can’t DirectInput handle that properly?

  24. dave says:

    >>In order to do this, you should check the

    >>lpReserved parameter to DllMain;

    >>it’ll be nonzero for DLL_PROCESS_DETACH

    Aaron Ballman:

    >What part of "reserved" don’t you understand?

    The part of reserved that *I* don’t understand is where the documentation calls the parameter ‘reserved’ and then tells you what the normal and documented behaviour of this reserved parameter might be.

    Quoth the docset:

    lpvReserved

    [in] If fdwReason is DLL_PROCESS_ATTACH, lpvReserved is NULL for dynamic loads and non-NULL for static loads.

    If fdwReason is DLL_PROCESS_DETACH, lpvReserved is NULL if DllMain has been called by using FreeLibrary and non-NULL if DllMain has been called during process termination.

  25. Norman Diamond says:

    I should say up front that I do not agree

    with many steps in the way processes exit on

    Windows XP.

    Thank you very much for pointing things out like that, it really helps.  Notice how few flames are in the comments, so I’m not the only one who understood  ^_^  I’ll try to avoid wondering if you agree with Win32s or 95 or NT since they all did things differently.  Or if the unwanted thought pops up anyway, I won’t post a question about them.

    If you’re getting a DLL_PROCESS_DETACH due to

    process termination, don’t try anything

    clever. Just return without doing anything

    and let the normal process clean-up happen.

    The kernel will close all your open handles

    to kernel objects.

    Are you sure that the kernel really closes all those handles?  Are you sure that they don’t get rudely terminated with less cleanup of resources than a normal CloseHandle does?  A few minutes ago in response to your other article I mentioned I didn’t blame Windows for a particular resource leak, but now I wonder.

    Thread injection is how the debugger breaks

    into a process.

    Somehow I think it would still be possible for CreateThread to reject a call from within its own process.

    Thursday, May 03, 2007 7:58 PM by KJK::Hyperion

    an unhandled CTRL-C calls ExitProcess

    I thought I read that it injects a thread to do exactly that?

  26. KJK::Hyperion says:

    Jim Lyon: an unhandled CTRL-C calls ExitProcess, and Task Manager never injects threads, a TerminateProcess suffices. On the other hand, I wish it had a "softer" kill where it does inject a thread, that calls *ExitProcess* instead (which cannot otherwise be called remotely, for obvious reasons); a harmless example is how DirectInput will not be able to re-enable hot-tracking if the process isn’t exited cleanly, so you lose hover behavior on standard menu bars until you log off and back on

  27. KNOCKS says:

    very good read! Thanks a ton for a very informative article

  28. Skywing says:

    Dave: The parameter does actually have meaning other than as a boolean flag (at least on NT), but that meaning is not documented.  Hence the `reserved’ part.

  29. Reuven Lax says:

    Ahhh, but you left out one little detail.  IIRC correctly open season is declared on all critical sections except for one — the loader lock.  This means you can still have nasty hangs involving that lock if you’re not careful.

  30. KJK::Hyperion says:

    DriverDude: because DirectInput is a regular DLL, hot-tracking is a per-session UI setting that outlives processes and cannot be "undone", because it’s just a global boolean that processes set on and off?

    The problem is not limited to DirectInput, but shared by any similar library that has to temporarily mess with global UI settings. Haven’t you ever had an old game crash on you and leave your display in 320×200, 16 colors?

    UI tricks is the only scenario where you should really use ExitProcess

  31. KJK::Hyperion says:

    Norman Diamond: process termination closes handles. That is a basic fact of life. No shortcuts are taken – unfortunately! as a stuck CloseHandle might be one thing, a stuck process termination is significantly worse. Also ExitProcess cannot be called on another process, so your other question doesn’t even make sense

  32. Igor says:

    You can always CreateRemoteThread() and ExitProcess() :)

  33. Process Shutdown is evil , as Raymond Chen recently blogged about in wonderful detail. This prompts me

  34. ChrisG says:

    Process refs let apps and the components that create threads within them negotiate a coordinated process exit, enabling everyone that participates the ability to cleanup before ExitProcess() is called. Using this avoids the difficulty that will occur  if you try to do complex stuff in DLL entry points on process shutdown (as your article explains is very problematic).

    • The main thread, the one that will call ExitProcess() enables other threads to extend its lifetime using SHSetInstanceExplorer() with an object that extends its lifetime. This thread will delay its call to ExitProcess() until the ref falls to zero.

    • Threads created in such processes get a reference to the main thread used to extend its lifetime using SHGetInstanceExplorer().

    • SHCreateThread(…,CTF_PROCESS_REF, …) takes a process ref for you (calling SHGetInstanceExplorer()) making it easy for threads to participate in this system.

  35. There was a mail thread that happened recently on one of those "if you aren’t a fulltime employee then

Comments are closed.