The alertable wait is the non-GUI analog to pumping messages


When you are doing GUI programming, you well know that the message pump is the primary way of receiving and dispatching messages. The non-GUI analog to the message pump is the alertable wait.

A user-mode APC is a request for a function to run on a thread in user mode. You can explicitly queue an APC to a thread with the QueueUserAPC function, or you can do it implicitly by passing a completion function to a waitable timer or asynchronous I/O. (That's why the code that indicates that a wait was interrupted by an APC is WAIT_IO_COMPLETION: Originally, the only thing that queued APCs was asynchronous I/O.)

Of course, when an APC is queued, the function cannot run immediately. Imagine what the world would be like if it did: The function would interrupt the thread in the middle of whatever it was doing, possibly with unstable data structures, leaving the APC function to run at a point where the program is in an inconsistent state. If APCs really did run this way, it would be pretty much impossible to write a meaningful APC function since it couldn't reliably read from or write to any variables (since those variables could be unstable), nor could it call any functions that read from or wrote to variables. Given these constraints, there isn't much left for a function to do.

Instead, APCs are dispatched when you perform what is known as an "alertable wait". The "Ex" versions of most wait functions (for example, WaitForSingleObjectEx, WaitForMultipleObjectsEx, MsgWaitForMultipleObjectsEx, and SleepEx) allow you to specify whether you want to wait alertably. If you do wait alertably, and one or more APCs are queued to the thread, then all the pending APCs are run and the wait operation returns with a code indicating that the wait was interrupted by an APC. If the APC you are waiting for has not yet run (maybe you were interrupted by some unrelated APC), then it is your responsibility to restart the wait and try again.

Why doesn't the operating system automatically restart the wait? "Imagine what the world would be like if it did": Suppose you want to issue asynchronous I/O and then go off and do some other stuff, and then wait for the asynchronous I/O to complete so you can use the result.

// When an asynchronous read completes, we fire off the next
// read request. When all done, set fCompleted.
BOOL fCompleted = FALSE;
BOOL fSuccess;
void CALLBACK CompletionRoutine(DWORD, DWORD, LPOVERLAPPED)
{
 if (<finished>) {
  fSuccess = TRUE;
  fCompleted = TRUE;
 } else {
  // issue the next read in the sequence
  if (!ReadFileEx(hFile, ..., CompletionRoutine)) {
   fSuccess = FALSE; // problem occurred
   fCompleted = TRUE; // we're done
 }
}
...
// start the read cycle
if (ReadFileEx(hFile, ..., CompletionRoutine)) {
  DoOtherStuffInTheMeantime();
  <wait for fCompleted to be set>
  DoStuffWithResult();
}

How would you write the "wait for fCompleted to be set" if the operating system auto-restarted waits? If you did an alertable infinite wait, say with SleepEx(INFINITE, TRUE), then the APCs would run, the operating system would auto-restart the waits, and the sleep would just run forever. You would be forced to use a non-INFINITE sleep and poll for the completion. But this has two serious flaws: One is that polling is bad. The second is that the rate at which you poll controls how quickly your program reacts to the completion of the read chain. Higher polling rates give you better responsiveness but consume more CPU.

Fortunately, waits are not auto-restarted. This gives you a chance to decide for yourself whether you want to restart them or not:

...
// start the read cycle
if (ReadFileEx(hFile, ..., CompletionRoutine)) {
  DoOtherStuffInTheMeantime();
  while (!fCompleted) SleepEx(INFINITE, TRUE);
  DoStuffWithResult();
}

The SleepEx loop just keeps waiting alertably, processing APCs, until the completion routine finally decides that it's had enough and sets the fCompleted flag.

Comments (20)
  1. Anonymous says:

    I still don’t see why WaitForSingleObjectEx needs to be aborted by an APC.  If I wait on an event, why would I care that another APC was delivered?   I can signal the event from my APC

  2. KJK::Hyperion says:

    Well well well, APCs. My favorite topic! Why Raymond, of course APCs could be made to interrupt a thread whenever… if you provide a way to block them temporarily. Like KeEnterCriticalRegion/KeLeaveCriticalRegion in kernel mode Windows, or like POSIX signals. But right now is way too late to introduce asynchronous signals. You’d run with asynchronous signaling disabled all the time in fear that some old or 3rd party code wouldn’t take it nicely

    And you may or may not know that Win32 does restart waits in certain cases (specifically, when a wait is interrupted by an alert) – all the Ex wait functions loop on while(Status == STATUS_ALERTED). Altough in those cases no user code is executed, so it’s harmless

    [while we are on the topic of APCs, a bit of obscure trivia for the hardcore fanboys like me: the Windows kernel has explicit support for POSIX signals. Specifically, a POSIX signal would be a "special user-mode APC" ("special" meaning it has no NormalRoutine, and "user-mode" meaning it will only interrupt user-mode waits), where the KernelRoutine messes with the user-mode stack to arrange for the signal dispatcher to be called. KeInitializeApc won’t let you create a "special user-mode APC", but if you tweak the KAPC by hand before calling KeInsertQueueApc you can get one. A hackish, user-mode only implementation is possible with normal user-mode APCs, and I think that’s what SUA currently uses, but the special user-mode APC method yields the best performance]

  3. Anonymous says:

    I never really understood why APCs could not be made to interrupt a thread whenever…

    Back in my DEC VMS days, (going back quite a few years of course), I remember a similar concept, the Asynchronous System Trap (AST).  When an AST was setup to trigger, perhaps based on a timer, it would immediately interrupt the current process.  Of course we didn’t have multiple threads, but we counted on the ability to interrupt the main thread whenever required.  

    When I started with Windows, I was quite surprised that APCs could not interrupt a thread whenever necessary.  Since quite a few of the Windows concepts seemed to mirror VMS so closely, I assumed that an APC would function similarly to an AST.

     

  4. mgrier says:

    Re: ASTs:

    ASTs were such a pain because nobody could really maintain the discipline about what to do and not to do in an AST and how to synchronize the AST with the non-AST-level code.  (Instead of annotating the waits which could be interrupted you had to temporarily diable and then reenable ASTs around critical sections of your non-AST-level code.)

    APCs are a pain because everyone has to remember to do alertable waits and since APC delivery doesn’t block APC delivery, you probably do NOT want to make alertable wait calls from within your APC.

    Unless you do in which case you’re into the same kinds of reentrancy hazards as exist in private message pumps.

  5. dave says:

    Imagine what the world would be like if it did

    Well, it would be like RSX-11M, or VAX/VMS (which are both Dave Cutler operating systems, of course).

    Though, seriously, when programming on those operating systems (which lacked thread support), one often adopted an event-driven model in which all work is done in AST handlers (=APC routines). Thus, in practice, you were only ‘alertable’ at well-defined points, when you exited the AST.

    I suspect the real reason that user-mode APCs can only be deliverable at well-defined points is that programming is vastly more complicated than when I used RSX-11M.

    Consider if an APC could interrupt some window procedure – and the APC then issued some call which would need to run the window procedure. Eek.

    (And things like mutexes don’t help you – they’re used for inter-thread locking, not intra-thread locking. An APC can’t use a mutex to synchronize with activity in the same thread at non-APC level.)

  6. Gabe says:

    True asynchronous notification (like Unix signals, DOS TSRs, or VMS ASTs) is not very useful because you can’t do much with it beyond setting a flag to be checked at set points where your program may be safely interrupted. If you are checking a flag to see if an interrupt occurred, you could just as easily check for APCs instead.

    Most things you would want to do, like memory allocation, floating point math, buffered IO, GUI operations, and much more just cannot be done safely because you may be interrupting the libraries while they’re running. You have to be very careful how you modify any data structures that might be read in your handlers.

    Of course, you can always just disable the interrupts when they would be inconvenient. You could wrap all memory allocations or deallocations in disable/enable functions for any handler that allocates memory, for example. Once you’ve done that, though, you have specified all of the points where interrupts are OK, so you could just check for APCs then.

    If you truly want asynchronous notification, you can just spawn off a thread that does {while (1) SleepEx(INFINITE, TRUE);}. Then your APCs can block on mutexes or critical sections so that whatever’s being interrupted can complete before letting the APC have the resource.

  7. Chris Becke says:

    When I first discovered Win32 APCs I got rather excited by the prospects of (finally) making a clean inter thread signalling mechanism.

    Then I discovered that GUI thread pump functions – i.e. GetMessage – do not internally use alertable waits.

    Oh well. Back to posting messages to windows. erk.

  8. shread threads says:

    Why even try to get apis thread safe? All threaded apps are buggy anyway, just take a look at explorer.

  9. dave says:

    All threaded apps are buggy anyway

    That’s just an uninteresting special case of the theorem ‘all apps are buggy’.

    But nevertheless, we still write programs.

  10. Gabe says:

    Chris, GetMessage can’t be alertable because your program may not be in a consistent state for your APC to run. Of course it doesn’t really matter because you can just use MsgWaitForMultipleObjectsEx with an alertable wait. If it returns telling you that there’s a message available, call GetMessage.

  11. Mike says:

    While Explorer indeed is buggy, and does neither understand nor use threading, and indeed frequently screws up (severely) when faced with mounted/dismounted driveletter (hell, even adding adding a file can make it screw up), it has nothing to do with neither APC’s nor threading. Explorer is implemented by VB-fags that think the world is a DOS-box. The shell-views added on top of it is just another layer som idiot added to make it even more plausible things would get out of synch.

    But to extrapolate from those Microsoft designers and developers incompetence to say "X = shit" is false. Think BeOS. Everything in the tracker ran inside threads. Responsive as nothing else. Never screwed up due to SHIT EMIT^WDs.

  12. Gabe says:

    It’s pretty easy to verify that Explorer is multithreaded. Simply open two Explorer windows in the same process. Do something that makes a window stop responding (like trying to access a nonexistent network share), and you’ll see that the other window is still perfectly responsive.

    You could argue that Explorer should have ALL of its IO done on a separate thread, but it was written over 12 years ago. Back then most GUIs either had a single message queue and/or no threading, so there wasn’t a lot of history to show why the current implementation is lacking. I’ve heard Explorer has been rewritten for Vista, so maybe they’ve moved all IO out of the GUI thread.

    Of course any bugs that may be in Explorer are almost undoubtedly not due to its being multithreaded, and I doubt it makes much use of APCs.

  13. Chris Becke says:

    Gabe Wrote: "Chris, GetMessage can’t be alertable because your program may not be in a consistent state for your APC to run."

    If a GUI thread is ever in a consistent state for asynchronous code to run, its when the GUI thread is stuck inside GetMessage. Hell, another thread can do a SendMessageXXX call to trigger the execution of code by GetMessage.

    As it stands, user APCs are all but useless for use with most application frameworks as most programmers, who might otherwise use user APCs inside some component of a project – cannot, as their ownership of a project extends only over their own component, not their projects framework.

  14. Ian Boyd says:

    i have no idea what you’re talking about.

    http://tinyurl.com/rwrnt

    And i don’t think i ever will.

  15. Norman Diamond says:

    WaitForMultipleObjectsEx,

    > MsgWaitForMultipleObjectsEx,

    > and SleepEx

    Why is there no sEx version of Sleep?????  What a boring system.

    Wednesday, May 03, 2006 4:33 PM by mgrier

    > ASTs were such a pain because nobody could

    > really maintain the discipline about what to

    > do and not to do in an AST and

    True, but I never did find any documentation of what kind of discipline needed to be maintained.  By now I’ve forgotten most details, but have a vague feeling that I could only rely on examining a few items and deciding whether or not to set an event flag.

    Wednesday, May 03, 2006 5:37 PM by dave

    > one often adopted an event-driven model in

    > which all work is done in AST handlers

    And then often found that it didn’t work, right?  It looks like you suffered from the same lack of documentation as I did.

    By the way that is one of approximately four ways in which I found documentation at the old DEC to be severely lacking.  (This refers only to the reference text.  The code examples were pretty much the opposite, i.e. it was rare to find an example that would work as written.)

    Thursday, May 04, 2006 11:35 AM by dave

    >>All threaded apps are buggy anyway

    >

    > That’s just an uninteresting special case of

    > the theorem ‘all apps are buggy’.

    That’s just an uninteresting special case of the theorem "all programs are buggy".

    (Though some of us do fix bugs in our kernels, drivers, interrupt handlers, memory managers, AND apps when we become aware of the bugs.)

  16. KJK::Hyperion says:

    Explorer locks up because the New Shell predated OLE 2. New Shell needed OLE 2, but it wasn’t done yet, so it got its own "mini-OLE" that supported only a handful of features, and especially only supported single-threaded apartments (STAs). Raymond blogged about it, look in the archives

    OLE 2 had multithreaded apartments too (which aren’t as multithreaded as you may think), and later (around Windows 2000) got free-threaded and thread-neutral components as well (which are truly multithreaded)… but the shell still only supports STAs, and will in fact refuse to load free-threaded or thread-neutral components altogether

    What the shell people did to work around that and get at least a bit of multitasking is creating multiple threads (typically one for each shell view, i.e. one for the desktop and one for each folder window), each with its own STA. Thread pools ("shell tasks") and a tiny bit of overlapped I/O support were introduced later, but the bulk of it stays anchored to the old model

    And besides, I have seen far far worse. At least the shell GUI is reentrant, if not multithreaded, so you can at least do the "multi-single-threaded" trick. Most other toolkits don’t have it that easy: Visual Basic’s GUI library is notoriously single-threaded, as is Borland’s VCL, and you can easily verify with a slow enough network share that Mozilla has a single-threaded GUI as well (attaching a slow network file will, in my experience, completely stall Thunderbird)

  17. Gabe says:

    KJK, are you sure about your history there?  OLE2 was originally created for Office 4.0, which was 16-bit, but shipped with NT 3.1 in 1993 in 32-bit form.

    Are you saying that Explorer was sufficiently advanced by the time NT 3.1 RTMed that OLE2 could not be incorporated into it?

  18. Dean Harding says:

    Gabe, KJK::Hyperion: Actually, Raymond has talked about this before. It wasn’t because OLE2 wasn’t "sufficiently advanced" enough, it was because it needed too much memory, and Windows 95 needed to run in 4MB. So they basically didn’t load OLE32.dll until *absolutely nessecary* – which meant that the shell basically duplicated just the small bits that it needed.

    See: http://blogs.msdn.com/oldnewthing/archive/2004/07/05/173226.aspx

  19. Bryan says:

    mgrier made this very good point:

    > you probably do NOT want to make alertable wait calls from within your APC

    What surprised me when working with APCs was that GetUserName() enters an alertable wait state, so another APC can be delivered, and hence my code was unexpectedly re-entrant.

    Given that APIs like GetUserName() don’t document whether or not they can enter an alertable wait state, the only safe approach seems to be to do as little as possible in an APC.

Comments are closed.