The dialog manager, part 4: The dialog loop

The dialog loop is actually quite simple. At its core, it's just

while (<dialog still active> &&
       GetMessage(&msg, NULL, 0, 0, 0)) {
 if (!IsDialogMessage(hdlg, &msg)) {

If you want something fancier in your dialog loop, you can take the loop above and tinker with it.

But let's start from the beginning. The work happens in DialogBoxIndirectParam. (You should already know by now how to convert all the other DialogBoxXxx functions into DialogBoxIndirectParam.)

INT_PTR WINAPI DialogBoxIndirectParam(
    HINSTANCE hinst,
    LPCDLGTEMPLATE lpTemplate, HWND hwndParent,
    DLGPROC lpDlgProc, LPARAM lParam)
  * App hack!  Some people pass GetDesktopWindow()
  * as the owner instead of NULL.  Fix them so the
  * desktop doesn't get disabled!
 if (hwndParent == GetDesktopWindow())
  hwndParent = NULL;

That's right, we start with an app hack. The problem of passing GetDesktopWindow() instead of NULL was discussed in an earlier entry. So many people make this mistake that we had to put this app hack into the core OS. It would be pointless to make a shim for it since that would mean that thousands of apps would need to be shimmed.

Since only top-level windows can be owners, we have to take the putative hwndParent (which might be a child window) and walk up the window hierarchy until we find a top-level window.

 if (hwndParent)
  hwndParent = GetAncestor(hwndParent, GA_ROOT);

With that second app hack out of the way, we create the dialog.

 HWND hdlg = CreateDialogIndirectParam(hinst,
               lpTemplate, hwndParent, lpDlgProc,

Note: As before, I am going to ignore error checking and various dialog box esoterica because it would just be distracting from the main point of this entry.

Modal windows disable their parent, so do it here.

 BOOL fWasEnabled = EnableWindow(hwndParent, FALSE);

We then fall into the dialog modal loop:

 MSG msg;
 while (<dialog still active> &&
        GetMessage(&msg, NULL, 0, 0)) {
  if (!IsDialogMessage(hdlg, &msg)) {

Per the convention on quit messages, we re-post any quit message we may have received so the next outer modal loop can see it.

 if (msg.message == WM_QUIT) {

(Astute readers may have noticed an uninitialized variable bug: If EndDialog was called during WM_INITDIALOG handling, then msg.message is never set. I decided to ignore this fringe case for expository purposes.)

Now that the dialog is complete, we clean up. Remember to enable the owner before destroying the owned dialog.

if (fWasEnabled)
 EnableWindow(hwndParent, TRUE);

And that's all. Return the result.

 return <value passed to EndDialog>;

Congratulations, you are now an expert on dialog boxes. Tomorrow we'll look at how you can put this new expertise to good use.

Exercise: Find a way to sneak through the two layers of hwndParent parameter "repair" and end up with a dialog box whose owner is the desktop window. Explain the dire consequences of this scenario.

Comments (24)
  1. Anonymous says:

    Create a window that is a child of the desktop window then create a dialog from it?

    I just tried that with MessageBox though and nothing particularly nasty seemed to happen,

  2. Anonymous says:

    That’s because Windows XP SP1 (or was it 2?) contains special code specifically to fix people who have this bug. Try it on Windows 2000.

  3. Anonymous says:

    I come to wonder why the GetDesktopWindow() function exists in the first place if we must not use it.

  4. Anonymous says:

    Well, the fix for that is simple. Just reverse the two app hacks, so you find the ancestor first, and then test if it is the desktop window.

  5. Anonymous says:

    It is kind of sad, by the way, that the MSDN page for DialogBox does not mention any requirements at all for the hwndParent parameter. People are hardly to blame for passing the desktop window here.

    Apparently, the Windows programmers have gone through all this trouble to fix applications that pass the wrong window handle, but nobody thought of mentioning this to the MSDN team.

  6. Anonymous says:

    It doesn’t list any special requirements because you’re expected to "follow the ball" and understand that "since this is an owner window, all the standard rules for owner windows apply".

  7. Anonymous says:

    "It doesn’t list any special requirements because you’re expected to "follow the ball" and understand that "since this is an owner window, . "

    While that’s theoretically true, since thousands of developers have made the mistake in reality, maybe ‘theoretically true’ isn’t all that useful, after all.

    API Documentation is hard, and there’s lots to it, I know, but how hard would a cautionary sentence or two be? Then again, now that the mistake’s been made and worked around, maybe there isn’t any point…

  8. Anonymous says:

    I second the motion that something should be mentioned concerning GetDesktopWindow() and CreateDialog(), preferably in one or both of those function entries in MSDN.

    Strangely enough, the GetDesktopWindow() actually refers to the "Initializing a Dialog Box" article which uses GetDesktopWindow() to position the dialog.

    I’m not saying this to point fingers or absolve developers of their responsibility to learn APIs and related concepts, but isn’t more explicit documentation always better?

    Also, the problem with "follow the ball" is that new developers simply don’t understand the important Windows principles and thus make mistakes like are commonly referred to here. I wonder what percentage of Windows developers do actually understand things like this…I wouldn’t care to guess, but I’m sure its disappointing low.

    Good stuff though…keep it up.

  9. Anonymous says:

    What sort of explanatory "let me help you follow the ball" remarks would you recommend? Something like, "Remember that modal dialogs disable the owner window. In particular, passing the desktop window (GetDesktopWindow()) as the owner causes the desktop to be disabled. Disabling a window disables its children. All windows are children of the desktop. Therefore, this disables all windows. You probably didn’t want that." added to all the modal dialog functions?

    What about all the functions that use modal dialogs, such as PropertySheet, GetOpenFileName, ChooseColor, lineConfigDialog, RasPhonebookDlg, … There are thousands of such functions, new ones being written all the time. Should they all have the "Choose your owner window wisely" text? When the RAS team writes documentation for a new API, do they have to "follow the ball" all the way to the bottom for every single system service that their new API uses indirectly?

  10. Anonymous says:

    Well, maybe not in every API help entry, but create a separate "quick reference" guide, containing entries like "Don’t pass GetDesktopWindow() as modal dialog parent", "Repost WM_QUIT message", "Don’t send WM_CLOSE to disabled windows". A tight list of "don’t do that", that’s all

  11. Anonymous says:

    I think just including a link to a page that lists the requirements for windows that own dialogs would be great. And yes, I would include that link everywhere an owner window is passed to a function, even in the documentation for RasPhonebookDlg.

    If you are just trying to open a phone book dialog, you might not be intimately familiar with dialog box construction. And in some way, it seems logical to pass GetDesktopWindow() if you don’t want the dialog to be owned by a specific application.

    Since the documentation for DialogBox does not even mention the fact that NULL is a valid value to pass for the owner window, I guess most programmers who pass GetDesktopWindow() are in fact trying to be careful. Just one extra line of documentation could have avoided all this.

  12. Anonymous says:

    The point would be to document it somewhere, and if it is already documented make it more visible from the relevant entries. CreateDialog() could also point out that a NULL input hParentWnd is valid. I actually find the MSDN documentation very good in general, but that doesn’t mean it cannot be improved.

    I would say some of this stems from the Windows API seeming to be simple at first glance but you actually need a decent grasp of the concepts as mentioned. A new Windows developer sees CreateDialog() take a HWND and GetDesktopWindow() returns a HWND, and all of a sudden you have 100s (1000s?) of programs doing it, big and small alike.

    Is there a solution to this kind of problem? I’d like to think so even I don’t have any easy answers…

  13. Anonymous says:

    I used to create documentation in the "follow the ball" way, but I found out that many of the users of the documentation are klutzes and just couldn’t hold the damn ball. So, I tend to repeat things over and over in the documentation to hammer the do’s and don’ts into the user of the documentation.

    On an aside, I’ve read in many of your posts about "shims". Do you have a post that describes shims and how they work?

  14. Anonymous says:

    It is the the GetMessage and the DispatchMessage that need to be matched. If you dispatch a message via DispatchMessageW to an ANSI window, the thunk layer translate the message from Unicode to ANSI before delivering it to the wndproc. Perhaps you were using a custom thunk layer (for Win95 compat) that didn’t do this translation?

  15. Anonymous says:

    Doesn’t that loop have a bug?

    The issue is this: you have to call the correct GetMessageA/DispatchMessageA vs. GetMessageW/DispatchMessageW depending on whether the hwnd has an ANSI vs. Unicode wndproc.

    We had to fix this repeatedly in private message loops in visual studio (where the plugins/packages were non-uniform in Unicode vs. ANSI/MBCS usage).

    Maybe you only have to be careful about this on certain OSes? The problem is/was that calling GetMessageA/DispatchMessageA on a message to a Unicode window would do the wrong thunking and you’d commonly see garbage characters either when typing non-ascii characters or when displaying them. (It was also a problem calling GetMessageW/DispatchMessageW on a message to an ANSI window.)

    As I recall the correct sequence was to peek the current message without removing it from the queue, check if the hwnd is ANSI vs. Unicode (IsWindowUnicode), then use the appropriate (A vs. W) GetMessage and DispatchMessage.

    I haven’t had to write a message loop in years now so maybe the state of the art has advanced.

  16. Anonymous says:

    Ha!! April Fools everybody! Good one Ray!

  17. Anonymous says:

    "Is there a solution to this kind of problem? I’d like to think so even I don’t have any easy answers… "

    Iterative refinement of the docs (and the API, to the extent possible).

    Even if you can’t identify all the trouble spots in an API ahead of time, the fact that this mistake was made so many times indicates that there’s a fundamental problem "following the ball".

    Text describing the problem in the docs to CreateDialog is a good start. Text in the CreateDialog docs that says something like "other API’s that accept parent window handles to create dialog boxes will have the same problem" is a good follow up.

    Maybe part of what’s at issue here is the role that API documentation plays: is it a simple specification or does it take a more tutorial role, guiding developers to correct usage? If you believe it’s a simple specification, it’s pretty poor, since it doesn’t document all the hidden "App Hack" behaviors. Imagine someone trying to (for whatever reason) literally depend on the documented behavior that passing in the desktop window to a CreateDialog call disables the entire window hierarchy. If you believe that the documentation should specify the API’s behavior, it should specify that desktop window handles in hwndParent are replaced with NULL. If you take the more tutorial stance, then the documentation ought to be at least as forgiving as the API code itself and document the typical mistake and its workaround.

    This blog has done a good job of revealing a little of the complexity involved in making Windows work. That said every workaround of a buggy application changes API behavior in a possibly (likely) undocumented manner. For someone who wants to know what his calls into Windows are doing, that’s unsettling.

  18. Anonymous says:

    "Maybe part of what’s at issue here is the role that API documentation plays: is it a simple specification or does it take a more tutorial role, guiding developers to correct usage?"

    That’s where the API source code comes in. Those few enterprising developers who know how to get their hands on it, live by it in a very practical way, whether in official capacity or under the radar. Then there’s a diminished debate about the purpose or accuracy of API documents, because they merely become abstracts.

  19. Anonymous says:

    I’m curious about the <dialog still active> flag. Obviously this is the same flag that is set by the EndDialog() function; but I’m curious where the actual data for this flag is stored. It can’t be a local variable, or EndDialog couldn’t affect it; and it can’t be static, since you can have nested dialog boxes. So it has to be part of the dialog-box (or window) data itself — or possibly a special posted message — both of which are possible answers given the parameters one passes to EndDialog(). So, Ray, could you elaborate just a little on that flag just to remove the last bit of "magic" from the code?

  20. Anonymous says:

    It’s part of the dialog manager’s internal bookkeeping. And it clearly has to be associated with the dialog itself, since you can have multiple active dialogs at once.

  21. Anonymous says:

    Why is it possible to disable the desktop?

  22. Anonymous says:

    Writing your own dialog loop.

  23. Anonymous says:

    Injecting navigation keys into the dialog modal loop.

  24. Anonymous says:

    Disable the owner window, but make sure it’s really the owner.

Comments are closed.