Make sure you disable the correct window for modal UI

Some time ago, I was asked to look at two independent problems with people trying to do modal UI manually. Well, actually, when the issues were presented to me, they weren't described in quite that way. They were more along the lines of, "Something strange is happening in our UI. Can you help?" Only in the discussion of the scenarios did it become apparent that it was improper management of modal UI that was the cause.

We already saw one subtlety of managing modal UI manually, namely that you have to enable and disable the windows in the correct order. That wasn't the root of the problems I was looking at, but enabling and disabling windows did play a major role.

When we took a look at the dialog loop, the first steps involved manipulating the hwndParent parameter to ensure that we enable and disable the correct window at the correct time.

 if (hwndParent == GetDesktopWindow())
  hwndParent = NULL;
 if (hwndParent)
  hwndParent = GetAncestor(hwndParent, GA_ROOT);
 HWND hdlg = CreateDialogIndirectParam(hinst,
               lpTemplate, hwndParent, lpDlgProc,
 BOOL fWasEnabled = EnableWindow(hwndParent, FALSE);

In both cases, the first two "if" statements were missing. We already saw the danger of disabling the desktop window, which is what the first "if" statement protects against. But the specific problem with modal UI was being caused by the missing second "if" statement.

Both of the problems boiled down to somebody passing a child window as the hwndParent and the code doing manual modal UI failing to convert this window to a top-level window. As a result, when they did the EnableWindow(hwndParent, FALSE), they disabled a child window, leaving the top-level window enabled.

The two problems had the same root cause but manifested themselves differently. The first problem led to strange behavior because the user could still interact with the top-level window since it was still enabled. Sure, a portion of the window was disabled (the portion controlled by the child window passed as hwndParent), but the caption buttons still worked, as did many of the other controls on the window.

In the second case, disabling the wrong window created a different problem: When the modal UI was complete, the window manager activated the top-level window that was the owner of the modal window since that window was never disabled. This caused the top-level window to receive a WM_ACTIVATE message, which it handled by putting focus on the control that had focus when the top-level window was deactivated. Unfortunately, that window was the window that was passed as the hwndParent, which was disabled by mistake. The attempt to restore focus failed, and when the manual modal UI finally finished up and enabled the child window, it was too late. You wound up with focus nowhere and a dead keyboard. This second problem was reported as simply "SetFocus is not working." Only after peeling back a few layers (and application of some psychic powers) did the root cause emerge.

Now, even though this was a subtle problem, you already knew all the pieces that went into it since I had covered them earlier. And as for those psychic powers that I used? It's really not that magic. In this case of psychic debugging, I worked backwards. In response to the report that SetFocus was not working, the next set of questions was to determine why. Is it a valid window handle? Does the window belong to your thread? Is it enabled?

Aha, the window isn't enabled. That's when the customer also mentioned that they were doing this inside a WM_ACTIVATE handler. If you're gaining activation, who were you gaining it from? Oh, a modal dialog, you say? One that you're managing manually? Once I discovered that they were trying to manage modal UI manually, I suspected that they were disabling the wrong window, since that fit all the symptoms and it's something that people tend to get wrong.

Most of what looks like psychic debugging is really just knowing what people tend to get wrong.

Comments (10)
  1. Cody says:

    So, did you ever consider being a diagnostician instead of a computer scientist?

  2. TV joke says:

    "So, did you ever consider being a diagnostician instead of a computer scientist?"

    I’m sure he did, but it’s much harder to get a cane and a permanent Vicodin "prescription" if only your hand is withered.

  3. JenK says:

    "Most of what looks like psychic debugging is really just knowing what people tend to get wrong."

    See also "intuitive testing".  A lot of "I had a hunch" bugs are based on the sorts of bugs a piece of code tends to attract and/or the types of bugs the tester has seen the dev write in the past.

  4. Bilbo says:

    In 25 years I imagine programmers visiting "The New Old New Thing," a continuation of this blog by one of Raymond’s offspring – Ramona Chen.  Not only will those programmers learn from the knowledge Raymond has passed down to her, they will reap additional benefit because of the natural evolution of the species.

    Instead of a debugging education via stories of Raymond’s clairvoyant feats they will benefit from direct psychic debugging.  Ramona Chen will not only root-out code errors through the use of clairvoyance and remote viewing, she will edit and rebuild the offending code by psychokinesis.

  5. JamesNT says:

    Possible other names for Raymond’s kids:

    1.  Samantha Chen
    2.  Alexander Chen

    3.  Allison Chen

    4.  Marcus Chen

    It should be federal law for Raymond to propogate.  We need a continuous flow of Chen generations to keep the horrible slashdotters from taking over the world.  Think "The Phantom."


  6. Cody says:

    James, the fact that someone would come up with that idea is somewhat disturbing.  As is the mental image.

  7. AC says:

    Somehow I think there will be plenty of Chen’s around for some time to come.

    Not necessarily having anything to do with Raymond.

  8. Neal C says:

    "Most of what looks like psychic debugging is really just knowing what people tend to get wrong."

    Most of what looks like psychic *anything* is just knowing what people tend to want.  You’d be surprised how similar your "psychic debugging" process is to the process used by your nearest "Psychic Readings" shop.  Though, you’ll never get them to admit it.

  9. Barry Leiba says:

    This is really a trackback, but since this doesn’t seem to have trackbacks, here it is as a comment:

    In my blog post here…

    …I say "I’d never thought of the term ‘psychic debugging’, but, yes, it’s a combination of a certain skill and a lot of experience with the root causes of most of the problems," and then I go on to tell a couple of stories from times long past.

  10. Neil says:

    Just to be confusing, I’ve seen a case where hiding a window failed to activate its owner window even though that window was enabled. For the curious, this only happened when the window itself had had a modal dialog; just opening and closing the owned window activated the owner fine.

Comments are closed.

Skip to main content