Returning values from a dialog procedure

For some reason, the way values are returned from a dialog procedure confuses people, so I’m going to try to explain it a different way.

The trick with dialog box procedures is realizing that they actually need to return two pieces of information:

  • Was the message handled?
  • If so, what should the return value be?

Since two pieces of information have to be returned, but a C function can have only one return value, there needs to be some other way to return the second piece of information.

The return value of the dialog procedure is whether the message was handled. The second piece of information – what the return value should be – is stashed in the DWLP_MSGRESULT window long.

In other words, DefDlgProc goes something like this:

    HWND hdlg, UINT uMsg, WPARAM wParam, LPARAM lParam)
    DLGPROC dp = (DLGPROC)GetWindowLongPtr(hdlg, DWLP_DLGPROC);
    SetWindowLongPtr(hdlg, DWLP_MSGRESULT, 0);
    BOOL_PTR fResult = dp(hdlg, uMsg, wParam, lParam);
    if (fResult) return GetWindowLongPtr(hdlg, DWLP_MSGRESULT);
    else ... do default behavior ...

If you return anything other than 0, then the value you set via SetWindowLongPtr(hdlg, DWLP_MSGRESULT, value) is used as the message result.

For example, many WM_NOTIFY notifications allow you to override default behavior by returning TRUE. To prevent a listview label from being edited, you can return TRUE from the LVN_BEGINLABELEDIT notification. But if you are doing this from a dialog procedure, you have to do this in two steps:

    SetWindowLongPtr(hdlg, DWLP_MSGRESULT, TRUE);
    return TRUE;

The second line sets the return value for the dialog procedure, which tells DefDlgProc that the message has been handled and default handling should be suppressed. The first line tells DefDlgProc what value to return back to the sender of the message (the listview control). If you forget either of these steps, the desired value will not reach the listview control.

Notice that DefDlgProc sets the DWLP_MSGRESULT to zero before sending the message. That way, if the dialog procedure neglects to set a message result explicitly, the result will be zero.

This also highlights the importance of calling SetWindowLongPtr immediately before returning from the dialog procedure and no sooner. If you do anything between setting the return value and returning TRUE, that may trigger a message to be sent to the dialog procedure, which would set the message result back to zero.

Caution: There are a small number of “special messages” which do not follow this rule. The list is given in the documentation for DialogProc. Why do these exceptions exist? Because when the dialog manager was first designed, it was determined that special treatment for these messages would make dialog box procedures easier to write, since you wouldn’t have to go through the extra step of setting the DWLP_MSGRESULT. Fortunately, since those original days, nobody has added any new exceptions. The added mental complexity of remembering the exceptions outweigh the mental savings of not having to write one line of code (“SetWindowLongPtr(hdlg, DWLP_MSGRESULT, desiredResult)”).

Comments (15)
  1. Anonymous says:

    Raymond, what’s the best way to ask you questions that aren’t necessarily about the topic you are posting on? I’m a little hesitant to post questions in the comment section.

  2. Raymond Chen says:

    Just post them into the comments. Note that this is not a product support blog, so I only answer questions I feel like answering.

  3. Anonymous says:

    Are you implying that there are some product support blogs?

  4. Jack says:

    "For some reason, the way values are returned from a dialog procedure confuses people"

    Well one reason is because the DialogProc API is completely different from a nearly identical WindowProc API. If there were consistency in the API design, then there would be a DefDialogProc to be called and returned for unhandled dialog functions, and regular ones could return an LRESULT.

    It doesn’t help that you’re allowed to return non-bool values because BOOL isn’t an enum but is just an int.. Et cetera, et cetera.

    Anyway, there’s plenty of reasons the mistakes happen.

  5. Raymond Chen says:

    You can do the DefDlgProc thing instead of a dialog procedure if you prefer the WndProc model for dialogs.

    When I wrote "this is not a product support blog" I meant that this blog is not for product support. I have no idea if there exist blogs for product support. (I wouldn’t think so, since a blog is really not suited to Q&A back-and-forth communications. It’s much more a one-way medium with a little bit of feedback.)

  6. Jack says:

    No, what I mean is for a window proc you do this:

    function myWindowProc
    if ( msg == WM_SOMEMSG )

    return myValue;

    return DefWindowProc(…);

    now if dialogs worked this way

    function myDialogProc
    if ( msg == WM_SOMEMSG )

    return myValue;

    return DefDlgProc(…);

    rather than…

    function myDialogProc
    if ( msg == WM_SOMEMSG )

    SetWindowLong( …, myValue );
    return TRUE;

    return FALSE;

    then it wouldn’t be confusing. Because there’s that disparity, there’s confusion.

  7. Anonymous says:

    My question (from first comment) is about window position persistence.

    It seems like a fairly common thing for an app developer to write. So what, it must have been done a couple of thousand times .. why doesn’t windows (explorer) do this for me?

    Would it be too much effort for an app’s window position to be remembered by Explorer instead of everyone having to re-write this code everytime they write a new application.

    Sure there are likely to be some apps that don’t want/need this but it wouldn’t be too hard for a developer to turn this off with a one-liner at the start of their app?

    It’s not like it’s hard but it’s one more place to introduce bugs ..

    It’s not necessarily Raymond’s answer I am interested in, just his opinion :)

  8. Centaur says:

    Not Explorer! Because this would mean this feature would not work when programs are started from other shells.

  9. Anonymous says:

    Other shells? Like?

  10. Mike says:

    Windows 95 was supposed to be doc-centric IIRC (Office ceased to be MDI around that time), so the cache key for the window position cache would have been the document identifier, not the app identifier. But some apps don’t really do documents.

    I would agree that shells should have policies about things like whether when you reopen "the same window" it remembers the position. The OS should NOT have a policy about this.

    I would agree that it should be easier to slap another shell on Windows, too.

    Finally, the way DialogProcs work works very nicely for 99.999% of dialogs. This should be kept in mind when critiquing their inconsistency with WindowProcs.

  11. Jack says:

    Why should it be kept in mind? At the best case, it’s the same number of lines of code… So a

    return FALSE;

    would become at

    return DefDlgProc( … );

    more typing, but the same number of lines. And that’s only the best case. I’m not really sure who it’s helping.

  12. Anonymous says:

    Mike Said: I would agree that shells should have policies about things like whether when you reopen "the same window" it remembers the position. The OS should NOT have a policy about this.

    Which is exactly my point. I can’t think of a sane reason for Explorer *not* to do this, and I’m hoping someone else can. I’ve no idea what other shells are out there and who in general would/does use them, at the moment how those would react to this is probably the only problem (as far as I can see – which admittedly is not far).

  13. Raymond Chen says:

    In my opinion, window positioning is something for the app to handle. And it’s certainly not Explorer’s job – what if Explorer isn’t running?

  14. Anonymous says:

    And just to add to the confusion WM_COMPAREITEM ignores DWL_MSGRESULT and uses the return value of the dialog proc. Go figure.

  15. Anonymous says:

    Oh wait you already said that, next time I’ll try not to be retarded…

Comments are closed.