Psychic debugging: Why doesn’t my program show up when I call ShellExecute?


This question came in from a customer (paraphrased):

If I run my program from the command prompt, it works great, but if I run it from my launcher via ShellExecuteEx, it never appears.

See how good your psychic powers are at solving this problem before I give you the second question that gives away the answer.

Any luck?

Here's a second question from a different source (but which coincidentally came in the same day).

I'm trying to use ShellExecute to open a document. The function succeeds (returns a value greater than 32), but I don't get anything on the screen.

if (ShellExecute(Handle, NULL, FileName, NULL,
    NULL, NULL) <= (HINSTANCE)32) ...

The problem the second person is having lies in the last parameter to the ShellExecute function. It's nShowCmd, which is supposed to be an SW_* value, but which this person is passing as NULL. It so happens, that the value zero corresponds to SW_HIDE, which explains why the program doesn't appear: You told it to run hidden!

Now go back to the first problem. Do you see what the person most likely did wrong? The code probably went like this:

SHELLEXECUTEINFO sei = { sizeof(sei) };
sei.hwnd = hwnd;
sei.lpVerb = TEXT("open");
sei.lpFile = pszFile;
ShellExecuteEx(&sei);

Since the sei.nShow member was not explicitly set, the value was implicitly set to zero by the incomplete initializer. And as we noted above, zero means SW_HIDE.

It turns out my psychic debugging was correct. That was indeed the source of the first person's problem. Now you can use your psychic powers, too.

Comments (22)
  1. Andrei says:

    My favourite mistake I have tried to avoid!

    This is the reason to avoid zeroes in enum’s.

  2. Doug says:

    I must assume your customers don’t fit the category of professional programmers.

    Professional: "Hmm, it doesn’t work.  Let’s look at the code and check the functions we are calling."

    Amateur: "Hmm, it doesn’t work.  Must make a support call."

  3. Tim Smith says:

    Doug, you couldn’t be further from the truth.

    Professional: "Hmm, it doesn’t work.  This operating system is so buggy.  Let me try this, I bet that will work."

    Then again, I wouldn’t exactly call those people professionals.  But I run into programmers who never assume it is their fault.

  4. Mike Dunn says:

    I often see this on message boards – folks who use NULL or zero for parameters they don’t (yet) understand, and expect that the API will "do the right thing" and use good defaults. The trouble is, that works in some cases (like the lpDirectory param to ShellExec) but not others (like nShowCmd).

    A quick RTFMsdn would solve the problem, and hopefully the person learns this after getting their answer of "you need to set nShowCmd correctly, see the docs for the legal values".

  5. Problem solved says:

    If SW_SHOW would be 0 and SW_HIDE would be 5 this problem wouldn’t exist. NULL/0 should always be a safe default.

    [I’m surprised. I would have expected that if SW_HIDE were 5 people would say “What a phenomenally stupid idea. The value for SW_HIDE should obviously be zero since zero means FALSE which means ‘hide’ and nonzero means TRUE which means ‘show’ (and the precise nonzero value indicates what type of show). Giving ‘hide’ the bizarro value of 5 just shows what a bunch of moronic developers they have at Microsoft.” -Raymond]
  6. 640k says:

    Both SW_FORCEMINIMIZE and SW_MAX are = 11, so I think the hope of not be treated as a moronic developer is already lost.

  7. GregM says:

    SW_MAX == largest value supported by ShowWindow, for easy range checking.  So yes, it is the same as SW_FORCEMINIMIZE, the largest value.

  8. Igor says:

    If SW_SHOW would be 0 and SW_HIDE would be 5…

    If you just read the MSDN you would know what value to put there instead of NULL and the problem wouldn’t even exist so you won’t have to solve it.

  9. Goran says:

    Because of (possible) problems like the one with SHELLEXECUTEINFO I always wondered should I subclass any old structure and add a couple of constructors to it, only to avoid thinking about what to put in every member it might have every time I use it.

    struct ShellExecuteInfo : public SHELLEXECUTE

    {

     ShellExecuteInfo(LPCTSTR File, LPCTSTR Verb bool show);

     // …

    };

    Yeah, it’s not great, but may be a small improvement ( and a huge pollution of the namespace :-( ). Because I’m not sure about it, I seldom do it (i.e. when I get really fed up with RTFing MSDN).

  10. peterchen says:

    NULL/0 should always be a safe default.

    :wthf:

    SW_* constants are used in many places, not everywhere the same value is the "sensible default".

    It’s your / our job to feed the correct parameters – all of them, not just those that we care about at the moment.

  11. Niklas says:

    Running something in the background is a quite sensible default (you
    start an application in the background for printing a document or
    something).

    It’s easy to make a great handwaving argument about how
    well-designed it is in that he who asks for nothing gets nothing. I
    won’t do that. I think you should know what to put and not to put in
    enumerations.

    [I’m assuming you’re joking.
    Enumerations were not in K&R and did not become part of the C
    language until 1989, six years after the ShowWindow function was
    written. Let me know when you’ve perfected that time machine. -Raymond
    ]
  12. required says:

    Actually there is nothing in the MSDN documentation of ShellExecute which says NULL will hide the application, because the possible values of nShowCmd are not shown, only names.

    It is just another example of an API tripping up a developer by using basic scalar types and assigning meaning to them beyond what can be gathered just by reading a description of the function. Compilers should check such things, based on the types of the arguments, not people.

  13. Neil says:

    If you use image execution options to debug the app startup is the debugger clever enough a) to show even though you specified SW_HIDE b) to pass SW_HIDE on to the app so that when you get to ::ShowWindow(hWnd, nCmdShow); you get to realise that nCmdShow is SW_HIDE?

  14. cmil says:

    "Actually there is nothing in the MSDN documentation of ShellExecute which says NULL will hide the application, because the possible values of nShowCmd are not shown, only names."

    No, the possible values are shown. SW_HIDE and NULL might be equal but they are not equivalent. SW_HIDE is always a legal value while under most circumstances NULL is not. Just because the abstraction isn’t enforced doesn’t mean it can be abused.

  15. Mike Fried says:

    It is too bad that the MSDN doesn’t explicitly state the values of the "enumerations" (or maybe I just couldn’t find them). I was recently poking around the WinMM.DLL methods for MIDI in/out (http://msdn.microsoft.com/library/en-us/multimed/htm/_win32_midi_functions.asp), and in order to get the values and types to put into my C# app (via P/Invoke) I had to open up VC++, type in the value I was looking for, and hit F12 to have VC++ search for the definition.

    My folks sent me my old MIDI controller from high school (for me that was in the mid 90s), and I wanted to hook it up to my tablet via USB and make it use Windows to act as its synthesizer (and maybe to do a little more), so I needed a way to read messages from MIDI in on my USB to MIDI adapter and send the messages to MIDI out on the Windows side. I chose C# instead of my normal C/C++ for fun — I wanted to work with P/Invoke and delegates.

    Needing to use my C++ tools to find out how I needed to marshall my data/types/etc for C# was unfortunate.

  16. 640k says:

    Some enums in windows.h (winbase.h) have a "max" defined which is 1 more than the actual largest value. Conclusion: You can’t trust it.

  17. Roie says:

    These people may have some background in SQL. In SQL, NULL is a valid value for any datatype, distinct from any other value including zero (for some reason, Oracle treats the empty string as NULL, but that’s because Oracle’s weird). It really does make sense, to your average programmer these days (i.e., one that didn’t grow up on K&R and RAM measured in kilobytes).

    In truth, I think the problem stems from the fact that ShellExecute is such an old routine, that hasn’t been superceded (yet).

  18. Good Point says:

    Roie, I would make one small editorial change:

    It really does make sense, to your average [VB] programmer these days

    If you want to use Win32 you should know what NULL is.

  19. Neil says:

    ShowWindow was doomed once it went beyond the simple showing and hiding of windows (including one constant which goes as far as to change your z-order, activate another window according to an undocumented algorithm and trim your working set). Instead ShowWindow should have been left to do what it does well, and all the iconising and maximising should have been given to a separate function.

  20. The real problem says:

    ShowWindow()’s name is to blame for this. To call a function names
    ShowWindow when you really wants to hide a window is plain wrong.

    [A common thread I find is that when people say that X is a bad idea, they rarely say what would be better. -Raymond]
  21. Norman Diamond says:

    > I would have expected that if SW_HIDE were 5

    > people would say “What a phenomenally stupid

    > idea.

    Should’ve used negative numbers (teasing here).

    Theoretically you could use all positive numbers and returned an error code if someone passed a zero.  Obviously it’s too late but this kind of design is reasonable.

    > Enumerations were not in K&R

    … for some language versions of that book.

    > and did not become part of the C language

    > until 1989

    Nothing became part of the C Standard until 1989, but a ton of stuff was added to the language by ATT and others before 1989.  I’m pretty sure ATT added enums while Microsoft was selling Xenix.

    [I can’t believe my eyes. You’re actually saying that Microsoft should have violated the C language standard. -Raymond]

Comments are closed.