What can I do with the HINSTANCE returned by the ShellExecute function?


As we saw earlier, in 16-bit Windows, the HINSTANCE identified a program. The Win32 kernel is a complete redesign from the 16-bit kernel, introducing such concepts as "kernel objects" and "security descriptors". In particular 16-bit Windows didn't have "process IDs"; the instance handle served that purpose. That is why the WinExec and ShellExecute functions returned an HINSTANCE. But in the 32-bit world, HINSTANCEs do not uniquely identify a running program since it is merely the base address of the executable. Since each program runs in its own address space, that value is hardly unique across the entire system.

So what can you do with the HINSTANCE returned by the ShellExecute function? You can check if it greater than 32, indicating that the call was successful. If the value is less than 32, then it is an error code. The precise value of the HINSTANCE in the greater-than-32 case is meaningless.

Why am I bothering to tell you things that are already covered in MSDN? Because people still have trouble putting two and two together. I keep seeing people who take the HINSTANCE returned by the ShellExecute function and hunt through all the windows in the system looking for a window with a matching GWLP_HINSTANCE (or GWL_HINSTANCE if you're still living in the unenlightened non-64-bit-compatible world). This doesn't work for the two reasons I described above. First, the precise value of the HINSTANCE you get back is meaningless, and even if it were meaningful, it wouldn't do you any good since the HINSTANCE is not unique. (In fact, the HINSTANCE for a process is nearly always 0x00400000, since that is the default address most linkers assign to program executables.)

The most common reason people want to pull this sort of trick in the first place is that they want to do something with the program that was just launched, typically, wait for it to exit, indicating that the user has closed the document. Unfortunately, this plan comes with its own pitfalls.

First, as we noted, the HINSTANCE that you get from the ShellExecute function is useless. You have to use the ShellExecuteEx function and set the SEE_MASK_NOCLOSEPROCESS flag in the SHELLEXECUTEINFO structure, at which point a handle to process is returned in the hProcess member. But that still doesn't work.

A document can be executed with no new process being created. The most common case (but hardly the only such) in which you will encounter this is if the registered handler for the document type requested a DDE conversation. In that case, an existing instance of the program has accepted responsibility for the document. Waiting for the process to exit is not the same as waiting for the user to close the document, because closing the document doesn't exit the process.

Just because the user closes the document doesn't mean that the process exits. Most programs will let you open a new document from the "File" menu. Once that new document is opened, the user can close the old one. (Single-document programs implicitly close the old document when the new one is opened.) What's more, closing all open windows associated with the document need not result in the program exiting. Some programs run in the background even after you've closed all their windows, either to provide some sort of continuing service, or just because they are just anticipating that the user will run the program again soon so they delay the final exit for a few minutes to see if they will be needed.

Just because the process exits doesn't mean that the document is closed. Some programs detect a previous instance and hand off the document to that instance. Other programs are stubs that launch another process to do the real work. In either case, the newly-created process exits quickly, but the document is still open, since the responsibility for the document has been handed off to another process.

There is no uniform way to detect that a document has been closed. Each program handles it differently. If you're lucky, the program exposes properties that allow you to monitor the status of an open document. As we saw earlier, Internet Explorer exposes properties of its open windows through the ShellWindows object. I understand that Microsoft Office also exposes a rather elaborate set of automation interfaces for its component programs.

Comments (17)
  1. Damit says:

    This seems like a plus point for "everything is a file" or something like AppleScript.

    I’m curious, will there ever be any kind of standardization (whether by Microsoft or a third party) in terms of the way programs should respond to ShellExecute?

  2. Tim says:

    Out of interest, I checked the MSDN page for ShellExecute to see if it was ‘obvious’, or ‘obvious if you’re Raymond’ :-), and it fell firmly into the first camp, specifically saying that numbers above 32 are essentially meaningless and it’s not a proper instance.  Which is nice.

    On that subject, can you prod the DirectX team to have a similar level of rigourousness in their docs?

    For example, one of the cases I came across in the past week or so, SetStreamSource():

    http://msdn.microsoft.com/library/default.asp?url=/library/en-us/directx9_c/IDirect3DDevice9__SetStreamSource.asp

    Given you can have multiple streams, and that streams can be deleted and therefore their interface pointers are no longer valid, how would you tell the Direct3D device that, e.g. stream 1 should be detached now?  Using that documentation, I mean.  And following your own rules about not using undocumented parameter values, etc.

    (If you care, the correct answer is to pass in a NULL pointer as the stream – nowhere is this definitively documented; it’s only mentioned in passing by being in some sample code elsewhere in the docs, and no reference is made in the accompanying text to the NULL value of the stream interface pointer.  And knowing how reliable sample code can be in terms of correctness, this is the kind of thing that can drive you nuts when dealing with DirectX).

    And I know you have no power whatsoever to control the DX team’s behaviour – I’m just making a point that sometimes people assume things about APIs because so many times in the past it’s the only way they’ve been able to accomplish anything, if you see what I mean.

  3. KJK::Hyperion says:

    Damit: OLE was proposed, but it didn’t catch on. OLE lets you, among many many other features, query globally all the opened documents in all running applications, and it’s how Explorer can tell you what application keeps a document open so that it cannot be moved/deleted/etc. (of course, since it mostly only works with Office, you’d be inclined to think it’s evil Microsoft and their evil hidden APIs)

  4. KJK::Hyperion says:

    Tim: use the feedback link at the bottom. I swear it works

  5. Damit:

    There is a good regression test for ShellExecute here. It should catch most behavior changes in programs from one version of windows to another.

    http://cvs.winehq.org/cvsweb/wine/dlls/shell32/tests/shlexec.c?rev=1.9&content-type=text/x-cvsweb-markup

  6. PatriotB says:

    KJK::Hyperion:  Ah yes, the Running Object Table (ROT)!  It would be nice if the Windows Shell documentation would remind people that putting their open files in the ROT will get them the "nicer" file-in-use handling.

    It would also be nice if the SDK team would stop dropping valuable tools from the SDK, such as IROTVIEW which let you see all the objects currently registered in the ROT.

  7. KJK::Hyperion says:

    Norman: just ditch ShellExecute and use ShellExecuteEx, then. Has nicer error handling and generally nicer everything. As for INT_PTR, I’ll just say beware, beware sign extension

  8. KJK::Hyperion says:

    and, Norman, the result cannot "happen" to be larger than that. Rigorousness OK, but it doesn’t take a lot of psychic skills to understand what goes on behind the curtain. The return value is obviously born as a positive int and casted into an HINSTANCE with IntToPtr(). And int is always 32 bit, even on Win64, so there’s no kind of trickery that could make it overflow

  9. Norman Diamond says:

    > You can check if it greater than 32,

    > indicating that the call was successful. If

    > the value is less than 32, then it is an

    > error code.

    Your employer disagrees with you, very firmly.

    The MSDN page says to cast it to an int and then compare the resulting value to 32.  So for example if the value of the HINSTANCE happens to equal 2**33 + 1 then you cast it to an int, look at the result of 1, and conclude that the HINSTANCE value represented an error.

    Last month I posted in Microsoft’s public newsgroup on kernel issues, asking if maybe programs should cast the HINSTANCE to INT_PTR instead of int.  Your colleagues kindly, politely, and firmly provided proof that no, MSDN is right, it should be cast to int.

    > [some defective programs] looking for a

    > window with a matching GWLP_HINSTANCE (or

    > GWL_HINSTANCE if you’re still living in the

    > unenlightened non-64-bit-compatible world).

    Yup, well I tried to be enlightened and 64-bit aware, but I was wrong.  INT_PTR is correct SOMETIMES.  (Of course the primary defect that you described in those programs is somewhat worse.)

  10. bviksoe says:

    Oh no, you said: DDE.

    And here I thought you weren’t allowed to swear or use obscene words on this blog! Well, good thing the topic was filed under Historic. So many bad memories…

    Could it be that Windows Explorer is the only application on the face of this earth to use this technology still?

  11. oldnewthing says:

    It’s okay to cast to an int because, as both the linked article and the documentation for ShellExecute note, the error code is an integer, cast to HINSTANCE for compatibility. The return value will never be 2**33+1 since that doesn’t fit in an int.

    When I wrote, "You can check if it is greater than 32," I mean "You can check if it is greater than 32 after casting it to an int" – but I assumed that was obvious since I’m assuming people reading the entry are already familiar with ShellExecute. Do I have to spell out very last excruciating detail? Do you spell out every last excruciating detail when you talk with your friends?

    "When you get to the gas station, turn left."

    "Even if the traffic light is red?"

    "Sigh. Turn left *when safe and legal*."

  12. To go even more off topic.

    bviksoe: Some programs that try to be compatible on the oldest Windows version possible such as Win9x still use DDE. I tracked down a bug in firefox recently that was causing a hang during DDE communication.

  13. Norman Diamond says:

    When I wrote, "You can check if it is greater

    > than 32," I mean "You can check if it is

    > greater than 32 after casting it to an int"

    > […] Do I have to spell out very last

    > excruciating detail?

    In this case there was a reason.  In the base note you also wrote this:

    > GWLP_HINSTANCE (or GWL_HINSTANCE if you’re

    > still living in the unenlightened non-64-bit-

    > compatible world).

    As stated, I thought it might be appropriate to be 64-bit aware and cast the HINSTANCE to an INT_PTR instead of to an int.  Your employer said no, and here you agree with your colleague, but it’s still not completely obvious.  It is good that MSDN is literally correct here.  In fact if MSDN were literally correct in enough other places then I wouldn’t have wondered whether to doubt this one.  But your base note didn’t say int, and any 64-bit-aware person who reads your base note might not be aware that they should only test the low-order 32 bits.

    Saturday, May 06, 2006 6:45 AM by KJK::Hyperion

    > Rigorousness OK, but it doesn’t take a lot of

    > psychic skills to understand what goes on

    > behind the curtain. The return value is

    > obviously born as a positive int and casted

    > into an HINSTANCE with IntToPtr().

    Um, since the HINSTANCE used to be an address sometimes, it’s not obvious to me that it’s always born as an int.  I only would have guessed that error returns were born as ints … sometimes.  Considering that 64-bit-aware programs must cast it down to 32 bits, obviously even some error returns are born as long longs but have the error value shown in the low-order 32 bits.

  14. oldnewthing says:

    Cast it to int, cast it to INT_PTR, it won’t make any difference. (I’m not sure what you’re referring to in that last sentence. What error codes are born as long longs?)

  15. Norman Diamond says:

    > Cast it to int, cast it to INT_PTR, it won’t

    > make any difference.

    That is obvious only to someone who either knows or has reverse engineered the code.  MSDN isn’t quite sure about that, your colleagues seem sure about the opposite, and even you didn’t quite seem so sure a few replies back.

    > What error codes are born as long longs?

    I don’t know, but the responses of both you (prior to this one) and your colleagues made it obvious that there are some.

    In a 64-bit program, HINSTANCE is 64 bits.  Now consider a case where ShellExecute has set that field to the value 2**33 + 1.  If the program casts this to type INT_PTR then the result of the cast will be greater than 32 and the program will think that it represents success.  If the program obeys MSDN and your colleagues, casts this to type int, and sees the result of 1, then the program will observe that it represents failure.  Since your company is so firm about the incorrecness of casting this to INT_PTR, it must be because some failure returns will be misinterpreted as successes unless they get cast down to int.

  16. When you are talking about office programs what happens when you open a new document seems to depend…

Comments are closed.