What were ShellExecute hooks designed for?

Windows 95 introduced (and Windows Vista removed) the concept of ShellExecute hooks. These are objects which implemented the IShellExecuteHook interface. That interface had just one method: IShellExecuteHook::Execute, which took a SHELLEXECUTEINFO structure and returned S_OK to indicate that the item was executed, S_FALSE to allow processing to continue, and an error code to halt processing.

The intended purpose of the shell execute hook was to allow you to extend the set of strings that can be executed. For example, Internet Explorer 1 used a shell execute hook so that you could type http://www.microsoft.com/ into the Run dialog and invoke the Web browser. This was necessary because the original version of Windows 95 had no special knowledge of URLs. Without the hook, typing a URL into the Run dialog would have generated an error.

Let's look at those return values again. The simplest return value is S_FALSE, which means that you didn't recognize the item and wish to continue normal processing. Returning S_OK means that you recognized the item and successfully executed it. Returning an error code means that you recognized the item but an error occurred when you tried to execute it. Returning an error stops further processing, which is important in this case, because you do want to stop processing: You recognized what the user was trying to do and you couldn't do it. If you had returned S_FALSE, then the shell (and other shell execute hooks) would take their shot at executing the item. Since it's something only you recognize, they would fail, and the user would get some sort of error like "File not found" instead of an error that accurately describes why you couldn't execute the item.

That's the theory, but I've seen a few applications that use shell execute hooks for completely different purposes. And they are doomed to failure. One application installs a shell execute hook in order to implement some sort of weird concept of "security". When their hook is called, they check whether the user is "allowed" to run the program; if allowed, they return S_FALSE and if blocked, they return E_ACCESSDENIED. Another application uses their hook to log all the programs that the user runs, presumably so the logs can be inspected for auditing purposes. After logging the action, they return S_FALSE to allow normal execution to continue.

Both of these methods are really an abuse of the shell execute hook model since they aren't extending the set of items that can be executed by the shell. They're just using the hook model to sneak some code into the execute path. But that's also the problem.

First, they are intercepting only actions that go through the ShellExecuteEx function. If somebody just calls CreateProcess directly, then the shell execute hooks won't run, and whoever it is just snuck past your "security" and "auditing" system.

Second, these hooks both assume that they are the only shell execute hook in the system. (Remember, there can be more than one.) Suppose there are three hooks installed, the hook installed by Internet Explorer to handle URLs, the "auditing" hook, and the "security" hook, and suppose that the shell decides to invoke them in that order. (The order in which hooks are executed is unspecified because, if you use them as intended, the order doesn't matter.) If the user types a URL into the Run dialog, the URL hook launches the Web browser and returns S_OK to say, "I took care of this one." Your "auditing" hook never got to see the URL, and your "security" hook never got a chance to reject the operation. Hooks that sit ahead of the "security" hook not only get to bypass security, but they also elude the "auditing" hook.

So much for "auditing" and "security".


Since shell execute hooks effectively become part of the ShellExecute function, they are unwittingly subject to all the application compatibility constraints that ShellExecute must follow. Welcome to the operating system. Your world has just gotten a lot more complicated.

There is no "list of application compatibility constraints" (oh how life would be simple if there were); basically, anything that ShellExecute did in the past is a de facto compatibility constraint. Some programs crash if you create a worker thread inside ShellExecute; other programs crash if you call PeekMessage inside ShellExecute. Some programs call ShellExecute and immediately exit, which means that your shell execute hook had better not return before its work is done (or at least before the work has been given to another process to finish). Some programs call ShellExecute on an unusually small stack, so you have to watch your stack usage. And even though it is called out in the documentation that programs should not call shell functions from inside DllMain, that doesn't stop people from trying, anyway. Your shell execute hook had better behave in a sane manner when called under such "impossible" conditions.

Comments (22)
  1. asf says:

    I’m using them in my UAC like thing I created for XP, I was hoping to bring it along when/if I move to Vista, guess not (UAC on Vista removes my control, if a exe is marked as requireAdmin, there is NO WAY for me to run it non elevated,)

  2. godzila says:

    Apparently your implementation wasn’t correct either. If another hook was installed and fired before your UAC like thing and returned S_OK. Then you’re UAC thing would be circumvented. Sounds like it’s a good thing they got rid of this hook.

  3. John says:

    Is there any kind of detection of infinite loops?  I.e. calling ShellExecute inside of a ShellExecute hook?  Also, while the order of hook execution is “unspecified”, is it safe to say that the order is consistent (assuming no modifications are made between successive calls to ShellExecute)?

    [Some shell execute hooks call ShellExecute on purpose because they want to convert one type of execution into another. (And I see no value in knowing the answer to the second question given that it’s all unspecified anyway. It’s an oddly-phrased question. “If nothing changes, will things be the same?”) -Raymond]
  4. John says:

    I was just wondering if the hooks will always be called in the same (unspecified) order.  Perhaps the Security or Audit example hooks are written in such a way that calling them in one order “succeeds” and calling them in another order “fails”.  If the order in which they are invoked can change between calls to ShellExecute than you would see only intermittent “failure”.  I would guess the hooks are stored in a list container, so they are probably always called in the same (unspecified) order.

    [Different versions of the shell have done it differently. Some cache the hooks in a per-process list (so the order remains the same within a process). Others recalculate the hooks each time (so the order can change even within a process). -Raymond]
  5. Gabe says:

    I can’t wait to find out the reasoning behind removing them in Vista.

    I would assume that it’s because nobody used them for their intended purpose, and they just created a security hole.

  6. Yuhong Bao says:

    I feel COM is overkill for just one function.

  7. asf says:

    godzila: its not about circumventing "my UAC", its just used to detect stuff that "needs" elevation (control panel applets etc) so, if someone hooked in before me, the only thing that would happen is that you would not get the elevation dialog and the process would start with a restricted token…

  8. jon says:

    You can actually get them back on Vista by adding a registry key, no doubt even this will be gone in Windows 7. So much for the "backwards compatibility above all else" mantra :(

  9. Right.  It’s backward-compatibilty above all else, except security.  And maybe reliability.  It’s like that Dilbert comic, where he says something like "Quality comes first."  And then someone says, "What about security?"  and then "What about the shareholders?  and then "What about legal compliance?"

    "Okay, quality comes fourth."

  10. Matt says:

    I guess the “Q” stands for “uality”.  http://books.google.com/books?id=-hUPgZK4rjMC&pg=PA222&lpg=PA222&ots=z-JvQvMvbq&sig=Ash7Px1nUvg5fvDOMCihEoxBt4M (scroll up to the bottom of page 222)

  11. recursivelyenumerable says:

    reminds me of how subclassing is often abused.

  12. Yuhong Bao says:

    "Internet Explorer 1"

    And 2 also.

  13. Bob says:

    Yes, asf, you’re very clever.

    And therefore the behaviour of your clever little system will silently change and confuse the users the moment someone installs a completely unrelated application.

    Oh, wait, that’s why you’re doing it wrong and why every smart person has now found a good reason to praise Microsoft for creating Vista.

    Microsoft is being backward compatable – with every other OS version, all ShellExecute hook based hacks would break silently and at random for no obvious reason, and that feature has been retained.  ;)

  14. dsn says:

    Another problem that occurred to me with ShellExecute hooks – what if two programs both register hooks for the same type of item?  For example, if both Firefox and IE had a hook to detect urls, it sounds like you would randomly get either IE or Firefox handling them, with no way to predict or control which it would be.

  15. asf says:

    Bob: I never said it was for use by other people, this thing is for my own use. Now tell me, on your precious Vista, when a exe is marked as requireAdmin, how can you run it non elevated when UAC is on?

  16. Phylyp says:

    I know Raymond doesn’t name names, but were there are "big name" programs that actually did this? Any hints, clues?

  17. Richard says:

    A while back, I was investigating integrating an additional executable file format into Windows (note, this format has no associated extension, and is detected by filesystem attributes instead). As far as I could tell, that’s impossible (though I’d love it if someone would say, "you’re wrong, here’s how…"). A ShellExecute hook was as close as I could get.

    So, my software is broken in Vista? What should I do instead?

  18. Greg D says:


    Write your own shell?

  19. Yuhong Bao says:

    "A while back, I was investigating integrating an additional executable file format into Windows (note, this format has no associated extension, and is detected by filesystem attributes instead). As far as I could tell, that’s impossible (though I’d love it if someone would say, "you’re wrong, here’s how…"). A ShellExecute hook was as close as I could get."

    As I remember, CreateProcess launches a process in another subsystem by retrieving the path to the EXE launcher for that subsystem from the registry, then launches that EXE passing the original path as a command-line parameter.

  20. Yuhong Bao says:

    "So much for "auditing" and "security". "

    Yep, don’t reinvent the wheel. NT already has auditing and security features, use them.

  21. Not Yuhong Bao says:

    Yep, don’t reinvent the wheel. NT already has auditing and security features, use them.

    And if you need compatibility with Windows 95 using the same codebase?

  22. Alexander Grigoriev says:

    Not Yuhong Bao:

    "And if you need compatibility with Windows 95 using the same codebase?"

    And what if you need compatibility with DOS?

    Sometimes the compatibility has to end.

Comments are closed.