One possible reason why ShellExecute returns SE_ERR_ACCESSDENIED and ShellExecuteEx returns ERROR_ACCESS_DENIED


(The strangely-phrased subject line is for search engine optimization.)

A customer reported that when they called ShellExecute, the function sometimes fails with SE_ERR_ACCESSDENIED, depending on what they are trying to execute. (If they had tried ShellExecuteEx they would have gotten the error ERROR_ACCESS_DENIED.)

After a good amount of back-and-forth examing file type registrations, a member of the development team had psychic insight to ask, "Are you calling it from an MTA?"

"Yes," the customer replied. "ShellExecute is being called from a dedicated MTA thread. Would that cause the failure?"

Why yes, as a matter of fact, and it's called out in the documentation for ShellExecute.

Because ShellExecute can delegate execution to Shell extensions (data sources, context menu handlers, verb implementations) that are activated using Component Object Model (COM), COM should be initialized before ShellExecute is called. Some Shell extensions require the COM single-threaded apartment (STA) type.

As a general rule, shell functions require STA. Recall that MTA implies no user interface. If you try to use an apartment-threaded object from your MTA thread, a marshaller is required, and if no such marshaller exists, the call fails.

This also explains why the failure occurs only for certain file types: If handling the file type happens not to involve creating a COM object, then the MTA/STA mismatch situation never occurs.

Comments (15)
  1. John says:

    Yep.  An error code of "access denied" is OBVIOUSLY related to COM marshalling.  Only an idiot couldn't have figured that out.

  2. Stuart says:

    Would it have made sense to make the ShellExecute function fail aggressively if called from an MTA thread? That way programmers would discover their mistake right away rather than having it fail "apparently randomly" for certain file types, or start failing later if the handler for a particular file type gets changed to require COM in the future.

    Obviously I understand that for backward compatibility it's impossible to change that NOW. Just curious about your opinion on whether or not it would have made sense to define ShellExecute that way when it was first implemented…

  3. SI says:

    Did MTA's exist back when ShellExecute was first implemented?

  4. Yuhong Bao says:

    SI: No, in fact it was added after 32-bit COM/OLE2 was originally released with DCOM95 and NT 4.0. For that matter, ShellExecute itself was created before the creation of COM/OLE2, which makes the claim "Because ShellExecute can delegate execution to Shell extensions (data sources, context menu handlers, verb implementations) that are activated using Component Object Model (COM), COM should be initialized before ShellExecute is called. " quite odd.

  5. Joshua says:

    Please tell me that using ShellExecute to spawn an EXE image does not require COM.

    If this is not the case than UAC is broken to the point of requiring being completely ripped out.

  6. configurator says:

    @Joshua: It doesn't. It requires COM when the execution is delegated to shell extensions (data sources, context menu handlers, verb implementations) that are activated using COM.

    What does it have to do with UAC though?

  7. Pierre B. says:

    What I usually take from post of the COM / appartment related posts is that low-level design decisions that seem sane at their level can make things hard when you look at the big picture. Here, all it comes down to, in the end, is the dual facts that windows' handles are thread-specific and that win32 message passing is intimately linked to threads.

    Example alternative design would be to have an explicit message port to send and receive, with an implicit default port for backward compatibility. Then multiple thread could share a single source of messages. The Amiga had that model. Well, except the implicit default since it didn't need backward compatibility. MACH, the left-head of the two-headed MacOSX kernel, is also port-based.

    Hindsight is 100%

  8. Ben Voigt says:

    @Pierre: Windows have thread affinity, but window handles are machine-global, across processes and I believe even login sessions.

    @Joshua, configurator: On a plain Windows install no COM objects are involved in launching executables, but that can be changed (in the registry).  Scenarios where starting programs through ShellExecute use COM typically involve virus infection.

  9. blah says:

    MTA implies poor service and fare hikes.

  10. John says:

    @Larry:  The documentation makes no mention of specific error codes in regards to COM initialization.  Furthermore, how would you tell the difference between "COM is not properly initialized" and "access to the file is denied"?  Using the same error code for multiple failures is bad; not being able to differentiate between them is worse.  This is just poor API design.  If you don't have COM initialized as STA, calling this function is pretty much a crap-shoot.  The documentation more-or-less states as much, but I think it needs to be worded more strongly.  That still doesn't address the problem with the error code, though.

  11. W says:

    @Joshua

    To spawn an .exe you probably should use CreateProcess instead of ShellExecute. That way it is launched as a PE image regardless of file-extension. ShellExecute is convenient because it allows the opening of documents with the associated program which isn't necessary for PE images.

  12. Larry Lard says:

    @John that's not the obviousness. The obviousness is that an error calling ShellExecute suggests (closely) reading **the documentation for ShellExecute**.

  13. Joshua says:

    If you spawn a process that requires elevation, CreateProcess errors with ERROR_ELEVATION_REQUIRED but ShellExecute will display the password prompt and succeed if permission is granted.

  14. Pierre B. says:

    Ben: yes, sorry, I used the wrong term. The handles have to be globally unique. I meant the window.

  15. 640k says:

    @ Yuhong Bao: No, in fact it was added after 32-bit COM/OLE2 was originally released with DCOM95 and NT 4.0.

    ShellExecuteEx doesn't have to be broken because of ShellExecute is broken.

Comments are closed.