The challenges in changing the way Explorer hosts shell extensions


Various types of shell extensions such as thumbnail extractors are run in a separate process, and if one of those shell extensions crashes, it takes out the COM Surrogate rather than the main Explorer process.

Anonymous wondered if this model could be extended to all types of shell extensions, perhaps not all at once, but gradually. The dangers of extending this model to existing shell extensions are compatibility (of course) and re-entrancy.

The thumbnail extractor interface was lucky in that the only parameter an extractor received was an IShellItem representing the object for which the caller wishes to retrieve a thumbnail. There's no foothold into the user interface, which means that it can be moved to a place that has no user interface.

Unfortunately, most shell extensions are not so lucky. Many of them receive some user interface object (usually a window handle) as a parameter. If those types of shell extensions have historically been hosted in-process, then the implementors of those shell extensions will do things like take that window handle and subclass it (possibly walking around the window hierarchy for a while until it finds a window it wants to subclass). Moving the shell extension into another process would break it, because you can't subclass windows in another process.

Even if you manage to find an interface that is given no foothold into the user interface, you still may not be able to move it to a host process due to the danger of re-entrancy. When you invoke a COM method call from a single-threaded apartment (and all UI work is done in single-threaded apartments), and the object that is the recipient of the call lives on another thread or even in another process, COM will send the request to that other thread and pump messages waiting for the reply. This means that a method call which previously never pumped messages now does, opening windows of re-entrancy, and the great thing about windows of re-entrancy is that you never hit them yourself, but your customers somehow manage to find them without any problem.

Comments (26)
  1. A. Skrobov says:

    the great thing about windows of re-entrancy … you never hit them … customers manage to find them

    I just couldn't parse that into a meaningful sentence

  2. Steve says:

    My understandiong of what he was saying is that because you are aware of the reentrancy you avoid it, your customer's do not and so find them in everything they try to do.

  3. Dave says:

    I think he's saying that despite your best test efforts you'll not trigger any errors caused by reëntrancy but give your customers five minutes they'll find several.

    [Exactly. I had to fix one of those types of bugs just last week. A re-entrancy bug never hit in testing, but found by customers pretty frequently. -Raymond]
  4. arousedboat says:

    Reëntrancy was killed by grunge.

  5. wqw says:

    I thought out-of-process calls are pumping only synchronization messages on caller thread, not all (incl UI) messages?

    [Dunno what you mean by synchronization messages, but non-queued messages are dispatched, and that's the main re-entrancy source. (Another is if an inbound method call arrives while you are waiting for the outbound call to complete.) -Raymond]
  6. Joshua says:

    There is a trick to solve the re-entrancy, but that won't help people trying to subclass windows.

  7. xpclient says:

    I can never forgive Microsoft for taking out IColumnProvider and compatibility with around 60% shell extensions with Vista. Only about 40% of shell extensions on the market work with Vista and later (and I mean 32-bit Vista, not even 64-bit Windows for which Microsoft forgot to build a bridge). Vista broke shell extensions of almost all type – Property sheet, infotip, context menu, icon handler, thumbnail handler, AutoPlay handler and entirely got rid of column handlers and Windows 7 also later joined the massive functionality degradation party. Vista/W7 shell (Explorer, Taskbar, Start Menu) is a massive step back in compatibility, usability, customizability and features. It's as if the NT 6 shell was done by interns at Microsoft who still don't seem to get their act together for Windows 8.

  8. Some guy says:

    Reëntrancy

    As a German I get a headache trying to read this. xD

  9. John says:

    @xpclient:  IColumnProvider was a useful extension mechanism; I still miss it.  Other than that I have not seen many compatibility issues introduced by Vista or 7.  Maybe the ones that broke were just poorly written in the first place.  Unfortunately I have to agree with you that lately things are being dumbed down a bit; my favorite is the file times going from hh:mm:ss to "X minutes ago".  Ironically this is only for files modified recently; files modified less recently retain the hh:mm:ss format.  So the more recently a file has been modified the less accurate its last modified time is.  Beautiful.  Now Windows 8 is bringing the cell phone UI to the desktop; please excuse me while I gag on my own vomit.

  10. Anonymous Coward says:

    @Some guy: is coördinate better or worse?

  11. Joshua says:

    @xpclient: stop your pointless gripe. The shell can be pulled and replaced any time: Some people run alternate shells. If your Windows program is broken by the shell being replaced, you need to stop being a Windows programmer.

  12. Some guy says:

    The ö in coördinate gets parsed as an o-umlaut, which is immediately discarded/corrected by the spell checker. It just causes a minor delay because the "mis-spelling" is kind of unusual.

    The ë makes my parser throw fits, though, definitely worse.

    (Either the blog sw ate my post again or this is gonna double post)

  13. If a bug hits once a week in normal usage, you can hit it once an hour by throwing 200 machines at the problem.

  14. xpclient says:

    @Joshua, we are talking about shell extensions here. How would replacing the shell help in any way with achieving shell extension compatibility? You seem to have no idea about how the shell took a step back w.r.t number of features and shell extension compatibility in Vista/7.

  15. grumpy says:

    @Maurits: not if the machines and their operators are paid for by the company that also produced the code. Such is the nature of heisenbugs. :-)

  16. cheong00 says:

    @Maurits: Related read: One in a million is next Tuesday. blogs.msdn.com/…/104165.aspx

  17. Leo Davidson says:

    @xpclient, At least two of the alternative file managers still support IColumnProvider, even on Vista and Win7. Windows did not drop support for those shell extensions; Windows Explorer did.

    I believe that was the sort of point that Joshua was trying to make.

    (You don't have to replace the entire shell, either. That's more trouble than it is worth, IMO. You can replace Windows Explorer with something else as the default folder double-click handler.)

  18. 640k says:

    Yeah, you could run XP (virtual guest) in W7, therefore IColumnProvider is still supported by W7.

  19. xpclient says:

    @Leo Davidson, the problem with using third party file managers is that they are slow to adopt or completely missing the new functionality every new release of Explorer/Windows brings. For example, many do not yet support libraries, saved searches, property handlers, preview pane, Windows Search-based recursive filtering or the new file copying improvements in Windows 8 like pausing (you get the idea). There are certain core OS components for which no replacement is not good enough and Explorer is one of them. Plus, almost every third party file manager is extremely ugly. Not that Explorer is particularly beautiful but it's far less uglier. Using an Explorer replacement is just not feasible. Microsoft should have made efforts to preserve shell extension compatibility in Vista and later. (Dozens of property sheet, infotip, thumbnailing etc extensions do NOT work in Vista but work in XP). I don't know the reasons for such high level of broken compatibility (might be UAC or might be deprecated shell interfaces).

  20. Dave says:

    Various types of shell extensions such as thumbnail extractors are run in a

    separate process, and if one of those shell extensions crashes, it takes out

    the COM Surrogate rather than the main Explorer process.

    What about running the extension not just in a separate process but with a restricted token so that it can't do any damage?  I realise that its original goal was simply to prevent crashes, but given that malformed images and codec exploits are a primary exploit vector for 0-day malware, this seems like a useful way to sandbox anything that deals with potentially tainted data.  Is there any technical reason why this couldn't be done?  Just curious…

  21. Dave says:

    Argh, I can't edit my previous post: The question was based on the previous discussion of COM surrogates some time ago, "Explorer has learned not to trust thumbnail extractors; they have a poor track record for stability".  They also have a poor record for security, sandboxing them seems like one way of dealing with this.  This helped Android a while back with a malformed MP3, since they run their media codecs in a sandbox someone who used the MP3 exploit couldn't get very far with it.

  22. Leo Davidson says:

    @xpclient: You just haven't found the right file manager…

    I won't say the name as I don't want to look like I'm advertising a particular product, but there is at least one which covers all of that functionality (and far from playing catch-up, in several cases it had those things or similar years earlier (and with things like preview handlers, supports both them and its own older viewer APIs so you get both) and which looks pretty much however you want it to look.

    But it sounds to me like you just want to complain rather than actually solve your problems. :)

    Complaining about Windows Explorer is like complaining about Notepad, Calc or MSPaint. They're fine for casual usage but if you find the bundled tools too limited then you should look for a tool aimed more at what you want to do.

    My only annoyance with IColumnProvider being dropped by Windows Explorer is it means — especially once XP is phased out completely — that shell extensions are less likely to implement it in the future. But source control is about the only type of shell extension where I want/need an IColumnProvider and hopefully Tortoise* will continue to implement it for the file managers which continue to support it.

  23. Leo Davidson says:

    @Dave: "What about running the extension not just in a separate process but with a restricted token so that it can't do any damage?"

    That's not a bad idea for new code, as some kind of opt-in, but I'm pretty sure it would break too many existing shell extensions to be able to make it mandatory or opt-out. Too many of them assume (and require) that they can do things which a sandbox/restricted token would break.

    (Just look at the preview handlers in your registry, and how few of them opt into running under low-integrity. And that is things which had to be designed for an API which had the low-integrity opt-in from the start.)

  24. Arno says:

    @Joshua: what's the trick to solve the reentrance? I know that what's done is done, but the combination of "PeekMessage may create reentrance, but you should know that and prepare for it" and Apartment-threaded COM using the message pump to wait for returning calls is one of the most deadly pitfalls I encountered so far in the Windows architecture. In particular, IMessageFilter does not always help because PeekMessage dispatches "sent" messages internally. I am pretty sure that many programs capable of hosting out-of-process COM servers (for example, Office) have a myriad of little bugs caused by this reentrance possibility.

  25. Dave says:

    @Leo Davidson:

    Just look at the preview handlers in your registry, and how

    few of them opt into running under low-integrity. And that is

    things which had to be designed for an API which had the

    low-integrity opt-in from the start

    I thought LI for preview handlers was opt-out, so you have to explicitly set DisableLowILProcessIsolation = 1, see msdn.microsoft.com/…/cc144143%28v=vs.85%29.aspx. Does this mean that developers are mostly setting DisableLowILProcessIsolation, so they're choosing to opt-out for LI?  The linked page actually warns that "it is not recommended to do so. Systems could eventually be configured to reject any process that is not low IL".

  26. benjamin says:

    My only problem with Microsoft breaking context handlers' backward compatibility is that it didn't go far enough. I know this topic's done to death, but after just straight-up malware I can't begin to count how many times I've seen misbehaving context handlers mess up the UX.

Comments are closed.