How do I perform shell file operations while avoiding shell copy hooks?

Okay, the subject line of the article gives away the answer to the puzzle, but here's the puzzle anyway: A customer reported a problem with the SHFile­Operation function:

Consider the following program:

#include <windows.h>
#include <shellapi.h>
int main()
SHFILEOPSTRUCT fileStruct = {};
    fileStruct.wFunc = FO_RENAME;
    fileStruct.pFrom = L"C:\\a\0";
    fileStruct.pTo   = L"C:\\b\0";
    fileStruct.fFlags= FOF_NO_UI;
    return 0;

If "a" is a file, then everything works fine, but if it's a directory, then Application Verifier raises the following error:

Heap violation detected
Memory access operation in the context of a freed block: reuse-after-delete or double-delete

Can you help explain what we're doing wrong? So far as we can tell, all our parameters are correct.

This is one of those "It doesn't work on my machine" issues, because the provided sample program runs fine on a freshly-installed copy of Windows. We asked the customer to send us a crash dump file, and from that crash dump the source of the problem was obvious:

eax=00000001 ebx=00000000 ecx=73d34c58 edx=00270001 esi=09fa2ff8 edi=00000000
eip=10001131 esp=0026dea8 ebp=0026df24 iopl=0         nv up ei pl nz na po nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010200
10001131 8b4604          mov     eax,dword ptr [esi+4] ds:002b:09fa2ffc=????????
0:000> k
  *** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
0026df24 75a3554a Contoso+0x1131
0026df64 75a1b07c ole32!ActivationPropertiesIn::DelegateCreateInstance+0x108
0026dfb8 75a1aff1 ole32!CApartmentActivator::CreateInstance+0x112
0026dfd8 75a1ae16 ole32!CProcessActivator::CCICallback+0x6d
0026dff8 75a1adc7 ole32!CProcessActivator::AttemptActivation+0x2c
0026e034 75a1b0df ole32!CProcessActivator::ActivateByContext+0x4f
0026e05c 75a3554a ole32!CProcessActivator::CreateInstance+0x49
0026e09c 75a352ce ole32!ActivationPropertiesIn::DelegateCreateInstance+0x108
0026e2fc 75a3554a ole32!CClientContextActivator::CreateInstance+0xb0
0026e33c 75a35472 ole32!ActivationPropertiesIn::DelegateCreateInstance+0x108
0026eb18 75a45916 ole32!ICoCreateInstanceEx+0x404
0026eb78 75a45877 ole32!CComActivator::DoCreateInstance+0xd9
0026eb9c 75a45830 ole32!CoCreateInstanceEx+0x38
0026ebcc 75c61fe0 ole32!CoCreateInstance+0x37
0026ee64 75c61354 shell32!_SHCoCreateInstance+0x1ac
0026ee88 75c1b904 shell32!SHExtCoCreateInstance+0x1e
0026eec8 75becbcf shell32!SHExtCoCreateInstanceString+0x43
0026f10c 75beca76 shell32!CreateCopyHooks+0xe1
0026f344 75bec9da shell32!CallCopyHooks+0x4b
0026f370 75bec95b shell32!CallFileCopyHooks+0x29
0026f5bc 75bec8a4 shell32!CFileOperation::CopyHooks+0x119
0026fa3c 75c37955 shell32!CCopyWorkItem::_UpFrontConfirmations+0xb7
0026fc6c 75c378b0 shell32!CCopyWorkItem::ProcessWorkItem+0x83
0026fca0 75c37fda shell32!CRecursiveFolderOperation::Do+0x1d5
0026fce4 75c39a19 shell32!CFileOperation::_EnumRootDo+0x14e
0026fd4c 75c397b9 shell32!CFileOperation::PrepareAndDoOperations+0x27f
0026fd74 75c396c7 shell32!SHFileOperationWithAdditionalFlags+0xe9

The crash is in some third party component named Contoso, which is running because it is being Co­Create'd. The call came from Create­Copy­Hooks, and it doesn't require very much in the way of psychic powers to conclude that the shell is creating the Contoso object because it registered as a copy hook.

This also explains why the problem occurs only on the customer's machine: The customer installed the Contoso shell extension and we didn't.

Okay, so the problem is that the Contoso shell extension has a use-after-free memory corruption bug. (Some Web searching revealed that a lot of people had encountered problems with the Contoso shell extension.)

The FOFX_NO­COPY­HOOKS flag comes in handy here. Setting this extended flag disables copy hooks for your file operation. Extended flags cannot be passed to the classic SHFileOperation function because the SHFILEOPSTRUCT structure uses a 16-bit WORD for the fFlags member, but the FOFX_NO­COPY­HOOKS flag has the numerical value 0x00800000 which doesn't fit in a 16-bit integer. (The "X" at the end of the prefix is another clue.) The way to set extended flags is to use the IFileOperation interface.

// Just for fun, I'll use ATL templates instead of raw C++.
HRESULT RenameAtoB()

 CCoInitialize init;
 hr = init;
 if (FAILED(hr)) return hr;

 CComPtr<IFileOperation> spfo;
 hr = spfo.CoCreateInstance(CLSID_FileOperation);
 if (FAILED(hr)) return hr;

 hr = spfo->SetOperationFlags(FOFX_NOCOPYHOOKS);
 if (FAILED(hr)) return hr;

 CComPtr<IShellItem> spsi;
 hr = SHCreateItemFromParsingName(L"C:\\a", NULL,
 if (FAILED(hr)) return hr;

 hr = spfo->RenameItem(spsi, L"b", NULL);
 if (FAILED(hr)) return hr;

 hr = spfo->PerformOperations();
 if (FAILED(hr)) return hr;

 return S_OK;
Comments (20)
  1. NB says:

    Can you install hooks that cannot be bypassed?

    (In case they are really, really important.)

  2. Adam Rosenfield says:

    @NB: You just want to escalate an arms race.  If someone doesn't want to run your unbypassable hooks, they could just reimplement the part of SHFileOperation/IFileOperation they need using CreateFile/ReadFile/WriteFile/etc.  How do you expect to hook that?

  3. SI says:

    Just install a rootkit, to ensure your hooks cannot be bypassed

  4. Mike Caron says:

    Would it actually be appropriate disable hooks in this case? Sure, Contoso is a piece of crap, everyone knows that. But, most customers don't have it installed, preferring the LitWare extension instead which provides a valuable service. People would be ticked off knowing that it was being bypassed in this case!

  5. pcooper says:

    Is the intention that programs should often/always run operations with the setting to disable hooks? Or is this supposed to be used for some corporate/kiosk deployment with custom software that's needing to work around the broken hook but still needs the shell extension installed? I guess I'm just wondering when I might want to use this technique.

    [The scenario here was some code that just wanted to copy a directory tree and figured they could use the shell copy engine instead of writing a recursive tree copy manually. They didn't need or wany any UI or other customization; they just wanted CopyFileRecursive. -Raymond]
  6. alegr1 says:

    And this is why Microsoft should implement blacklisting to actively disable faulty third-party code. And a shamelist: "The Contoso extension was found to cause Explorer crashes. It's disabled for now. Contact the vendor for the update."

  7. Skyborne says:

    @alegr1: vendor X has gone out of business and/or quit supporting Contoso but it's crucial for my business with vendor Y's application which we have a 3-year contract on, please have microsoft send me the update codes.

  8. Joshua says:

    @Skyborne: That's why these things have overrides.

  9. JM says:

    I think this behavior is completely unacceptable for Contoso software. These guys must pay MS millions to have their name dropped all over the place in sample projects and documentation. The least they can do is not reflect badly on Microsoft with bug-ridden software.

  10. Brian says:

    @JM: Contoso switched to Google Apps April 1st 2011, so they'll probably be writing buggy Google Apps from now on ;)  

  11. ebraminio says:

    For readers that don't know mean of Contso, it is used here as a placeholder name for a real product. I put that on Contoso article of English Wikipedia :)

    [This is hardly the first time I've used Contoso as a placeholder name. -Raymond]
  12. Kawahee says:

    At work when customers report crashes in our application our support team gets memory dumps and I !analyze them (and that's about where my knowledge of WinDbg ends). More often than not its code in our app; three or four times it's been a shell extension.

    Are these developer support cases chargeable? Or unlimited as per some contract? It sounds like Microsoft's willing to do my job for me.

  13. Damien says:

    @Adam Rosenfield – I'd *suspect* sarcasm from NB here – if they're a regular reader, it's the obvious next level to go to.

  14. Anonymous Coward says:

    I find it odd that users wouldn't notice Explorer crashing when copying files. Was there something specificly strange about the files copied? A possible downside of passing the flag could be that you might be bypassing source control, logging, auditing, … As I understand it, it was fine for this customer, but this flag really isn't for general use.

  15. cheong00 says:

    Is there any builtin Windows utility that uses this flag? Seems like it could be handly support tips.

  16. Dave says:

    I build networking apps.  Our code has a built-in scanner that checks for the presence of McAfee, Symantec, and Norton firewall software if it encounters some should-not-occur networking error.  A lot of the time it's that lot.  So having to modify your app to work around another vendors' bugs isn't that far-fetched.

  17. whoop says:

    Why not use console command move or xcopy instead ?

    Isn't this an example of recreating the wheel poorly ?

    oh well

    [This comment system really do love to eat my comments. Raymond, doesn't anyone want to fix it or is it 'by design' ?]

    [Every few months, the people who run the servers send out email saying "We will be using our maintenance window to install a patch that will hopefully fix the comments problem." Seventh time's a charm, right? (Note: I made up the number 7.) -Raymond]
  18. Mark S says:

    Dave – I write .NET framework apps and also have had to work around a lot of third-party garbage screwing with the network stack and causing weird errors in the bowels of the framework networking code.  Lots of time wasted trying to find out what "I" had done wrong.

  19. metafonzie says:

    @Mark S

    Do you mean the third party LSPs? They are indeed horrible (esp Ponjour from Abble). But the worst offenders are shitty (mostly USB & Wireless) drivers that increase system DPC latency thus guaranteeing a horrible audio experience. Its ironic that I bought this several-thousand-dollar Zony Hiyo ( ;-) ) laptop from a MS store in Bellevue which was "MS Signature" branded (i.e. no bloatware) and it came with the worst possible drivers. I continue to get horrible DPC latency spikes even after a fresh W7 install with minimal drivers from Zony's website.

  20. Joshua says:

    [Every few months, the people who run the servers send out email saying "We will be using our maintenance window to install a patch that will hopefully fix the comments problem." Seventh time's a charm, right? (Note: I made up the number 7.) -Raymond]

    It will get fixed when it crosses Raymond's annoyance threshold and he does something about it.

Comments are closed.