Don’t forget to double-null-terminate those strings you pass to SHFileOperation

About once every two months for the past six months (I stopped checking further back), somebody reports a problem with the SHFileOperation function. Often, they don’t include very much information at all. They just say, “I call the function and it doesn’t work.” Here’s an example:

I’m hitting a problem with SHFileOperation when using it to frob files in the gonzo directory when the user’s SID ends in an odd number.

    // Delete the file.
    // szDeletePath contains the full path to the file.
    shFileOp.hwnd = NULL;
    shFileOp.wFunc = FO_DELETE;
    shFileOp.pFrom = szDeletePath;
    shFileOp.pTo = NULL;
    shFileOp.fFlags = FOF_NO_UI;
    iRet = SHFileOperation( &shFileOp );

The function returns file not found, but the file is definitely there.

If you read the variable names carefully, you can see the problem.

The pFrom and pTo members of the SHFILEOPSTRUCT structure are double-null-terminated strings. (There’s even a callout box for this in the MSDN documentation.) But a variable named szDeletePath is probably a single-null-terminated string. (The name for a double-null-terminated string would be szzDeletePath.)

My psychic powers tell me that szDeletePath is not double-null-terminated.

So far, my psychic powers haven’t failed.

Now, you might say that the fact that people make this mistake so often is a sign that the function is flawed. And if the function were designed today, I would agree with you. But this function in its public form is over fifteen years old (and in its private form, is around 20 years old), and back in those days, programmers were assumed to have the time to understand the subtleties of what they were doing.

Comments (30)
  1. MItaly says:

    Hooray! The Hungarian notation at least once has been useful! Don’t miss the next time this will happen, at the next blue moon!

  2. John Topley says:

    Did SHFileOperation debut in its private form in Windows 3.x or Windows NT 3.x then? I’d always assumed that the SH* APIs were new for Windows 95.

    [Not sure what your point is. Yes, before it made its public debut, SHFileOperation was a private function. (Didn’t I basically already say this in the article?) -Raymond]
  3. Marquess says:

    Really, makes you appreciate .NET, where people finally got around to a decent String[] type. And strong typing.

  4. Joe says:

    @Marquess: Except most parts of the .NET libraries don’t support paths longer than MAX_PATH, so System.IO becomes of limited use and you have to fall back to the win32 functions.

    It’s an irritating weakness – and if it were changed, suddenly a whole bunch of software would pseudo-magically support long paths. But, as ever, there are probably backwards compatibility issues.

    [For example, extending beyond MAX_PATH means that “foo” and “foo.” (trailing dot) and ” foo ” (leading and trailing space) are no longer the same file. -Raymond]
  5. Marquess says:

    MAX_PATH: 260 characters.

    NTFS file name limit: 255 characters.

    Did nobody think that this might become a problem, back in 1993? No CreateFileEx?

    [Of course somebody thought of it, but there are also arguments against setting MAX_PATH too high. E.g., stack buffers. (In Win16, MAX_PATH was 64, and it became common to put path buffers on the stack.) I’m always impressed by people who view engineering decisions in black and white. (Like an idiot, I’m prolonging the off-topic discussion instead of just deleting it.) -Raymond]
  6. Boris says:

    How would you design it today given the same language limitations (no .NET)? Require to specify a length to the string you’re passing? Separate file names by some other character? What if teat character because valid in a future file system?

  7. googly says:

    @Marquees: CreateFileW.

    Use "\?\C:\very_long_path_to_file.ext" or "\UNC\very_long_path_to_file.ext"

    File name canonicalization will be turned off though, so you have to perform that yourself.

  8. Tadmas says:

    @Boris: If we had a time machine and redesigned the function, you could change the member from LPCTSTR to LPCTSTR* with an additional count member.  At least that way if you try to pass a simple string without reading the documentation, the compiler complains.

  9. Dan says:

    Simple debugging of this using Process Monitor would have revealed the FILE_NOT_FOUND generated on a query of the second filename, which would have been garbage.

  10. Peter da Silva says:

    20 years ago isn’t so long. This function is screaming for a null-terminated vector of string pointers, which has been pretty much the normal way to pass an array of strings in C since, oh, the early ’70s. About the only time I can recall seeing a double-null-terminated array since I’ve been writing in C is in implementation-dependent internals of the way crt0.o reconstructs argv on the PDP-11. In 1978 or 79.

    But then I’m an unreconstructed oldschool type who doesn’t see anything wrong with "foo" and "foo." being different files. So what do I know.

  11. Marquess says:


    Yeah, I’m aware of this. But as you pointed out, you have to canonicalize the file name yourself, and the OS itself could do that best. Now if there were a function for that …

  12. Steve says:

    It is my opinion that just because something has been wrong for 15 (or 20) years does not make it any less wrong.

  13. Kv3 says:


    I think what Raymond is saying is that while it may seem like a mistake now, it was considered fine back then. And anyway, changing it now would just break backwards compatibility(and annoy the people who already know how to properly use it) for no good reason.  

  14. Duke of New York says:

    20 years ago is a d— long time. Think about the computer specs that were available then.

  15. Christian says:

    double null terminated strings are great and efficient and I wrote new code using them last year

  16. Marquess says:

    Null-terminated strings are painfully inefficient. Finding the length of a string is an O(n) operation. It really should be O(1). And that’s only one of their problems.

  17. sideh says:

    Steve, it sounds like you have the time machine that Raymond keeps talking about.

  18. yuhong2 says:

    I remember the old Shell: Revealed blog mentioning this, seems like it is gone now.

  19. Cheong says:

    Yeah… It made me wonder why they don’t just use something like Pascal string for paths, back in the days when I first learn about Win16 APIs… (And yes, the programming language we were taught in school is Pascal…)

  20. Maurits says:

    It’s not a double-null-terminated string, darn it.

    It’s an empty-string-terminated list of strings.

  21. Will says:


    For long UNC paths you should use:


    Note that you can’t use DIR from CMD on that type of long path for some reason, though it works fine for the non-UNC long path version.

    The error being: The filename, directory name, or volume label syntax is incorrect.

    Not sure why DIR fails there since you can use FindFirstFile()/FindNextFile() just fine with that type of path (at least on Windows 7). Also not sure where a built-in canonicalization API is that can handle those long paths (I read the documentation, I know that the prefix turns it off, but is there something that can handle it maybe that you can call manually that won’t blow up because the path is too long? people doing their own canonicalization logic seems prone to failure).

  22. Christian says:

    I never need to get the number of strings in my double null terminated lists. I only traverse them. And enjoy freeing only one pointer. And there they are very efficient: malloc and free aren’t free, you know? They add several bytes of management data and fragment the heap.

    I just need to free one pointer to free the list.

    I even allocate memory for the filesize of a file with entries and then walk through the data once to remove e.g. newlines or filtered entries. At the end my destination pointer stops in the middle of the data and the rest is dead memory. But that doesn’t matter, soon the whole memery block will be discarded anyway

    Other parts of the code are using .net and not c…

    But I really believe that if you just need to traverse through lists of strings and then just need to traverse through the characters in these strings, that this data structure is very very efficient. Not just O(1), but only one traversal at all.

  23. Christian says:

    It’s not a double-null-terminated string, darn it.

    It’s an empty-string-terminated list of strings.

    They are actually differnet formats.

    That was discussed last time Raymond brought this format up (also the efficiency, especially with win16): It’s a common mistake to encode an empty list as or to require it that way. But it surely is more safe to terminate it with two all the time.

  24. John Topley says:

    @Raymond I wasn’t making a point, I was just interested because of find the history of Windows interesting; that’s why I visit your site. Next time I won’t bother asking.

    [I didn’t understand what your point was because you were asking a question that was already answered in the article… -Raymond]
  25. The Other Joe says:

    Double-null terminated strings are an invention of the Gods themselves. Compact and easy to parse; who can’t love em?

    Mike reminded me of something that has cracked me up for a long time. People who insist on checking error returns on functions that if they do fail mean you system is so screwed up there’s nothing you can do about it anyway. (Heck, even I do it sometimes and cackle evilly while doing so.)

  26. Mike Dimmick says:

    The odd thing is that few Windows APIs need to care about how long the string could be. Those that simply take a path as an input parameter – basically just CreateFile – could be modified so that the programmer could provide a non-canonical long path.

    On the other side, most functions that *provide* a path have a parameter indicating the size of the buffer that was provided. There are a few that don’t – GetTempFileName is one that I can think of, off the top of my head. A major one is the WIN32_FIND_DATA structure used by FindFirstFile(Ex), though IIRC this only gives the file name, not the full path. But I don’t think it would be too problematic, in the main, to modify the Win32 functions to support longer paths than the classic definition of MAX_PATH.

    However, many programs – even most – may have assumed they will never get anything bigger than MAX_PATH and so they may well not handle the errors that would occur if asking for the path of a file that’s longer than that. I admit that I do generally assume that GetModuleFileName(NULL) will not fail. It might be too large a set of programs to allow them to opt out through compatibility shims – it might have to be opt-in and even then it would be risky.

  27. Gabe says:

    It seems like John Topley was attempting to discern which version of Windows first privately used this API. Since that information was definitely not in the article, his post makes sense.

    [I omitted the details from the article on purpose. It’s not relevant to the story, and it hardly takes any effort to figure out what version of Windows publically supported SHFileOperation or what version of Windows preceded it. It’s the sort of comment I would have expected from Yuhong Bao. -Raymond]
  28. Aneurin Price says:

    Raymond: <implies that ShFoo is present as a private function in version X>

    Poster: Oh, that’s interesting – I thought the ShXXX functions were only present in version Y. Was I wrong then?

    Raymond: <overflowing contempt>

    It upsets me to see people treating each other so badly, especially when there is no reason for it.

    Please trying to remember that you are talking to actual people; if somebody expresses interest in something you find boring, isn’t it better to ignore it than to ridicule it?

    [The problem is that he was playing “out the function”, something that is explicitly forbidden by the ground rules. The vagueness was intentional. -Raymond]
  29. John Topley says:

    @Gabe, @Aneurin Don’t worry, Raymond just lost a reader, although I doubt he’s bothered since he seems to find people who comment on his posts a nuisance anyway.

    [Over the years I have been unfortunately become biased towards assuming that commenters are trying to prove something/boost their egos/be snarky/nitpicky, because annoying people are more likely to comment than friendly people. Sorry you got caught in the dragnet. (Fortunately this attribute of blog commenters doesn’t appear to apply in face-to-face communications.) -Raymond]
  30. John Topley says:

    Hi Raymond,

    Don’t worry about it. I have seen some of the other commenters that you attract on here, so I do understand why you assume that.

Comments are closed.