# The case of the system() call that returned before finishing

A customer was having trouble with some code that copies a bunch of files and then processes the copy. Here's the code they shared (suitably redacted):

// copy the files
char command[512] = {NULL};
sprintf_s(command, 512, "copy /Y %s %s >nul 2>&1",
sourcePath, destPath);
printf("The command is %s\n", command);
Log("The command is %s", command);
system(command);

// process the files
char searchPattern[256] = {NULL};
sprintf_s(searchPattern, 256, "%s\\*", destPath);
printf("The directory is %s.\n", searchPattern);
Log("The directory is %s", searchPattern);

hFile = FindFirstFileA(searchPattern, &FindFileData);

// error when searching files
if (hFile == INVALID_HANDLE_VALUE) {
DWORD lastError = GetLastError();

if (lastError == ERROR_FILE_NOT_FOUND) {
printf("No files under directory %s.\n", searchPattern);
Log("No files under directory %s", searchPattern);
return S_OK;
} else {
printf("FindFirstFile failed in directory %s with error: %d.\n",
searchPattern, lastError);
Log("FindFirstFile failed under directory %s with error: %d",
searchPattern, lastError);
return E_FAIL;
}
}

do {
printf("The file is %s.\n", FindFileData.cFileName);
Log("The file is %s", FindFileData.cFileName);
...


The customer reported that "It appears that Find­First­FileA does not wait for system() to finish copying the files. Here's a sample log file:

The command is copy /Y \\server\path\to\data\* D:\data >nul 2>&1
The directory is D:\data\*
The file is .


"Observe that Find­First­FileA did not find the files we copied. How can we wait for the system() function to finish copying the files before the program proceeds to the Find­First­FileA?"

In the ensuing discussion, people suggested using Copy­File or SHFile­Operation instead of shelling out to cmd.exe. Issues with spaces and other characters in the directory names. But can you find the reason why Find­First­FileA couldn't find the files?

Look carefully at the last line of the log: "The file is ." Part of this is confusing because the program both prints its output to the screen as well as to the log file, but prints them differently. The output to the screen includes a period at the end; the output to the log file does not.

And that's the key. Since the output to the log file does not include a period at the end, it means that the period in the output is the actual contents of FindFileData.cFileName.

The customer misread their log file. The issue isn't that the Find­First­FileA ran before the files were copied. The issue is that the first thing found by Find­First­FileA was the file whose name consists of a single period.

Recall that every directory has two artificial entries, one for the directory itself (.), and one for the directory's parent (..). What you found was the first artificial entry, the one that represents the directory itself. Instead of giving up right away, keep looking, and the files you copied will show up later.

(Assuming they were all successfully copied. The program doesn't actually check.)

Tags

1. Kirby FC says:

Your detective work is better than most TV shows. You should be a character on CSI.

1. guest says:

CSI = Computer Science Investigation?

1. Ray Koopa says:

I’m still for ‘Call Stack Investigation’

I want to like your comment but can’t. So here’s <3 <3 instead.

2. The MAZZTer says:

As a side note, I prefer to use libraries whenever I can, but when a good library isn’t available it is nice to be able to launch an external process to handle, say, archive operations (7z.exe) or manipulating a connected Android device (adb.exe/fastboot.exe). Heck, there are whole scripting languages built on that concept (bash, batch, powershell… sorta).

Since you’re using an external process, you gotta have your 7-zip archive or extracted files already on disk for 7z.exe. If you had a library you could skip that step and work on archives/extracted files in memory. Being able to talk to the Android device directly instead of creating a new process for each command could significantly speed up what you’re doing depending on what it is.

All that said I’m not going to defend what was done here with “copy”. Ick.

3. Roger says:

On Unix systems the filesystem really does have the . and .. as directory entries (implemented using hard links). Windows filesystems pretend the entries are there, but they aren’t physically stored. If you work on cross platform fileservers this is one area you have to correctly map semantics. The single most complicated piece of code I implemented was listing a directory, having to rewrite the code entirely 4 times! The samba guys even wrote a proxy that would send directory listing operations to multiple different Microsoft products and compare the results (yes they differ). The complexity comes down to these artificial entries, ordering, wildcards, case insensitivity, the listing being resumable, different limitations on file name length and characters, artifacts of 8.3 names and the list goes on.

Who would have thought that listing a directory is that complex!

1. Neil says:

Novell Netware servers didn’t have . and .. directory entries either, but they provided a client side emulation NET.CFG parameter SHOW DOTS=ON (there was also a software interrupt you could invoke to dynamically change the setting).

4. Paul Topping says:

I always hated that “feature” of directory enumeration APIs. Why enumerate . and . . when every directory contains them? Sure, I know there’s a reason, though I have forgotten it, but it is so crazy to have to write code to ignore them.

1. Roger says:

The reason is very simple. When the Unix guys did their filesystem originally, they explicitly added . and .. as real directory entries in the filesystem. That meant the path parsing was simpler because it didn’t need special code to treat . and .. differently – they just worked due to really being in the filesystem. Users and programs found . and .. useful. Consequently systems since then have made those part of dealing with paths in whatever way makes sense to them.

As for returning them in enumeration it is a sensible decision since users and programs really can use those as part of a path and it would be a bit strange if they didn’t exist, requiring every program to have to deal with it. For example a user or program can specify a path like “/a/b/../c/d” and every one of those components appears in directory enumeration. If they didn’t then any program dealing with paths would also have to add in extra code to special case them.

TLDR backwards compatibility with a convenience feature from the 1970s.

1. Henri Hein says:

I’m not quite following what the possibility of a non-normalized path has to do with directory enumeration. Regardless, I’m with Paul Topping. As far as I can remember, every single FindFirstFile/FindNextFile loop I have written have had code to ignore ‘.’ and ‘..’

1. smf says:

I don’t understand how someone can read this blog and yet wonder why it was implemented like this.

1. Henri Hein says:

There’s understanding and then there’s liking.

2. smf says:

. & .. are real entries in some filesystems that are supported by Windows: http://www.tavi.co.uk/phobos/fat.html

So you might not have a use for it, but other people do whether you like it or not.

Thank god Directory.GetFiles() doesnt include them.

–> Directory.GetDirectories()

2. Roger says:

You can split the pathname into components. You can list parent directories to see if components are listed. For example this is useful if a pathname is provided to a system call, you get an error back (“no such file or directory”) and are trying to work out which of the components is the problem since it may be a parent component rather than the leaf.

A tool can not depend on . and .. being there anyway. So, you can’t use this as free lunch feature to enable the user to navigate to the current or parent directory anyway. Because file systems doesn’t give any guarantees that those entries are there. For example, microsoft’s SMB shares doesn’t include them, and most people navigate SMB folders daily. All apps has to have special handling of . and .. anyway (the dir command in cmd.exe adds them explicit when enumerating a SMB share).

Adding . and .. entries, sometimes, doesn’t solve anything for any app, it is just an extra tax for all app developers. Maybe it helps file system developers, which is a 0.01% minority of all develops. This is not a good design decision, it’s a bad one.

1. zboot says:

Right. Because hardly anyone uses the work of those 0.01% file system developers.

5. Henri Hein says:

Your title brought to mind the Jeff Dean fact about him optimizing a function so that it returned before it was invoked.

6. Mitchell says:

I think they should’ve been able to figure this one out on their own. Anyone stepping through FindFirstFile would’ve seen the behavior (or their subsequent logging steps should’ve shown it?). I know I’ve seen this many times and just written code to ignore it.

1. anai says:

Yes, we only want files so !(FindFileData.dwFileAttribute & FILE_ATTRIBUTE_DIRECTORY) in there somewhere, ‘standard’ stuff once you get caught once….

But secondary to the point of the piece, which I think is that the difference in outputs coincidentally hides the FindFirst behavior when examining the log.

Final thought, surely “The file is ..” on screen (or wherever stdout ‘goes’) would make the viewer wonder.