The redirection can come anywhere on the line, so watch out for those spaces


Wekcome to Batch File Week. Batch is probably one of the most hated programming languages that people are still writing new programs in. You hate it, but you have to deal with it. So just deal with it.

That said, here we go.

A customer couldn't understand why parenthesizing a few ECHO statements resulted in a syntax error.

This version of the batch file works:

@echo off
set log=C:\Space Path

echo Test >>%log%\Output.txt
echo Redirection >>%log%\Output.txt

On the other hand, this version, which should be equivalent, spits out a syntax error.

@echo off
set log=C:\Space Path

(
  echo Test
  echo Redirection
) >>%log%\Output.txt

The error is

Path\Output.txt was unexpected at this time.

If the syntax of the second command is illegal, then so too should the first, since they are basically the same thing. Why is one legal and the other not?

Recall that you're allowed to put the redirection operator anywhere on the line, and when it is parsed, it is removed from the command line, and what remains needs to be a legal command.

In the first batch file, the echo statement expands to

echo Test >>C:\Space Path\Output.txt

The redirection operator detector takes out the redirection operator and the file name (which due to the embedded space is parsed as merely C:\Space), leaving

echo Test Path\Output.txt

Result: The string Test Path\Output.txt is placed into the file C:\Space.

Now let's look at the second batch file. The command being parsed is

(
  echo Test
  echo Redirection
) >>%log%\Output.txt

which expands to

(
  echo Test
  echo Redirection
) >>C:\Space Path\Output.txt

Again, the redirection operator detector takes out the redirection operator and the file name, leaving

(
  echo Test
  echo Redirection
) Path\Output.txt

And that is a syntax error.

In other words, the first batch file works because the extra junk you appended happened to be in a place where junk was legal. (It gets treated as more arguments to the ECHO command.) But the second batch file puts the extra junk in a place where junk is not legal, and so it is rejected as a syntax error.

If you take a step back, you'll see that the real problem is that the batch file uses a path with spaces but fails to quote it properly.

Whose idea was it to allow redirection operators to appear anywhere in the line? Answer: Ken Thompson, in version 2 of sh. If you think this is a stupid feature, you can let him know.

Comments (53)
  1. Karellen says:

    This version of the batch file works:

    Hmmm…..only for suitably un-rigorous definitions of “work”. You talk around this, but don’t really make it explicit, that a straightforward reading of the code shows that it is supposed to write the lines “Test” and “Redirection” to the file “C:\Space Path\Output.txt”, whereas it actually writes the lines “Test Path\Output.txt” and “Redirection Path\Output.txt” to the file “C:\Space”.

    If that code “works”, it must surely be a promising candidate for the venerable International Obfuscated Batch Coding Competition! (Second in its class, with only the IOPCC above it for those truly wishing to risk their sanity staring into the abyss.)

    ObSnark – if DOS was actually influenced by ‘sh’, command options would have been preceded by ‘-‘, and the path separator would have been ‘\’.

    1. The MAZZTer says:

      I think it’s pretty clear by “works’ he means it runs.

      1. Karellen says:

        Yeah, but my point was that he doesn’t really make that explicit, and in my experience the most common definition of “works” as applied to software is “does what the author intended”.

        Granted, some code mostly does what the author intended but has bugs in edge cases, in which case you could say that it “mostly works”. That code is two lines long, and neither of them ever does what the author intended.

        1. Karellen says:

          Sorry, four lines long, and half of them never do what the author intended. (my mistake)

        2. zboot says:

          He also doesn’t teach a programming class before every post. There’s a level of understanding expected of readership beyond being capable of reading.

        3. Wear says:

          I did found it interesting that the customer reported the issue without ever checking the output file to see what was in it and I did notice that it was never explicitly stated that the first option doesn’t actually do what it looks like it should.

          But what’s being output to the file isn’t the focus of the article or the customers question. The second option doesn’t work because it throws a syntax error and in that context the first option does work in the sense that it doesn’t throw an error.

    2. Yuri Khan says:

      DOS inherited slash as the switch character from CP/M, and the backslash was chosen later for the path separator as the next most similar character.

      https://blogs.msdn.microsoft.com/larryosterman/2005/06/24/why-is-the-dos-path-character/

      1. Tom Passin says:

        Back in the early 1970s, there was a PDP-15 next to my desk. It wasn’t mine, but I did get to play with it from time to time. Ten years or so later, when I encountered CP/M, I realized that it seemed to be a nearly exact clone of the PDP-15 OS.

        1. John Styles says:

          Yes indeed. This fact seems surprisingly rarely remarked on. Fun fact, the 8.3 filenames in DOS come from CP/M. CP/M took the PDP 11 ones which were Radix 50 (50 in the sense of 40, obviously) with 3 characters fitting into a 2 byte int and had 6.3 (3 ints), got rid of the Radix 50 and added 2 characters to be generous.

    3. That’s because MS-DOS 1.0 (aka 86-DOS) was as pure a clone of CP/M as possible. MS-DOS 2.0 was largely rewritten in-house by Microsoft, which had already been reselling Unix as Xenix, and that heavily influenced the creation of new features in command.com. But the specter of backwards compatibility was already there by then, and it was too late for the basic syntax. (Until Windows came along.) Sh wasn’t exactly all that and a bag of chips itself back in the day, of course.

      If Microsoft had actually written MS-DOS itself, instead of buying and minimally modifying it, it probably would have much more closely reflected Unix in spite of its CP/M compatibility. I am saddened that DOS didn’t begin including more minimal Unix programs, like sed and grep, dd, etc, as drive sizes increased, but the Unix mindset really got siloed into the OS/2 and later NT teams.

  2. Ian Yates says:

    Link to “let him know” fails but it’s still in search engine cache.

    I only learnt about redirection being possible form anywhere in the line last week despite writing batch files for years. Someone suggested putting them at the start of the line which initially looked odd but does make a lot of sense.
    What wasn’t included elsewhere was how the redirection is effectively stripped out and then the rest of the line is interpreted. I like understanding that extra layer down so thanks for sharing in the post! ☺

  3. The MAZZTer says:

    I wasn’t even aware you could do that. Well I’m certainly not going to start now. Redirecting every line individually will at least make it clear where an error is.

  4. Stéphan Leclercq says:

    >> If you take a step back, you’ll see that the real problem is that the batch file uses a path with spaces but fails to quote it properly.

    If you take *TWO* steps back, you’ll see that the real real problem is that Microsoft should have immediately fired the guy who came with the idea of allowing spaces in a file name (or is it a filename ?)

    1. Yuri Khan says:

      If you take *three* steps back, scripting languages should have required string quoting from the start.

      1. pc says:

        And if you take four steps back, really it’s that Batch was never designed as a “scripting language” at all. It was just a way to run a “batch” of a few statements at once, with more and more features added such that what’s there now may be a scripting language but it sure has a lot of “gotchas”. I’m very excited to learn more during Batch Week.

        1. cheong00 says:

          Only if it’s really a batch of commands in same syntax.

          I wonder why we can just use %a in “for” statement in command line, but have to use %%a in batch files. Why can’t we just use the same statement?

    2. Entegy says:

      Users create file names with spaces. You should be thanking Microsoft for forcing this issue a long time ago and making developers deal with this situation a long time ago. If your script handles file paths, it will encounter a space at some point. You should be quoting any file path even if you think you’re 100% sure your script will never encounter spaces in the file path. Failure to do so is a failure of a script.

      1. cheong00 says:

        It reminds me the issue that tax return process program installer of certain bank can’t install on WinXP around 2004, and the issue is that the installer cannot handle space in path.

        Since the installer has a look and feel of installers at Win3.X days, I subconsciously replaced “Program Files” with “progra~1” and the installation just works. And later when staffs of the bank call me (actually it was redirected by accounting department) to know how can I install it, I just forgot it at the moment, and replied “I forgot but it should be trivial thing that I can redo if I was sent to install it again”.

      2. Stéphan Leclercq says:

        The shell already hides the extension for users that don’t care to understand them. It could also have displayed underscores as spaces, and replace spaces by underscores when the user types them in a file_name.

        1. xcomcmdr says:

          Yuck !

        2. alegr1 says:

          Whoever came with the idea of hiding file extensions from the user should have beer fired immediately.

          1. Stéphan Leclercq says:

            Whoever came with the idea of file extensions should have been fired. Yes !! I mean you, Gary Kildall !!! :-)

          2. Brian says:

            Gary Kildall didn’t come up with the idea of file extensions. They were there in all those earlier DEC operating systems.
            The “Make” utility (for describing how compilers, linkers and other tools are used to “build” code) would be much harder without file extensions.
            Files need some kind of file-type information. Apple hides it away. DEC OSes, CP/M, DOS and Windows all make them right there in your face. I for one like the ability to change a file type simply by editing its name

          3. Stéphan Leclercq says:

            Don’t get me wrong. I do agree that extensions are a useful way of categorizing files when you have multiple representations of the “same” data such as source code vs compiled code, editable document vs pdf redistributable, etc. Requiring every file to be explicitly categorized according to its format and failing to open it if the categorization is wrong, well, that’s more objectionable…

            Would you be happy if the C compiler rejected your code because you failed to explicitly use hungarian notation on each and every variable that you passed to a WIN32 API function call ?

        3. morlamweb says:

          So, rather properly support spaces in file paths and names, you’d rather pile a kludge on top of a kludge and hide MORE information from the user? Sorry, 1985 called; they want their filesystem back : )

          1. Stéphan Leclercq says:

            No. I would *love* to properly support spaces, backslashes and colons into filenames. Unfortunately that would require so many changes in the way command line parsers work, that most of the batch files ever written would have to be changed. The risk is just too high compared to the usefulness of the feature. No one objects when spaces are not allowed in email addresses, web addresses, etc. File names are computer stuff, file names containing spaces are hard to parse for computers and for humans alike (have you ever received an e-mail requesting you to read the readme text file? Does that refer to “readme” file, or the “readme text” file ? – I know my example is contrived but you get the idea) Every time you cannot easily tell apart spaces used as part of a name or as separator, you are in trouble. Shell interpreters are such places. I can come up with so many examples where having to selectively quote names… You need to quote “c:\program files\”, but quoting “*.txt” would stop the globber in Unix shells to expand the command line. When you use options, should you quote them as a whole, only the value ? Some programs would require /f=”x y.txt” while some others would require “/f=x y.txt”.

            It’s easy for you to say “get rid of all programs that do not properly handle spaces in the filenames in their command lines”, but how do you actually do that in practice when you need to maintain legacy applications ?

          2. morlamweb says:

            @Stephan Leclercq: so, you start the discussion on spaces in file names and paths, and then expand it to include backslashes and colons? How does that make sense? Colons and slashes have long-established roles to play in file paths and are fundamentally different from spaces.

            Applications have had decades to get on board the spaces-in-file-paths train. There’s no going back now.

          3. Stéphan Leclercq says:

            Nice way of not answering the question, thanks. There are some characters that you cannot use in file names. Colons, slash, backslash, greater than, smaller than. For a regular user, there is no fundamental reason why you can’t use them, and people have come to accept that “because this is computer stuff”. What is so special about spaces, except that you can’t see nor hear them and so cannot easily tell the difference between no space, one space and two consecutive spaces, scratching your head on why you cannot open your file. Raymond’s week of batch is filled with examples of programs that behave strangely in presence of spaces, it wasn’t worth the headaches of understanding why a poorly written BAT stops working because an unrelated poorly written BAT just created a file named “c:\program”. Just explain to users that spaces are not allowed in file names just as they are not allowed in email addresses and they will go along just fine.

  5. Muzer says:

    People have justified the placing of redirection operators at arbitrary points on the line by saying it can make pipelines look neater. Eg something like:

    outfile

    Not sure if this was considered at the time or just a good reason in retrospect.

    (I also find it useful to write things like:

    echo >&2 some stuff
    echo >&2 some much longer stuff

    …rather than:

    echo some stuff >&2
    echo some much longer stuff >&2

    …as it means all the redirections line up and it’s much more obvious if you miss one).

  6. Peter Doubleday says:

    Arguably the “stupid idea” was to use “set” as a sort of hash-define, rather than as a “let” alias, if you see what I mean. (Still an identical issue across both DOS and Unix, I believe.) A reasonable parser (for some value of “reasonable”) would be able to correctly handle the token-pair representing redirection — this pair being the redirection operator and the path-name. No need to “strip it out,” or at least none until the whole rule/line has been parsed.

    Then again, the resource restrictions faced by either early Unix shells or the original DOS batch language were pretty extreme. A full-scale parser with a complex error-handling system was clearly out of the question.

    1. On the other hand, treating it as a macro preprocessor lets you do things like “set options=–long –list –of –options” followed by “some-command $options blah blah”.

      1. Yuri Khan says:

        By a mysterious coincidence, I read just today of a shell that treats $variable expansions as single units by default, and has special syntax $@variable for “exploding”, i.e. splicing the elements of a list variable into the list where the variable is mentioned.

        https://elvish.io/ref/language.html#use-explosion

        1. Voo says:

          Splatting/exploding parameters is actually quite a common feature. Lots of mainstream languages support it for good reason. Perl, python, PowerShell to name just a few.

      2. Peter Doubleday says:

        I suspect (without thinking too hard about it) that using [b]set[/b] as an aliasing device would also allow you to declare that long list of options. Or indeed any other similar requirement.

        Essentially, it’s all up to the back-end of the parser, isn’t it? Once you’ve identified your command, and accepted the fact that there is an [b]%alias%[/b] construct following it, all you really have to do is to expand the alias.

        The difference between this and a define-macro usage is that you are doing the expansion at the appropriate time.

        (Having said which, since AFAIK spaces were not allowed in FAT path-names, the issue obviously did not arise.)

        1. Ivan K says:

          Apparently they were allowed, but almost nothing supported them. https://blogs.msdn.microsoft.com/oldnewthing/20090709-00/?p=17563/

          1. ErikF says:

            Most DOS system utilities, let alone applications, wouldn’t handle spaces (the quoting system only appeared in Windows 95), so putting spaces in a filename was a recipe for disaster! I accidentally put a space in a file (I think it was using GW-BASIC, but the details are fuzzy now), and the only way of doing anything with it was to “REN FILE?NAME FILENAME”. However, Alt-255 was a handy way to put a pseudo-space in a file name.

  7. Stefan Kanthak says:

    Regarding stupid features: the customers stupid idea was not to enclose all pathnames in quotes (except in the SET statement).
    https://support.microsoft.com/en-us/kb/102739 is about 25 years old

    1. Ray Koopa says:

      whenever i see paths not quoted, i stop looking any further, quote them and run again.

      1. Stefan Kanthak says:

        Did you fix the unquoted pathname in an out-of-process COM server of Windows Defender that was introduced with the Creators Update already?

  8. Ken Hagan says:

    “ObSnark – if DOS was actually influenced by ‘sh’, command options would have been preceded by ‘-‘, and the path separator would have been ‘\’.”

    The key word was “influenced”. Another, perhaps more important influence, was IBM’s JCL, which used ‘/’ to introduce parameters. DOS 1 inherited that, and consequently when DOS 2 introduced sub-directories it had to use ‘\’.

    Yes, it is all rather sad. No, MS still haven’t finished that time-machine yet.

  9. voo says:

    Every time I see such things or have to read old batch scripts I’m so happy that I can use PowerShell for all scripts these days. Yes poor soul if you have to support Vista you’re screwed, but if not and you’re writing a new script there is just really no reason to write it as a batch script instead of in PowerShell*.

    Hell at work we’ve been gradually replacing all the old build scripts with PowerShell replacements over the last few years and it has made life tremendously better. Particularly being able to host PowerShell modules on our company NuGet server is great for shared functionality.

    *wrappers to simplify calling the script excluded.

  10. Yukkuri says:

    I, for one, am excited for batch week

    1. Yukkuri says:

      Also it is interesting that the early sh had a hard-coded search path

  11. alegr1 says:

    Helpful tips for quoting paths:

    1. Since BAT/CMD file arguments can be paths in quotes, but you can’t be sure about them, you can unquote them and requote where necessary by using tilde:

    %~1 – will give you the first parameter unquoted.

    2. If you have to invoke CMD.EXE explicitly and supply it a command filename, be aware that it cannot handle the quoted filename there. You’ll have to convert it to short path.

    3. If you *have* to use unquoted name somewhere, you can make a BAT subroutine (call of the internal label) to convert it to short name which doesn’t require quotes.

    As a brag, I wrote the entire script to index PDBs sources for Git, as a CMD file. Although I later rewrote most of it in Bash, to allow parallel invocation of git commands, to speed it up.

    1. laonianren says:

      For point 2 you might think you need some kind of elaborate quoting but you don’t. This works:

      cmd /C “”bat spc.bat” “p1 spc” “p2 spc””

      Other weirdness: If “bat spc.bat” contains

      echo %1
      echo %2
      echo %*

      Try running:

      cmd /C “bat spc.bat” bad^^^^ param

      The ^s eat one of the newlines and you get:

      bad@echo param
      bad param

      1. laonianren says:

        Oops! Each of those echoes should be preceded by an @.

    2. Might be worth noting that a few OEMs and a lot of administrators explicitly disable short file names in policy. Your comment is literally the first time I’ve heard someone bring up short names in years.

      If you’re using a fundamentally broken program (anything that doesn’t support quoted LFNs), you’re definitely in the weeds of creating a local environment workaround, though. If short names aren’t available, you get into having to rename a file, or symlink it, or stage it into a temp folder….

  12. florian says:

    Only the smartest and most elaborate (and therefore often the most unobtrusive) programs are able to handle unquoted file names with spaces — the most recent one I’m able to test right now is notepad.exe from Win7/SP1.

    Woohoo, batch weeek !!!

    1. Stefan Kanthak says:

      While NOTEPAD.exe handles both quoted and unquoted pathnames with and without spaces, quite some DLLs invoked via RUNDLL32.exe fail with quoted pathnames in their command line.
      Try for example RUNDLL32.exe CRYPTEXT.dll,CryptExtOpenCer “.CER”

      1. RUNDLL32 works mostly on hopes and prayers if the API isn’t actually designed for it, and even if it is, not all APIs are created equal. Some can handle quoted filenames, some expect a single unquoted filename even for names with spaces (which will work!).

  13. Neil says:

    Redirection operators can appear almost anywhere in a line… but I once accidentally wrote timeout/t 1>nul but that doesn’t do what I want of course,

    1. Neil says:

      … as you can see when you read the next episode. Spooky!

Comments are closed.

Skip to main content