Beware of digits before the redirection operator


If you want to put the string “Meet at 2″ into the file “schedule”, you might be tempted to use

echo Meet at 2>schedule

If you try this, however, you’ll see the string “Meet at” on the screen and the “schedule” file will be blank. [Typo fixed, 10am]

What happened?

A digit immediately before a redirection operator modifies which stream the redirection operator applies to. If you’re going to redirected an alternate output stream, it’ll nearly always be the standard error stream, or stream 2. To put the error output into a file, you would write something like this:

sort /invalidswitch 2>errorfile

There is also the operator “>&” that reopens a stream as another stream. The idiom

some-command >output 2>&1

says, “Put the normal output into the file output, and then change the error output stream (2) to refer to the normal output stream (1).” The result is that both the regular output and error output end up in the output file.

But what if you really want to put the string “Meet at 2″ into the file “schedule”?

You can insert a space between the “2” and the “>”. This works for most programs since they ignore trailing spaces on their command line, but this was a trick question: The echo command is one of the few commands that actually pays attention to trailing spaces. As a result, the contents of the “schedule” file is “Meet at 2<space><cr><lf>”. Maybe this is close enough for you, in which case you can skip the next paragraph.

But what if you don’t want that trailing space? For that, you can use the metacharacter escape character, the ^:

echo Meet at ^2>schedule

The last gotcha is that the pesky “2” might come from environment variable expansion.

set message=Meet at 2
echo %message%>schedule

The trailing “2” in %message% interacts with the greater-than sign, leading to an unintended redirection. For this, you can insert a space before the greater-than sign, assuming you are in a scenario where that space is not going to cause you any problems. (And if you’re in a scenario where that space will cause a problem, you can use a trick we’ll look at next time.)

Mind you, if you’re going to take an environment variable whose contents you do not control and expand it onto your command line unquoted, you have much worse problems than a trailing digit messing up your file redirection. Somebody might have decided that the message should be “&format C: /y“. Inserting this into the command line unquoted would yield “echo &format C: /y>schedule” which is a pretty good way to ruin somebody’s day. (Well, okay, you can’t format a drive with an active pagefile, but you get the idea.)

Comments (29)
  1. BryanK says:

    If you try this, however, you’ll see the string "Meet at 2" on the screen and the "schedule" file will be blank.

    The "schedule" file would be blank, yes.  But wouldn’t you see "Meet at " on the screen, not "Meet at 2"?  (In fact I just tried it on 2K Pro; the "2" is not printed.)  ;-)

  2. Adam says:

    Raymond> As a result, the contents of the "schedule" file is "Meet at 2<space><cr><lf>"

    Wow. That made me think "Where does the <cr><lf> come from?" as I assumed that cmd.exe would strip everything after the ">" character and probably barf on anything after the filename.

    So, I tried:

    c:>echo foo 2>wibble bar

    foo  bar

    c:>

    !

    Then:

    c:>echo foo >wibble bar

    c:>type wibble

    foo  bar

    c:>

    I never even guessed it might do that before!

  3. Carlos says:

    Escaping command lines is a chore.  First of all you have to guess that the target app is using CommandLineToArgvW, and then do the reverse of CommandLineToArgvW’s arcane unescaping rules.  (Backslash is the escape character; always escape quotes; only escape backslashes if they precede a quote.)

    And if you’re passing the string to cmd.exe (rather than CreateProcess) you also have to escape special shell characters with ^.  Many apps get this wrong.

  4. Arlie Davis says:

    cmd.exe must go.

    If only MS had just adopted a decent shell a long time ago…

    Yes, I’m aware of Raymond’s recent criticism of Monad / Power Shell.  And I don’t know if Monad is any good.  But anything has got to be better than cmd.exe.

    If you want to lose your mind, investigate how aliases, tab-completion, and command history work.  It’s an unholy collaboration between cmd.exe and the CSRSS itself.

  5. Adam says:

    BryanK> "I’m pretty sure the <cr><lf> comes from cmd.exe’s echo builtin"

    So it does. And always has done. Including the UNIX /bin/echo and all shells builtins that I’m aware of.

    How did I forget that? Must be getting old…

  6. oldnewthing says:

    Arlie Davis: It wasn’t a criticism of PowerShell. It was just pointing out that "Here is a replacement shell" is harder than it looks. The remarks apply just as equally to rc, bash, csh, etc.

  7. strik says:

    This echo discussion reminds me of a problem I encountered lately: Is there a built-in possibility NOT to output the CR/LF at the end?

    I have yet to find a solution (other than using echo from Cygwin, that is).

  8. josh says:

    I forget where I picked it up, but you can actually put the redirection on the left side of the command:

    >schedule echo Meet at 2

    This is particularly nice if you’re creating a file with more than one line, because things line up better.

  9. pavolmarko says:

    Is there a built-in possibility NOT to output the CR/LF at the end?

    This site has some options: http://www.ericphelps.com/batch/lines/

  10. josh says:

    @strick:

    From echo, I don’t think so.  In the old days I believe the trick was to use "prompt" instead, which got ugly.  It looks like you can use set/p to do it as well, which is less nasty.

    echo|set/p=text but no newline

  11. strik says:

    pavelmarko: Thank you for the link. Unfortunately, these do not seem to work to write arbitrary texts, like e.g. echo -n on Unix does.

    And: Yes, after writing my post I found out that BryanK already mentioned this problem here. :)

  12. stri says:

    @josh: Thanks, this is it (almost). This set/p trick adds a SPACE after the line, but besides from that, it works as expected.

    I will have to find out if the additional SPACE is a problem in my case.

    Again, thank jost and pavelmarko for your tips!

  13. For those of you who are more interested in the gritty details of command line redirection, RaymondC…

  14. Gabe says:

    According to O’Reilly’s "Learning the bash Shell" (by Cameron Newham & Bill

    Rosenblatt), the command line is parsed by a 12-step process, as follows:

    1) Split command line into tokens by metachars (space, tab newline, ;, (, ), <, >, |, and &).

    2) Handle compound (block) command keywords (or {} structure).

    3) Handle command aliases.

    4) Handle brace expansion.

    5) Substitute home directory for ~ or ~user shortcuts (tilde expansion).

    6) Perform $varname variable substitution (parameter expansion).

    7) Perform $(cmd) command substitution.

    8) Perform $((expr)) arithmetic substitution.

    9) Retokenize the command line by $IFS delimiters (word splitting).

    10) Perform pathname wildcard expansion (similar to MSVC’s wild.c).

    11) Perform command lookup as function, built-in, or executable file via $PATH.

    12) Run the command, after setting up I/O redirection, etc.

    Can anybody produce a similar list for cmd?

  15. BryanK says:

    I should note that while cmd.exe expands environment variables before breaking up commands (so you can put the & in there, and cmd.exe will run two different commands), bash (at least v3.1 from Cygwin) does not:

    $ VAR="& ls"

    $ echo $VAR >file

    $ cat file

    & ls

    $

    After $VAR is expanded, echo is given two arguments: "&" and "ls".  (If $VAR had been in quotes, then echo would have been given one argument, "& ls".  Doesn’t matter though.)  The shell does not interpret (or reinterpret) the command line after substituting variables.  (If you *want* it to, you have to use eval.)

    And it works the same way whether you use the shell builtin echo or the /bin/echo program, so it’s not something special the shell does.  It’s just that the shell interprets ; and & (I’ve tried this with semicolons as well) before it does parameter expansion.

    Adam — I’m pretty sure the <cr><lf> comes from cmd.exe’s echo builtin — at least, that’s the way bash’s echo builtin works (and /bin/echo as well).  If you want to suppress the line-end, you can give bash’s echo (or /bin/echo) the -n option.  I don’t know if cmd.exe’s echo has a similar option.

  16. mikeb says:

    Just an additional bit of info on the >& operator.

    As Raymond says:

    … some-command >output 2>&1

    will redirect both stderr and stdout to the file ‘output’.

    However,

    … some-command 2>&1 >output

    will not – only stdout will be redirected to the file ‘output’.  Stderr in this case is still directed to the console.  This confused me at first.

    It seems intuitive (to me anyway) that the above line would redirect stderr (2) to stdout (1) then redirect stdout to the file – so everything should go into the file, right?

    But what is really happening is that stderr (2) is redirected to whereever stdout (1) is going *at that point in the cmd line*.  So stderr is directed to the console.  Then stdout is directed to the file, but that does nothing to the current state of stderr (so it’s still directed to the console).

  17. josh says:

    @stri:  If the space is a problem, try adding parentheses:

    (echo|set/p=foo)

    Weird that it was adding a space only if you redirected the output.  Tacking on "&echo etc" wouldn’t have the extra space.  Parens seem to make it fine either way.

  18. Perf Wiz says:

    I don’t have a pagefile. I just buy more RAM and write small programs.

  19. KJK::Hyperion says:

    BryanK: actually now (as of Windows 2000) both expansion styles are possible. %VAR% is expanded first, and the new style !VAR! is expanded last

    Arlie: not so evil, just an undocumented parameter (the reserved parameter to ReadConsoleEx) that tells the console implementation what control characters terminate line-based input early

    Gabe: I reverse-engineered the steps a long long time ago, thanks to a funny quirk in the language. Every step has a distinct set of "special" characters, and for each step the caret only acts as an escape character if followed by one of those, and is ignored otherwise. I don’t remember the *exact* steps now, but the first ones are:

    1) join multiple lines (caret escapes newline)

    2) expand %VARIABLES% (caret escapes %)

    3) …

    It gets really horribly painfully complicated from here by the fact that built-in commands and external commands are treated very differently. For starters, they use different tokenizers (that’s why "cd.." is parsed as "cd .." and why "cmd.exe" instead stays whole). Then, most built-in commands have their own command-line parser, some changing the syntax considerably

    The non-obvious example is "(" – yes, open-parenthesis is a command, yes you can do most "command" stuff with it (included I/O redirection, which unfortunately can get very quirky if you get too creative, like if you try to pipe one "(" into another, and I don’t really recommend it), and easily the most powerful one. Its parser digests anything up until an unescaped ")" that occurs outside another built-in command (built-in command parsers nest), and parses multiple lines as multiple statements. It’s also the most reliable way to delimit a statement, such as the "true" branch of an IF

    It’s… well… you don’t really want to know. You just "get it" after some heavy batch file development, I think

  20. And this is why I flat out refuse to do any more shell programming, on any shell, ever. 30+ years of kludges upon kludges upon kludges just doesn’t appeal to me as an environment to do anything in.

    A coworker and I once tried to come up with a Bash script that could be safely and directly used as a CGI script, without enabling command injection. We think we actually managed, but A: We’re not actually *sure* and B: even if we were right, it sure was hard.

    (I know bash isn’t the topic at hand but the general principle holds.)

  21. strik says:

    @josh: Thank you, the parenthesis solution is it! Yes, doing some more tests, I also found out that this extra space is not only added. I did some tests with "od" (from Cygwin) and found out that is does not contain that extra space, either.

    Again, thank you!

  22. Commandlinux Anonymous says:

    The command line is parsed by a 12-step process.

    First step: Admit you have a commandline problem.

    Second step: ?

  23. It doesn’t have to come at the end.

  24. Centaur says:

    OK. So what’s the proper way to evaluate an environment variable in cmd so that it doesn’t enable command injection? Assuming that the variable might contain redirection characters < > >> >&, piping characters |, unconditional sequence characters &, conditional sequence characters || &&, and whatever else cmd supports?

  25. BryanK says:

    If the shell does variable substitution before breaking up the command line into individual commands, then there’s no way to do it.  (Short of trying to sanitize the variable, that is.  If you can somehow magically escape the redirection, piping, and sequence characters, it might work.)

    Actually, it might work to put the variable expansion in quotes; let me try that…

    Well, sort of.  If  you say:

    set VAR=^&^ dir

    and then:

    echo %VAR%

    you will get:

    ECHO is on

    <normal output of DIR command>

    However, if you say:

    echo "%VAR%"

    you will get:

    "& dir"

    as output.  That’s probably not any better (the quotes should probably not be there), but at least it didn’t evaluate the sequence character.

  26. Philip says:

    I have a query about echo and escape character, basically I am trying to echo the word ON to a file as in

    echo ON>some file

    to achieve a result of

    ON

    …however since the command becomes echo ON I get a blank file. I tried using the ^ before ON as an escape character, but that doesn’t work.

    any input?

    Thanks/

  27. Stefan Kanthak says:

    @Adam:

    as you already noticed

    echo first >stdout second

    writes two spaces between first and second.

    Compare that against

    >stdout echo first second

    echo >stdout first second

    echo first second >stdout

    and note that CMD.EXE apparently does not collapse multiple spaces like a POSIX shell would do.

    %* in a batch file also differs from $* in a POSIX shell script.

  28. Norman Diamond says:

    Friday, May 19, 2006 7:09 PM by Philip

    > any input?

    No but you have to pretend there’s some:

    (echo | set /p =ON) >abcd.txt

  29. It’s the general-purpose escape character.

Comments are closed.