Environment variable expansion occurs when the command is read


On the command line (and in batch files), environment variable expansion occurs when the command is read. This sounds obvious at first, but it has its own consequences.

In the online documentation for SET, one such consequence is spelled out:

    set VAR=before
    if "%VAR%" == "before" (
        set VAR=after
        if "%VAR%" == "after" @echo If you see this, it worked
    )

would never display the message, since the %VAR% in both "if" statements is substituted when the first "if" statement is read, since it logically includes the body of the "if", which is a compound statement.

In other words, the "if" command is not complete until the closing parenthesis is read. You can see this if you type the commands interactively:

C:\>set VAR=before

C:\>if "%VAR%" == "before" (
More? set VAR=after
More? if "%VAR%" == "after" @echo If you see this, it worked
More? )

C:\>

Notice that the "if" command didn't execute until you closed the parenthesis; the command interpreter kept prompting "More?" to collect the body of the "if". This means that everything you type as the body of the "if" is evaluated before the "if" condition or any of the lines in the body are evaluated. It's as if you had typed

C:\>if "before" == "before" (
More? set VAR=after
More? if "before" == "after" @echo If you see this, it worked
More? )

Note that this is different from most UNIX shells, which do not expand environment variables until the enclosing command is executed. For example,

$ var=before
$ var=after; echo $var
after

Notice that the $x is not expanded until the echo command's arguments are being computed. The analogous commands in the Windows command interpreter result in something quite different:

C:\>set VAR=before

C:\>set VAR=after & echo %VAR%
before

That's because the command interpreter expanded the environment variables at the time the line was read (not at the time the line is executed), yielding

set VAR=after & echo before

As a result, the old value of VAR is echoed. Some people treat this as a feature, allowing them to "restore" a variable without having to save it anywhere:

set VAR=newvalue & call helper.cmd & set VAR=%VAR%

This command sets the VAR variable to a new value, calls helper.cmd (which presumably uses the value of the %VAR% variable to control its behavior), then magically restores the variable to its original value since the %VAR% is expanded early, producing the old value.

But what if you want the variable to be expanded at execution time rather than at parse time? For that, you use "delayed expansion", which is enabled by the /V command line option or by using the SETLOCAL ENABLEDELAYEDEXPANSION command in a batch file.

C:\> copy con "%TEMP%\helper.cmd"
SETLOCAL ENABLEDELAYEDEXPANSION
set VAR=before
set VAR=after & echo immediate:%VAR%, delayed:!VAR!
ENDLOCAL
^Z
        1 file(s) copied.

C:\> "%TEMP%\helper.cmd"

C:\>SETLOCAL ENABLEDELAYEDEXPANSION

C:\>set VAR=before

C:\>set VAR=after   & echo immediate:before, delayed:!VAR!
immediate:before, delayed:after

C:\>ENDLOCAL

Immediate expansion is performed with percent signs, whereas delayed expansion is performed with exclamation points.

Why is immediate expansion the default? Because prior to Windows NT, that was the only type of expansion supported by the command interpreter. Retaining immediate expansion as the default preserved backwards compatibility with existing batch files. (The original command interpreter was written in assembly language. You really didn't want to be too clever or it would make your brain hurt trying to maintain the code. An interpreter loop of the form "Read a line, expand environment variables, evaluate" was therefore simple and effective.)

Armed with this understanding of immediate versus delayed expansion, perhaps you can explain what is really going on here. (Hint: It has nothing to do with ERRORLEVEL.)

Comments (27)
  1. Anonymous says:

    So the problem was that ERRORLEVEL was 0 at the point the batch script was loaded?

  2. Anonymous says:

    No, at the point the line – which spills onto multiple lines by using ( ) – was read.  Think of it as:

     for … do IsVarMap.exe … & echo %ERRORLEVEL%

  3. Anonymous says:

    Are "procedures" inlined/precompiled too (I suspect not)? If not, could one potential solution be to "call" a "procedure" (a label) in the same batch file, assuming this is for NT5 or later?

    In a way, it reminds me of the pattern of C++ std::for_each, where you give start, end and a function to call for each iteration.

  4. Anonymous says:

    Mike: "call :label" effectively reparses the batch file and resets the state – even the %0, %1, … constants are reset. You even have to call "setlocal" at the beginning of a procedure, or your variables are going to spill into the caller’s context

    So, yes, you can "call :label <arguments>" in a loop, even recursively, but I don’t recommend it, as string escaping gets funky across a call

  5. Anonymous says:

    Must…  destroy…  CMD.EXE…

    Destroy…

    too late…

  6. Anonymous says:

    The issue arises when you start programming in a language not originally intended for programming.

    “Proper” programming languages tend to be a little clearer about the difference between lexical processing and command interpretation.  And anyone who’s used a decent macrogenerator (not the C preprocessor) gets the concept.

    But command languages originally designed for people to type tend to create a little more confusion.

    [Very true. I doubt anybody actually enjoys writing batch files. It’s not so much a programming language as a script. -Raymond]
  7. Anonymous says:

    For extra pain – er, credit – try a .cmd script which generates registry entries (with variable substitution) which will then be executed. Some of them, via cmd.exe. Oh, you need to pipe variable stuff into another command while you’re doing it.

    (Oh, the joys of building a setup which partitions, formats and installs a system automatically – with different partitioning schemes for different arguments, and different applications installed depending on the machine’s DNS entry!)

  8. Anonymous says:

    I’m missing something..

    why is SETLOCAL ENABLEDELAYEDEXPANSION required ?

    I mean, cmd.exe can already tell between the twos because all old code is %WRITTENTHISWAY% while code using the new expansion style is !WRITTENTHISWAY! …

  9. Anonymous says:

    OrsoYoghi: I assume that’s for backward compatibility with old batch files that assumed that text between exclamation points was just text between exclamation points. You can’t just suddenly change the syntax of the file without requiring people to opt into the new functionality.

  10. Anonymous says:

    @OrsoYoghi: Yes, but what if old code depended on !STUFF_WRITTEN_LIKE_THIS! being treated as a literal string? The only backwords-compatible way is to use the old method with an option to enable the new one.

    (Then again, couldn’t you enable the new method by default in .cmd files, and leave .bat files alone?)

    [Making it on by default for .cmd files would break compatibility with NT4 .cmd files (which expected ! to mean !.) -Raymond]
  11. Anonymous says:

    I guess I was too slow. :) Sorry pcooper.

  12. Anonymous says:

    That’s interesting; the CMD team seem to think that it’s ok to implement features that break compatability, as long as you make them opt-in.
    The rest of Microsoft just uses “it will break back-compatability” as a reason not to improve things…

    [cmd.exe is a much more tightly-scoped problem. Command scripts communicate with other command scripts only through environment variables and the command line. Command scripts can’t inject themselves into other command scripts. There are no “cmd.exe plug-ins”. -Raymond]
  13. Anonymous says:

    I’d just like to say, thank God for Monad. :)

  14. Anonymous says:

    [Very true. I doubt anybody actually enjoys writing batch files. It’s not so much a programming language as a script. -Raymond]

    Oh, I dunno – I enjoy writing them.  CMD is like your weird Uncle Harold.  He’s fun to be around, despite his quirks.

  15. Anonymous says:

    Ah yes, yet another reason I use bash / rxvt / Cygwin.

  16. Anonymous says:

    Yes, thank god for bash. cmd.exe isn’t a script so much as a juiced up command.com. No wonder people hate the command line.

  17. MSDN Archive says:

    For extra pain – er, credit – try a .cmd

    > script which generates registry entries

    > (with variable substitution) which will then

    > be executed. Some of them, via cmd.exe. Oh,

    > you need to pipe variable stuff into another

    > command while you’re doing it.

    James, it sounds like you’re using a batch file to write .reg files and execute them. It’s much easier to use "%windir%system32reg.exe" from a batch file instead.

  18. Anonymous says:

    Geffner: I am using reg, rather than creating .reg files; the ‘will then be executed’ refers to the entries themselves being RunOnceEx entries which get run after the second reboot. The stuff I’m piping into another command is a diskpart script, not a set of registry entries.

    Nothing too painful, just a fiddly set of scripts to tie all the partitioning and installation together. Especially now we have two virtual machines and a dual-boot Linux setup to contend with, and could be running anywhere from the labs (a dozen new user accounts created/deleted each day, permanent Net connection, everything from AutoCAD to Visual Studio running) to staff laptops (one user, usually offline).

    A bit more work upfront, but since the hardware’s all heterogenous, a single "install" command with literally everything else automated saves enough staff time to justify it quite easily.

  19. Anonymous says:

    The real question is, of course, why sane people bother trying to program in a jumped-up MS-DOS ‘batch’ (which of course isn’t a batch system at all) language when they could just install Perl.

    Perl has the benefit of bizarre syntax, thus giving ex-batch-programmers a warm and fuzzy feeling that there’s something else for them to be puzzled over, but at the same time actually having the capabilities one needs.

    ;-)

  20. Anonymous says:

    Dave: Why don’t you consider ms-dos batch files as a batch system at all?

  21. Anonymous says:

    > The real question is, of course, why sane people bother trying to program in a jumped-up MS-DOS ‘batch’ (which of course isn’t a batch system at all) language when they could just install Perl. <<

    The reason is quite simple – because the cmd scripts will work without having to bother the user with installing Perl (or whatever other tool).

    The real problem with cmd scripting is that the syntax and semantics of the scripting language has been enhanced since the MS-DOS days in an ad-hoc manner rather than designing the language in a more formal manner.  At least that’s what it looks like.

    The other big problem (at least for me) is that applications are responsible for parsing command lines instead of having the shell parse the commad line, so it can be a nightmare to convince some tools to accept parameters that contain special characters which need escaping (particularly quotes) that is also acceptable to cmd.exe.

  22. Anonymous says:

    > The real question is, of course, why sane people bother trying to program in a jumped-up MS-DOS ‘batch’ (which of course isn’t a batch system at all) language when they could just install Perl. <<

    The reason is quite simple – because the cmd scripts will work without having to bother the user with installing Perl (or whatever other tool).

    The real problem with cmd scripting is that the syntax and semantics of the scripting language has been enhanced since the MS-DOS days in an ad-hoc manner rather than designing the language in a more formal manner.  At least that’s what it looks like.

    The other big problem (at least for me) is that applications are responsible for parsing command lines instead of having the shell parse the commad line, so it can be a nightmare to convince some tools to accept parameters that contain special characters which need escaping (particularly quotes) that is also acceptable to cmd.exe.

  23. Anonymous says:

    No offence to the cmd team, but MS could’ve done it "properly" years ago. The big mistake IMHO was NT (3.51) maintained DOS cmd compatibility — I’ll let the Win9x family off the hook. Although perhaps the ultimate blame lies with OS2.

    It would’ve been very very useful if shell-level scripting had been built into the Window Manager right from the start (imagine being able to test/manipulate your UI from a script supported by the OS? App.Window[1].Maximise (as an aside it would’ve been awesome if Noah Webster had had another hobby apart from butchering the english language)).

    As for the "just use [perl|cygwin|.*]" argument, it doesn’t always work in large organisations where introducing a new technology into the production environment is a long slow process requiring meetings of dozens, evaluation committees etc. .bat is very very good compared to facing that rubbish (maybe that’s only the Australian Govt for you).

  24. Anonymous says:

    Before anyone mentions it vba makes a good scripting language not.

  25. Anonymous says:

    Raymond, your explanation of why this behaviour is the default doesn’t make sense.  You say “prior to Windows NT, that was the only type of expansion supported by the command interpreter.”  But prior to NT, the command interpreter only supported executing a single command per line read, so the question of when variables were expanded never arose: it happened immediately prior to execution, which also happened to be immediately after reading them.

    When the NT interpreter was written, somebody decided that this way was the best way of doing it.  I’m sure that CMD.EXE has always been written in C or some other high-level language, so the complexity of doing it the other way can’t have been that bad.  So I still wonder why this decision went the way it did, when the other way is much more useful.

    [It also had to be compatible with OS/2 batch files, and OS/2’s command interpreter was written in assembly language as I vaguely recall. -Raymond]
  26. Anonymous says:

    OS/2 batch files = Rexx

    Quite superior to Batch.

Comments are closed.