Script recipe of the week: how to copy an opened file

I just read recently a post on the newsgroups - someone wanted to backup her PST file. The problem is that Outlook keeps this file open all the time, therefore we are unable to just copy this file once in a while. Argh...

 

A cute little script

Fortunately, we have Shadow copies in XP and Server 2003! So I just wrote a quick script that allows this PST to be quickly backed up. The script (called CopyWithVss.cmd) has a syntax which is very similar with the COPY command in shell.

CopyWithVss source_file destination file

Here is the script contents:

setlocal

@REM test if we are called by VSHADOW
if NOT "%CALLBACK_SCRIPT%"=="" goto :IS_CALLBACK

@REM
@REM Get the source and destination path
@REM

set SOURCE_DRIVE_LETTER=%~d1
set SOURCE_RELATIVE_PATH=%~pnx1
set DESTINATION_PATH=%2

@REM
@REM Create the shadow copy - and generate env variables into a temporary script.
@REM
@REM Then, while the shadow is still live
@REM recursively execute the same script.
@REM

@echo ...Determine the scripts to be executed/generated...

set CALLBACK_SCRIPT=%~dpnx0
set TEMP_GENERATED_SCRIPT=GeneratedVarsTempScript.cmd

@echo ...Creating the shadow copy...

%~dp0\vshadow.exe -script=%TEMP_GENERATED_SCRIPT% -exec=%CALLBACK_SCRIPT% %SOURCE_DRIVE_LETTER%

del /f %TEMP_GENERATED_SCRIPT%

@goto :EOF

:IS_CALLBACK
setlocal

@REM
@REM This generated script should set the SHADOW_DEVICE_1 env variable
@REM

@echo ...Obtaining the shadow copy device name...

call %TEMP_GENERATED_SCRIPT%

@REM
@REM This should copy the file to the right location
@REM

@echo ...Copying from the shadow copy to the destination path...

copy "%SHADOW_DEVICE_1%\%SOURCE_RELATIVE_PATH%" %DESTINATION_PATH%

 

How does it work?

Let's go through all the instructions, step by step.

1) The initialization phase:

setlocal

@REM test if we are called by VSHADOW
if NOT "%CALLBACK_SCRIPT%"=="" goto :IS_CALLBACK

@REM
@REM Get the source and destination path
@REM

set SOURCE_DRIVE_LETTER=%~d1
set SOURCE_RELATIVE_PATH=%~pnx1
set DESTINATION_PATH=%2

Here, "setlocal" is used to make sure that any variables that we set in this script will not be defined after its execution is done (we'll see later why). Let's skip the "if NOT" test for a second, and go to the next SET statements. Here, we are just interpreting the script arguments. So, if the script was invoked with the arguments CopyWithVss c:\path1\foo.txt d:\path2\bar.txt then, %1 will be c:\path1\foo.txt, and %2 will be d:\path2\bar.txt.

And the weird-looking expression "%~d1" is nothing more than the drive letter of the path contained in %1. ("d" stands for the "drive letter"). Therefore %~d1 will become c: . How about %~pnx1 ? Similarly, "p" stands for "path", "n" for "name", and "x" for extension. And more, %~pnx1 is just a shortcut for %~p1%~n1%~x1 which represents the concatenated path, name and extension of the file contained in %1. In other words, the string path1\foo.txt.

For more documentation on these expressions, you can do a "CALL /?" in your CMD shell.

2) Let's continue...

@echo ...Determine the scripts to be executed/generated...

set CALLBACK_SCRIPT=%~dpnx0
set TEMP_GENERATED_SCRIPT=GeneratedVarsTempScript.cmd

Now, similarly with the logic above, the variable CALLBACK_SCRIPT will be set to the full path (including drive letter, path, name and extension) of the actual script (since %0 is the name of the currently executing script). So, if you execute the script from z:\foo\CopyWithVss.cmd, then %~dpnx0 will represent this fully qualified path. Why we are doing all this manipulation will become apparent in a second.

3) The following line is the fun part:

%~dp0\vshadow.exe -script=%TEMP_GENERATED_SCRIPT% -exec=%CALLBACK_SCRIPT% %SOURCE_DRIVE_LETTER%

Now, what is this vshadow.exe tool? It is a simple command-line tool that allows anyone to create shadow copies. You can find the tool already compiled in the latest downloadable VSS SDK. Just copy VSHADOW.EXE in the same directory as the CMD above. What this tool does is that it creates a shadow copy of the volume specified as its last parameter. Aha! so this is why we had to isolate the drive letter of the source path. To go with the example above (%1 being c:\path1\foo.txt), then the last parameter is the C: drive letter, meaning that we will create a shadow copy of C:. Remember that shadow copies are per volume, not per file or directory! But don't worry, the shadow copy creation is a relatively painless operation in terms of performance. Most likely it takes less than the time needed to copy your PST file :-)

But what is a shadow copy, anyway? It is a regular volume that appears in the system for a brief time (at least in XP, because in Windows Server 2003 you have the ability to create a persistent shadow copy - just use the "-p" option in VSHADOW for example). This volume is identified by a temporary device name, looking like this: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy253. The fact that the volume is somewhat temporary makes our script a little bit harder. How do we retrieve the shadow copy device in our CMD script? VSHADOW solves this in a simple way, offering you the ability to generate a temporary script, containing a bunch of SET commands that will save the new devices as environment variables. Then, you can simply refer these devices in some shell commands, like COPY. So if you execute this script, it will set a variable called SHADOW_DEVICE_1 (and subsequently SHADOW_DEVICE_2,... N) for each volume to be shadowed. So this is the explanation for the "-script" option above: it will generate a script called GeneratedVarsTempScript.cmd that we have to execute to get the actual shadow device.

Now, here is another challenging problem: the shadow copy device dissapears immediately after the VSHADOW program exits! This is by design with all non-persistent shadow copies created by VSS. Internally, VSHADOW acts as a COM client, and the VSS service as a COM server. When the last referece to the COM interface referred by VSHADOW goes away, all the created shadow copies are going away too. In fact, this is a convenient feature for a backup application; for example if that application crashes, then you won't have leaking shadow copies: COM will automatically release the COM reference after a 6 minutes idle time, therefore automatically deleting the shadow copy.

But, this seems to be an impossible problem now - if the shadow copy is gone by the time VSHADOW exits, how we can still access its contents? Fortunately, VSHADOW has the ability to execute a "callback" program while the shadow copy is still alive. This callback script is passed using the second "-exec" option. Whew!

So what is being passed in the "-exec" option? Nothing more than the fully-qualified path of the original script! In other words, the script is recursively calling itself. We avoid the infinite cycle in a very easy way. Remember the "if NOT" instruction at the beginning of the script?

@REM test if we are called by VSHADOW
if NOT "%CALLBACK_SCRIPT%"=="" goto :IS_CALLBACK

That does the trick - since the env variable CALLBACK_SCRIPT is defined inside our script, it is (almost) guaranteed to not be set when the script is invoked for the first time and being set when invoked recursively for the second time. This is a neat trick that allowed us to avoid defining a separate CMD script. At the same time, the test allows us to detect what to do when we are inside VSHADOW... remember, all we wanted to do is to copy a file. This also explains why do we need a "setlocal" at the beginning of the script - otherwise, succesive executions in the same CMD instance will simply not work since the first execution will leave CALLBACK_SCRIPT defined, and the script will think that it is already in VSHADOW!

All this seems pretty complicated, but if you think a little bit about it, you can see that there is no other way around it. This is actually the simplest way to execute some custom commands while the shadow copy is still alive...

4) Let's jump now to the :IS_CALLBACK part - this is what is being executed when we are called by VSHADOW:

:IS_CALLBACK
setlocal

@REM
@REM This generated script should set the SHADOW_DEVICE_1 env variable
@REM

@echo ...Obtaining the shadow copy device name...

call %TEMP_GENERATED_SCRIPT%

@REM
@REM This should copy the file to the right location
@REM

@echo ...Copying from the shadow copy to the destination path...

copy "%SHADOW_DEVICE_1%\%SOURCE_RELATIVE_PATH%" %DESTINATION_PATH%

We can see above the call to the generated script (that should define the SHADOW_DEVICE_1 environment variable). Then, we just invoke the COPY command, passing the source and the destination path. Now it is clear why we needed the SOURCE_RELATIVE_PATH variable in the first place - since the file path on the shadow copy is actually relative to the root of the shadow copy volume.

5) Now, let's jump back to the original code branch:

%~dp0\vshadow.exe -script=%TEMP_GENERATED_SCRIPT% -exec=%CALLBACK_SCRIPT% %SOURCE_DRIVE_LETTER%

del /f %TEMP_GENERATED_SCRIPT%

@goto :EOF

Obviously, this deletes the temporarily-generated CMD script and exits the main script. Done!

 

Let's give it a try

Nothing easier... here is the generated output.

   Z:\outlook_backup>CopyWithVss.cmd x:\Store\PST\2004-01-27.pst x:\Store\PST\backup.pst Z:\outlook_backup>setlocal Z:\outlook_backup>if NOT "" == "" goto :IS_CALLBACK Z:\outlook_backup>set SOURCE_DRIVE_LETTER=x: Z:\outlook_backup>set SOURCE_RELATIVE_PATH=\Store\PST\2004-01-27.pst Z:\outlook_backup>set DESTINATION_PATH=x:\Store\PST\backup.pst
...Determine the scripts to be executed/generated... Z:\outlook_backup>set CALLBACK_SCRIPT=Z:\outlook_backup\CopyWithVss.cmd Z:\outlook_backup>set TEMP_GENERATED_SCRIPT=GeneratedVarsTempScript.cmd
...Creating the shadow copy... Z:\outlook_backup>Z:\outlook_backup\\vshadow.exe -script=GeneratedVarsTempScript.cmd -exec=Z:\outlook_backup\CopyWithVss.cmd x: VSHADOW.EXE 2.0 - Volume Shadow Copy sample client
Copyright (C) 2004 Microsoft Corporation. All rights reserved. 
(Option: Generate SETVAR script 'GeneratedVarsTempScript.cmd')
(Option: Execute binary/script after shadow creation 'Z:\outlook_backup\CopyWithVss.cmd')
(Option: Create shadow copy set)
(Gathering writer metadata...)
(Waiting for the asynchronous operation to finish...)
Initialize writer metadata ...
Discover directly excluded components ...
- Excluding writer 'MSDEWriter' since it has no selected components for restore.
Discover components that reside outside the shadow set ...
- Component '\System Files' from writer 'System Writer' is excluded from backup (it requires C:\ in the shadow set)
- Component '\WMI' from writer 'WMI Writer' is excluded from backup (it requires C:\ in the shadow set)
- Component '\Event Logs' from writer 'Event Log Writer' is excluded from backup (it requires C:\ in the shadow set)
- Component '\Registry' from writer 'Registry Writer' is excluded from backup (it requires C:\ in the shadow set)
- Component '\COM+ REGDB' from writer 'COM+ REGDB Writer' is excluded from backup (it requires C:\ in the shadow set)
- Component '\IISMETABASE' from writer 'IIS Metabase Writer' is excluded from backup (it requires C:\ in the shadow set)
Discover all excluded components ...
Discover excluded writers ...
- The writer 'System Writer' is now entirely excluded from the backup:
  (it does not contain any components that can be potentially included in the backup)
- The writer 'WMI Writer' is now entirely excluded from the backup:
  (it does not contain any components that can be potentially included in the backup)
- The writer 'Event Log Writer' is now entirely excluded from the backup:
  (it does not contain any components that can be potentially included in the backup)
- The writer 'Registry Writer' is now entirely excluded from the backup:
  (it does not contain any components that can be potentially included in the backup)
- The writer 'COM+ REGDB Writer' is now entirely excluded from the backup:
  (it does not contain any components that can be potentially included in the backup)
- The writer 'IIS Metabase Writer' is now entirely excluded from the backup:
  (it does not contain any components that can be potentially included in the backup)
Discover explicitly included components ...
Verifying explicitly specified writers/components ...
Select explicitly included components ...
Creating shadow set {e8d4d81c-337a-4df1-bda0-b7be96c5e0da} ...
- Adding volume \\?\Volume{785cc4a6-3d68-11d7-9cc5-505054503030}\ [X:\] to the shadow set...
Preparing for backup ...
(Waiting for the asynchronous operation to finish...)
(Waiting for the asynchronous operation to finish...)
Creating the shadow (DoSnapshotSet) ...
(Waiting for the asynchronous operation to finish...)
(Waiting for the asynchronous operation to finish...)
Shadow copy set succesfully created. List of created shadow copies: 
Querying all shadow copies with the SnapshotSetID {e8d4d81c-337a-4df1-bda0-b7be96c5e0da} ... * SNAPSHOT ID = {0b2fc776-3ebc-459c-9437-af02b4fbd46f} ...
   - Shadow copy Set: {e8d4d81c-337a-4df1-bda0-b7be96c5e0da}
   - Original count of shadow copies = 1
   - Original Volume name: \\?\Volume{785cc4a6-3d68-11d7-9cc5-505054503030}\ [X:\]
   - Shadow copy device name: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy253
   - Originating machine: aoltean-dev.ntdev.corp.microsoft.com
   - Service machine: aoltean-dev.ntdev.corp.microsoft.com
   - Not Exposed
   - Provider id: {b5946137-7b9f-4925-af80-51abd60b20d5}
   - Attributes:  Auto_Release Differential Generating the SETVAR script (GeneratedVarsTempScript.cmd) ...
Executing command 'Z:\outlook_backup\CopyWithVss.cmd' ...
----------------------------------------------------- Z:\outlook_backup>setlocal Z:\outlook_backup>if NOT "Z:\outlook_backup\CopyWithVss.cmd" == "" goto :IS_CALLBACK Z:\outlook_backup>setlocal
...Obtaining the shadow copy device name... Z:\outlook_backup>call GeneratedVarsTempScript.cmd [This script is generated by VSHADOW.EXE for the shadow set {e8d4d81c-337a-4df1-bda0-b7be96c5e0da}] 
Z:\outlook_backup>SET SHADOW_SET_ID={e8d4d81c-337a-4df1-bda0-b7be96c5e0da} Z:\outlook_backup>SET SHADOW_ID_1={0b2fc776-3ebc-459c-9437-af02b4fbd46f} Z:\outlook_backup>SET SHADOW_DEVICE_1=\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy253
...Copying from the shadow copy to the destination path... Z:\outlook_backup>copy "\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy253\\Store\PST\2004-01-27.pst"  x:\Store\PST\backup.pst<br>        1 file(s) copied. 
-----------------------------------------------------
- Mark all writers as succesfully backed up...
Completing the backup (BackupComplete) ...
(Waiting for the asynchronous operation to finish...)
(Waiting for the asynchronous operation to finish...) Snapshot creation done. Z:\outlook_backup>del /f GeneratedVarsTempScript.cmd 

In the end, we copied the original PST file from x:\Store\PST\2004-01-27.pst into x:\Store\PST\backup.pst (even the original PST was opened by Outlook).

That's it!

 

WARNING: Note that there are almost identical two versions of this tool: one for XP, and another one for Server 2003. Do not attempt to run the XP version on server or viceversa - it won't work!