Script recipe of the week: how to copy an opened file


I just read recently a post on the newsgroups – someone wanted to backup her PST file. The problem is that Outlook keeps this file open all the time, therefore we are unable to just copy this file once in a while. Argh…


 


A cute little script


Fortunately, we have Shadow copies in XP and Server 2003! So I just wrote a quick script that allows this PST to be quickly backed up. The script (called CopyWithVss.cmd) has a syntax which is very similar with the COPY command in shell.



CopyWithVss source_file destination file


Here is the script contents:



setlocal


@REM test if we are called by VSHADOW
if NOT “%CALLBACK_SCRIPT%”==”” goto :IS_CALLBACK


@REM
@REM Get the source and destination path
@REM


set SOURCE_DRIVE_LETTER=%~d1
set SOURCE_RELATIVE_PATH=%~pnx1
set DESTINATION_PATH=%2


@REM
@REM Create the shadow copy – and generate env variables into a temporary script.
@REM
@REM Then, while the shadow is still live
@REM recursively execute the same script.
@REM


@echo …Determine the scripts to be executed/generated…


set CALLBACK_SCRIPT=%~dpnx0
set TEMP_GENERATED_SCRIPT=GeneratedVarsTempScript.cmd


@echo …Creating the shadow copy…


%~dp0\vshadow.exe -script=%TEMP_GENERATED_SCRIPT% -exec=%CALLBACK_SCRIPT% %SOURCE_DRIVE_LETTER%


del /f %TEMP_GENERATED_SCRIPT%


@goto :EOF


:IS_CALLBACK
setlocal


@REM
@REM This generated script should set the SHADOW_DEVICE_1 env variable
@REM


@echo …Obtaining the shadow copy device name…


call %TEMP_GENERATED_SCRIPT%


@REM
@REM This should copy the file to the right location
@REM


@echo …Copying from the shadow copy to the destination path…


copy “%SHADOW_DEVICE_1%\%SOURCE_RELATIVE_PATH%” %DESTINATION_PATH%


 


How does it work?


Let’s go through all the instructions, step by step.


1) The initialization phase:



setlocal


@REM test if we are called by VSHADOW
if NOT “%CALLBACK_SCRIPT%”==”” goto :IS_CALLBACK


@REM
@REM Get the source and destination path
@REM


set SOURCE_DRIVE_LETTER=%~d1
set SOURCE_RELATIVE_PATH=%~pnx1
set DESTINATION_PATH=%2


Here, “setlocal” is used to make sure that any variables that we set in this script will not be defined after its execution is done (we’ll see later why). Let’s skip the “if NOT” test for a second, and go to the next SET statements. Here, we are just interpreting the script arguments. So, if the script was invoked with the arguments CopyWithVss c:\path1\foo.txt d:\path2\bar.txt then, %1 will be c:\path1\foo.txt, and %2 will be d:\path2\bar.txt.


And the weird-looking expression “%~d1” is nothing more than the drive letter of the path contained in %1. (“d” stands for the “drive letter”). Therefore %~d1 will become c: . How about %~pnx1 ? Similarly, “p” stands for “path”, “n” for “name”, and “x” for extension. And more, %~pnx1 is just a shortcut for %~p1%~n1%~x1 which represents the concatenated path, name and extension of the file contained in %1. In other words, the string path1\foo.txt.


For more documentation on these expressions, you can do a “CALL /?” in your CMD shell.


2) Let’s continue…



@echo …Determine the scripts to be executed/generated…


set CALLBACK_SCRIPT=%~dpnx0
set TEMP_GENERATED_SCRIPT=GeneratedVarsTempScript.cmd


Now, similarly with the logic above, the variable CALLBACK_SCRIPT will be set to the full path (including drive letter, path, name and extension) of the actual script (since %0 is the name of the currently executing script). So, if you execute the script from z:\foo\CopyWithVss.cmd, then %~dpnx0 will represent this fully qualified path. Why we are doing all this manipulation will become apparent in a second.


3) The following line is the fun part:



%~dp0\vshadow.exe -script=%TEMP_GENERATED_SCRIPT% -exec=%CALLBACK_SCRIPT% %SOURCE_DRIVE_LETTER%


Now, what is this vshadow.exe tool? It is a simple command-line tool that allows anyone to create shadow copies. You can find the tool already compiled in the latest downloadable VSS SDK. Just copy VSHADOW.EXE in the same directory as the CMD above. What this tool does is that it creates a shadow copy of the volume specified as its last parameter. Aha! so this is why we had to isolate the drive letter of the source path. To go with the example above (%1 being c:\path1\foo.txt), then the last parameter is the C: drive letter, meaning that we will create a shadow copy of C:. Remember that shadow copies are per volume, not per file or directory! But don’t worry, the shadow copy creation is a relatively painless operation in terms of performance. Most likely it takes less than the time needed to copy your PST file 🙂


But what is a shadow copy, anyway? It is a regular volume that appears in the system for a brief time (at least in XP, because in Windows Server 2003 you have the ability to create a persistent shadow copy – just use the “-p” option in VSHADOW for example). This volume is identified by a temporary device name, looking like this: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy253. The fact that the volume is somewhat temporary makes our script a little bit harder. How do we retrieve the shadow copy device in our CMD script? VSHADOW solves this in a simple way, offering you the ability to generate a temporary script, containing a bunch of SET commands that will save the new devices as environment variables. Then, you can simply refer these devices in some shell commands, like COPY. So if you execute this script, it will set a variable called SHADOW_DEVICE_1 (and subsequently SHADOW_DEVICE_2,… N) for each volume to be shadowed. So this is the explanation for the “-script” option above: it will generate a script called GeneratedVarsTempScript.cmd that we have to execute to get the actual shadow device.


Now, here is another challenging problem: the shadow copy device dissapears immediately after the VSHADOW program exits! This is by design with all non-persistent shadow copies created by VSS. Internally, VSHADOW acts as a COM client, and the VSS service as a COM server. When the last referece to the COM interface referred by VSHADOW goes away, all the created shadow copies are going away too. In fact, this is a convenient feature for a backup application; for example if that application crashes, then you won’t have leaking shadow copies: COM will automatically release the COM reference after a 6 minutes idle time, therefore automatically deleting the shadow copy.


But, this seems to be an impossible problem now – if the shadow copy is gone by the time VSHADOW exits, how we can still access its contents? Fortunately, VSHADOW has the ability to execute a “callback” program while the shadow copy is still alive. This callback script is passed using the second “-exec” option. Whew!


So what is being passed in the “-exec” option? Nothing more than the fully-qualified path of the original script! In other words, the script is recursively calling itself. We avoid the infinite cycle in a very easy way. Remember the “if NOT” instruction at the beginning of the script?



@REM test if we are called by VSHADOW
if NOT “%CALLBACK_SCRIPT%”==”” goto :IS_CALLBACK


That does the trick – since the env variable CALLBACK_SCRIPT is defined inside our script, it is (almost) guaranteed to not be set when the script is invoked for the first time and being set when invoked recursively for the second time. This is a neat trick that allowed us to avoid defining a separate CMD script. At the same time, the test allows us to detect what to do when we are inside VSHADOW… remember, all we wanted to do is to copy a file. This also explains why do we need a “setlocal” at the beginning of the script – otherwise, succesive executions in the same CMD instance will simply not work since the first execution will leave CALLBACK_SCRIPT defined, and the script will think that it is already in VSHADOW!


All this seems pretty complicated, but if you think a little bit about it, you can see that there is no other way around it. This is actually the simplest way to execute some custom commands while the shadow copy is still alive…


4) Let’s jump now to the :IS_CALLBACK part – this is what is being executed when we are called by VSHADOW:



:IS_CALLBACK
setlocal


@REM
@REM This generated script should set the SHADOW_DEVICE_1 env variable
@REM


@echo …Obtaining the shadow copy device name…


call %TEMP_GENERATED_SCRIPT%


@REM
@REM This should copy the file to the right location
@REM


@echo …Copying from the shadow copy to the destination path…


copy “%SHADOW_DEVICE_1%\%SOURCE_RELATIVE_PATH%” %DESTINATION_PATH%


We can see above the call to the generated script (that should define the SHADOW_DEVICE_1 environment variable). Then, we just invoke the COPY command, passing the source and the destination path. Now it is clear why we needed the SOURCE_RELATIVE_PATH variable in the first place – since the file path on the shadow copy is actually relative to the root of the shadow copy volume.


5) Now, let’s jump back to the original code branch:



%~dp0\vshadow.exe -script=%TEMP_GENERATED_SCRIPT% -exec=%CALLBACK_SCRIPT% %SOURCE_DRIVE_LETTER%


del /f %TEMP_GENERATED_SCRIPT%


@goto :EOF


Obviously, this deletes the temporarily-generated CMD script and exits the main script. Done!


 


Let’s give it a try


Nothing easier… here is the generated output.


 

Z:\outlook_backup>CopyWithVss.cmd x:\Store\PST\2004-01-27.pst x:\Store\PST\backup.pst

Z:\outlook_backup>setlocal

Z:\outlook_backup>if NOT “” == “” goto :IS_CALLBACK

Z:\outlook_backup>set SOURCE_DRIVE_LETTER=x:

Z:\outlook_backup>set SOURCE_RELATIVE_PATH=\Store\PST\2004-01-27.pst

Z:\outlook_backup>set DESTINATION_PATH=x:\Store\PST\backup.pst
…Determine the scripts to be executed/generated…

Z:\outlook_backup>set CALLBACK_SCRIPT=Z:\outlook_backup\CopyWithVss.cmd

Z:\outlook_backup>set TEMP_GENERATED_SCRIPT=GeneratedVarsTempScript.cmd
…Creating the shadow copy…

Z:\outlook_backup>Z:\outlook_backup\\vshadow.exe -script=GeneratedVarsTempScript.cmd -exec=Z:\outlook_backup\CopyWithVss.cmd x:

VSHADOW.EXE 2.0 – Volume Shadow Copy sample client
Copyright (C) 2004 Microsoft Corporation. All rights reserved.


(Option: Generate SETVAR script ‘GeneratedVarsTempScript.cmd’)
(Option: Execute binary/script after shadow creation ‘Z:\outlook_backup\CopyWithVss.cmd’)
(Option: Create shadow copy set)
(Gathering writer metadata…)
(Waiting for the asynchronous operation to finish…)
Initialize writer metadata …
Discover directly excluded components …
– Excluding writer ‘MSDEWriter’ since it has no selected components for restore.
Discover components that reside outside the shadow set …
– Component ‘\System Files’ from writer ‘System Writer’ is excluded from backup (it requires C:\ in the shadow set)
– Component ‘\WMI’ from writer ‘WMI Writer’ is excluded from backup (it requires C:\ in the shadow set)
– Component ‘\Event Logs’ from writer ‘Event Log Writer’ is excluded from backup (it requires C:\ in the shadow set)
– Component ‘\Registry’ from writer ‘Registry Writer’ is excluded from backup (it requires C:\ in the shadow set)
– Component ‘\COM+ REGDB’ from writer ‘COM+ REGDB Writer’ is excluded from backup (it requires C:\ in the shadow set)
– Component ‘\IISMETABASE’ from writer ‘IIS Metabase Writer’ is excluded from backup (it requires C:\ in the shadow set)
Discover all excluded components …
Discover excluded writers …
– The writer ‘System Writer’ is now entirely excluded from the backup:
  (it does not contain any components that can be potentially included in the backup)
– The writer ‘WMI Writer’ is now entirely excluded from the backup:
  (it does not contain any components that can be potentially included in the backup)
– The writer ‘Event Log Writer’ is now entirely excluded from the backup:
  (it does not contain any components that can be potentially included in the backup)
– The writer ‘Registry Writer’ is now entirely excluded from the backup:
  (it does not contain any components that can be potentially included in the backup)
– The writer ‘COM+ REGDB Writer’ is now entirely excluded from the backup:
  (it does not contain any components that can be potentially included in the backup)
– The writer ‘IIS Metabase Writer’ is now entirely excluded from the backup:
  (it does not contain any components that can be potentially included in the backup)
Discover explicitly included components …
Verifying explicitly specified writers/components …
Select explicitly included components …
Creating shadow set {e8d4d81c-337a-4df1-bda0-b7be96c5e0da} …
– Adding volume \\?\Volume{785cc4a6-3d68-11d7-9cc5-505054503030}\ [X:\] to the shadow set…
Preparing for backup …
(Waiting for the asynchronous operation to finish…)
(Waiting for the asynchronous operation to finish…)
Creating the shadow (DoSnapshotSet) …
(Waiting for the asynchronous operation to finish…)
(Waiting for the asynchronous operation to finish…)
Shadow copy set succesfully created.

List of created shadow copies:


Querying all shadow copies with the SnapshotSetID {e8d4d81c-337a-4df1-bda0-b7be96c5e0da} …

* SNAPSHOT ID = {0b2fc776-3ebc-459c-9437-af02b4fbd46f} …
   – Shadow copy Set: {e8d4d81c-337a-4df1-bda0-b7be96c5e0da}
   – Original count of shadow copies = 1
   – Original Volume name: \\?\Volume{785cc4a6-3d68-11d7-9cc5-505054503030}\ [X:\]
   – Shadow copy device name: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy253
   – Originating machine: aoltean-dev.ntdev.corp.microsoft.com
   – Service machine: aoltean-dev.ntdev.corp.microsoft.com
   – Not Exposed
   – Provider id: {b5946137-7b9f-4925-af80-51abd60b20d5}
   – Attributes:  Auto_Release Differential

Generating the SETVAR script (GeneratedVarsTempScript.cmd) …
Executing command ‘Z:\outlook_backup\CopyWithVss.cmd’ …
—————————————————–

Z:\outlook_backup>setlocal

Z:\outlook_backup>if NOT “Z:\outlook_backup\CopyWithVss.cmd” == “” goto :IS_CALLBACK

Z:\outlook_backup>setlocal
…Obtaining the shadow copy device name…

Z:\outlook_backup>call GeneratedVarsTempScript.cmd

[This script is generated by VSHADOW.EXE for the shadow set {e8d4d81c-337a-4df1-bda0-b7be96c5e0da}]


Z:\outlook_backup>SET SHADOW_SET_ID={e8d4d81c-337a-4df1-bda0-b7be96c5e0da}

Z:\outlook_backup>SET SHADOW_ID_1={0b2fc776-3ebc-459c-9437-af02b4fbd46f}

Z:\outlook_backup>SET SHADOW_DEVICE_1=\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy253
…Copying from the shadow copy to the destination path…

Z:\outlook_backup>copy “\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy253\\Store\PST\2004-01-27.pst” x:\Store\PST\backup.pst
        1 file(s) copied.

—————————————————–
– Mark all writers as succesfully backed up…
Completing the backup (BackupComplete) …
(Waiting for the asynchronous operation to finish…)
(Waiting for the asynchronous operation to finish…)

Snapshot creation done.

Z:\outlook_backup>del /f GeneratedVarsTempScript.cmd


In the end, we copied the original PST file from x:\Store\PST\2004-01-27.pst into x:\Store\PST\backup.pst (even the original PST was opened by Outlook).


That’s it!


 


WARNING: Note that there are almost identical two versions of this tool: one for XP, and another one for Server 2003. Do not attempt to run the XP version on server or viceversa – it won’t work!


 

Comments (27)

  1. Very useful script !.

    I sucess to do it.

    but, If SQL Server service are running,

    CopyWithVss is failed.

    ERROR Messages is:

    —-

    ERROR: Selected writer ‘MSDEWriter’ is in failed state!

    – Status: 8 (VSS_WS_FAILED_AT_PREPARE_SNAPSHOT)

    – Writer Failure code: 0x800423f4 (<Unknown error code>)

    – Writer ID: {f8544ac1-0611-4fa5-b04b-f7ee00b03277}

    – Instance ID: {ef567f78-4f42-49be-bb1c-2386a7f489cf}

    —-

    I want to copy Database/Log Files by this script, because I’m MVP for SQL Server.

    Regards,

  2. Adi Oltean says:

    >> ERROR Messages is:

    —-

    ERROR: Selected writer ‘MSDEWriter’ is in failed state!

    – Status: 8 (VSS_WS_FAILED_AT_PREPARE_SNAPSHOT)

    – Writer Failure code: 0x800423f4 (<Unknown error code>)

    – Writer ID: {f8544ac1-0611-4fa5-b04b-f7ee00b03277}

    – Instance ID: {ef567f78-4f42-49be-bb1c-2386a7f489cf}

    Intersting – this should not happen. Are there any VSS errors in the Application event logs? If yes, please cut & paste them in your reply.

    Note that with SQL, if the database and the logs are on different volumes, then you might want to change the script to snapshot simultaneously both the database and the log volumes. If everything is on the same volume, it should still work.

    By the way, the VSS SDK contains a set of test VSS scripts that can be used to backup and restore a SQL/MSDE database, and then verify that the restore was succesful.

    Thanks, Adi

  3. Hello, Adi

    Application Event log:

    Type: ERROR

    SOURCE: VSS

    Category: non

    Event: : 6004

    User: N/A

    Description:

    "Sqllib ERROR: Database testdb is not simple".

    ** I use Japanese Environment (OS, SQL Server).

    Therefore, real errormessage is in japanese,

    If i detach database ‘testdb’ by SQL Server Enterprise Manger, Or If I change ‘testdb’ recovery mode from ‘full’ to ‘simple’,

    CopyWithVSS command do complete !.

    Therefore,

    if any database’s recovery mode is not simple, CopyWithVSS fail.

    Regards,

  4. Adi Oltean says:

    I have two more questions: Do you have XP or Windows Server on your system? Also, is this a SQL Server 2000 database?

    Thanks, Adi

  5. Hello, Adi

    Yes, I have

    Windows XP Pro + SP2 (Japanese Edition)

    Windows Server 2003 Std (Japanese Edition)

    SQL Server 2000 Std SP3a (Japanese Edition)

    and, this problem seems to relate below kb.

    *Error 800423f4 appears in the backup log file when you back up a volume by using the Volume Shadow Copy service in Windows Server 2003

    http://support.microsoft.com/default.aspx?scid=kb;en-us;828481

    and Windows Server 2003 SP1 will fix this problem.

    — how about XP ?

    Regards,

  6. Adi Oltean says:

    I see… this is a known problem with VSS & SQL on Windows XP.

    On Windows XP, the MSDE writer for VSS works only with SQL/MSDE databases with the Simple recovery model.

    But on Windows Server 2003, the writer should work with databases in full recovery model too, as long as the VSS backup is in component mode (from VSHADOW.EXE), not in volume mode (like from NTBackup.exe).

  7. Diego says:

    Will work for lotus(domino server) databases?

  8. Adi Oltean says:

    It depends. The recommended procedure is to have a VSS writer that listens for backup events (shadow copy creation events) and flushes the database in order to minimize the recovery time.

    If there is no writer involved, then the contents of the shadow copy will look as if the machine was suddenly powered off. If Lotus can recover after a sudden reboot (as anyone would expect) then, yes, you can use this technique for Lotus databases.

  9. As I was enjoying a fine dinner of Lipton Italian Sides one night, I came across Adi’s post on how to copy an open file with shadow copies. Not bad, I thought, and stored that little nugget of knowledge away for when I might need to do just that.

    A…

  10. And you thought it couldn’t be done: copying locked files, such as your .pst file from Outlook, while

  11. And you thought it couldn’t be done: copying locked files, such as your .pst file from Outlook, while

  12. Last week, I completely virtualized my life. Well, at least my server life is all virtualized. I’d been

Skip to main content