NTFS Sparse Files with C#

Sparse files in NTFS has been a pet interest for a while now, but I just never had the combination of time and patience to deal with DeviceIoControl and get a C# wrapper in place. Well, with our offices closed over Eid here in Dubai I had the time and, as a side effect to procrastination, the patience. Plus the xbox I won is delayed until after the weekend, so...

   36 using (FileStream fs = NTFS.Sparse.SparseFile.Create(filename))

   37 {

   38     // Create a 640kB file containing 128k blocks of ones and zeros

   39     WriteBlocks(fs);

   40 

   41     // TODO: Set sparse range inside the file

   42     uint startIx = 128 * 1024;

   43     NTFS.Sparse.SparseFile.SetSparseBlock(fs.SafeFileHandle, startIx, 128 * 1024);

   44     startIx = 128 * 1024 * 3;

   45     NTFS.Sparse.SparseFile.SetSparseBlock(fs.SafeFileHandle, startIx, 128 * 1024);

This effectively creates a file with content and then after the fact punches two holes in the file by telling NTFS that those two sections are all-zero and do not need to have disk space allocated. When you test, bear in mind you won't get exactly half the file size back, there is still overhead for managing the empty sections. For a good description of sparse files and how to use them (in C++), take a look at this article on FlexHex.com.

The magic happens when you create the file:

   13 public static FileStream Create(string fileName)

   14 {

 

   27 

   28     uint dwShareMode = (uint)Win32.EFileShare.None;

   29     uint dwDesiredAccess = (uint)Win32.EFileAccess.GenericWrite;

   30     uint dwFlagsAndAttributes = (uint)Win32.EFileAttributes.Normal;

   31     uint dwCreationDisposition = (uint)Win32.ECreationDisposition.CreateAlways;

   32 

   33     SafeFileHandle fileHandle =

   34         Win32.Methods.CreateFileW(

   35             fileName,

   36             dwDesiredAccess,

   37             dwShareMode,

   38             IntPtr.Zero,

   39             dwCreationDisposition,

   40             dwFlagsAndAttributes,

   41             IntPtr.Zero);

   42 

   43     int bytesReturned = 0;

   44     NativeOverlapped lpOverlapped = new NativeOverlapped();

   45     bool result =

   46         Win32.Methods.DeviceIoControl(

   47             fileHandle,

   48             Win32.EIoControlCode.FsctlSetSparse,

   49             IntPtr.Zero,

   50             0,

   51             IntPtr.Zero,

   52             0,

   53             ref bytesReturned,

   54             ref lpOverlapped);

   55 

   56     return new System.IO.FileStream(fileHandle, System.IO.FileAccess.Write);

 And more magic behind the .SetSparseBlock method which you can call after you determined what parts is / needs to be zero and just directly set it as sparse instead of writing zeros.

   59 public static void SetSparseBlock(SafeFileHandle fileHandle, uint startIx, uint zeroBlockLength)

   60 {

 

   71     int dwTemp = 0;

   72     NativeOverlapped lpOverlapped = new NativeOverlapped();

   73 

   74     Win32.FILE_ZERO_DATA_INFORMATION fzd;

   75     fzd.FileOffset = startIx;

   76     fzd.BeyondFinalZero = startIx + zeroBlockLength;

   77 

   78     IntPtr ptrFZD = IntPtr.Zero;

   79     try

   80     {

   81         ptrFZD = Marshal.AllocHGlobal(Marshal.SizeOf(fzd));

   82         Marshal.StructureToPtr(fzd, ptrFZD, true);

   83 

   84         bool result =

   85             Win32.Methods.DeviceIoControl(

   86                 fileHandle,

   87                 Win32.EIoControlCode.FsctlSetZeroData,

   88                 ptrFZD,

   89                 Marshal.SizeOf(fzd),

   90                 IntPtr.Zero,

   91                 0,

   92                 ref dwTemp,

   93                 ref lpOverlapped);

   94     }

   95     finally

   96     {

   97         Marshal.DestroyStructure(ptrFZD, typeof(Win32.FILE_ZERO_DATA_INFORMATION));

   98     }

Lines 78 to 82 with 97 make me really nervous. This is my first attempt at interop, so I can do with some code review here! In fact, the attached project uses "unsafe" code to get the pointer of the struct directly instead of via the Marshal class.

The coolest part about sparse files is that if an application unaware of the sparse nature of the file accesses it, NTFS does the right thing anyway, so you can happily let unaware applications edit the files you create with your sparse aware application.

The last feature worth mentioning is the allocation of sparse space at the end of a file. I hear this is a typical scenario: writing one byte or more far past the end of the current file size. You can accomplish this easily since the file is already marked as sparse and NTFS understands you skipped a few bytes like so:

   47     // TODO: Set sparse block at the end of the file / extend existing file with zero'd content for length N

   48     fs.Seek(8192 * 1024, SeekOrigin.End);

   49     fs.WriteByte(1);

   50 }

Line 50 is the closing of an earlier using (FileStream ...

So there you have it: sparse files in .NET with just a little interop gymnastics. But I'm no interop or Win32 guru, so use at your own risk!

FYI: You can use "fsutil" from the commandline to verify this works. Only, one minor thing, when you "queryrange", it actually tells you the ranges of NONzero data. Thought you should know. That only took me an HOUR to figure out!!!

SparseFileTest.zip