Why can’t I create a file equal to the available disk space?


A customer was trying to exercise some boundary cases in their application, one of which entailed "out of disk space" scenarios. The way they did this was very simple: Call Get­Disk­Free­Space­Ex to see how much disk space is available, and then create a file of that size. This worked on some volumes, but on other volumes, it failed with "Not enough disk space." Even after making sure no other programs were accessing the drive in question, the error persisted. Why?

The amount of disk space required to create a file of a particular size is not exactly equal to the number of bytes in the file. After all, the file system also needs to remember the file name, the security attributes, various timestamps, and information to keep track of where on the disk all the bytes of the file are stored. And then there are the less obvious things like the change journal records to record that the file was created and object tracking records for link tracking.

Just because your volume has exactly X bytes free doesn't mean that you will be able to create a file whose size is exactly X, because additional disk space will be needed to keep track of the metadata. If you want to fill a disk to the brim, you could start by filling it nearly to the brim, then growing the file until you get "Not enough disk space." If you want to get fancy, you could grow the file by large chunks until you can't add large chunks any more, and then switch to smaller and smaller chunks. You may also want to use the Set­File­Valid­Data function to avoid wasting time writing zeroes to all the sectors.

Comments (35)
  1. Ronald LP says:

    Also a file has not a linear layout on disk. Here is an interesting article about NTFS file growth : blogs.technet.com/…/the-four-stages-of-ntfs-file-growth.aspx

    Ronald

  2. Medinoc says:

    On that same subject, it seems Windows Explorer is actually guesstimating the "size on disk" on its properties window, instead of actually poking at the level of Ronald LP's article: In Windows 7, the "size on disk" still jumps from "0 bytes" to "4kb" when you add a single byte to a 0-byte file.

    I haven't tested on Windows 10 yet.

    [Would you prefer that Explorer show 256 bytes for the size of a 0-byte file? (To account for the metadata. And how do you charge for single-instance-stored ACLs?) Anyway, it doesn't matter; Explorer doesn't have insight into NTFS's internal data organization. It just calls GetCompressedFileSize(). -Raymond]
  3. Brian says:

    @Medinoc, Explorer is not guessing, on the system you were looking at, the file system of the drive had 4KB clusters. So that one byte file is using up 4KB on disk. When you format a drive using Explorer, one of the options is the Allocation Unit Size. This represents the least amount of space a 1 byte file will use on the drive.

  4. Medinoc says:

    @Brian see Ronald's post. I'm on an NTFS file system.

  5. Eric TF Bat says:

    Is there a file system where the following trick wouldn't work?

    freesize = FreeSpaceInBytes(mydisk);

    filename = TempFileNameOnDisk(mydisk);

    tempsize = 10;

    CreateDummyFile(filename, tempsize);

    overhead = freesize – FreeSpaceInBytes(mydisk) – tempsize;

    DeleteFile(filename);

    // free space should now be equal to freesize again

    CreateDummyFile(filename, freesize – overhead);

    // free space should now be zero

  6. DWalker says:

    I would NOT have assumed that I could create a file whose size is exactly the number of bytes left over on the disk.  Perhaps that's because I know a little bit about the on-disk structures, attributes, and the journal.  Also, because I'm not sure how the usual room for "previous versions of files" and restore points is handled, when the disk gets nearly full.

  7. ChrisR says:

    @Eric TF Bat:

    Take a look at the article linked by Ronald LP.  The file system overhead is not necessarily not a constant size.  In the case of NTFS, it grows as the file size grows.  I would expect this to be the case with most file systems.

  8. Joshua says:

    It's not true of UNIX systems either. Somebody's been dealing with FAT too long.

  9. Dave says:

    You're not just testing your own reaction to "out of disk space", you're testing the reaction of all the other things currently running on the system. Expecting all of them to behave nicely when you take away all their disk space just scares the crap out of me. It's like a test that updates the system clock.

  10. Yukkuri says:

    Why can't I grow to fill all the empty space in the universe? No one else is using it!

  11. Yuri Khan says:

    Even if you could ask the system for the exact maximum size of a file you could create right now, in the next 20 milliseconds some other process might have consumed some disk space so your attempt to create the file would still fail.

  12. Chris I. says:

    @Dave in that case, how do you propose testing that *your own* application handles the out-of-disk-space scenario well?

  13. Kevin says:

    @ChrisI: No idea how to do it on Windows, but on Unix, you just tell the program to write to /dev/full, and it'll get an ENOSPC.  See full(4).

  14. Asdf28 says:

    @kevin On linux, there are 2 more ways of simulating out of space errors. An LD_PRELOAD shim to wrap a few syscalls and make them fail on command. Or a FUSE filesystem to make a particular directory fail on command.

  15. Martin says:

    NTFS compression make file fragmented. If you have uncompresed file in 1 continuous space like DDDDDDDD and you enable NTFS compression, the data will look like: CFCFCFCF (D=data,C=compressed data,F=free space). Is is just a bug or is this behaviour forced by some NTFS feature ?

  16. Brian_EE says:

    @Dave – who says that they are using the "System Volume" for testing this? Maybe the application is being asked to try to open a file on a secondary volume (as part of the test) and it is that volume that they want the test code to fully allocate.

  17. AnotherScaryDay says:

    @Dave: keep in mind that the volume running out of disk space doesn't have to be the system volume, in which case it shouldn't be too disruptive.  And even when it is the system volume, maybe your application is expected to work in some acceptable form in those conditions (eg. think a disk cleaner type application) in which case such a test setup seems fair game to me, scary as it may be.

  18. DWalker says:

    Yes, something like /dev/full would be useful for Windows… Of course, if you are developing software, you can create a small disk partition and fill it nearly up, or all the way up (by whatever means necessary) and you'll know that no one else is writing to it.  Then you have your application write to THAT disk.  You can even label the disk "Dev_Full".  :-)  That might be exactly what the original customer was trying to do, but they were trying to fill the disk by creating a file with a size "slightly too large".  

    Yes, I have seen what happens when the system or boot volume (whichever one it is) fills up.  It is not pretty.

  19. 640k says:

    Why can't windows tell an application how big the largest creatable file is? It ain't rocket science.

  20. Piotr says:

    @640k:

    It /is/ rocket science, as the overhead for creating that file might depend on the properties of that file. It's like asking how much could fit into the boot of a car. I can easily tell you how much volume is empty, but I can't know beforehand how much place the necessary packaging will take up. It depends on the contents.

    The more important aspect however is that the information about the largest creatable file isn't very useful. I can't think of a use case outside of debugging. In production, it won't be very helpful, either – you never want your file systems to fill to the brim, and in any case, the information might be outdated the moment you get it. Only after writing your file to the disk will you know if the information was still accurate, and with that you're not better off than before.

  21. alegr1 says:

    @640k: because in the next moment that information is already obsolete. GetFreeSpace is not a promise, it's an advisory.

  22. alegr1 says:

    What shall we use

    To fill the empty spaces

    Where we used to talk?

    How shall I fill

    The final places?

    How should I complete the wall

  23. Joe says:

    I'm doing disk full testing on CF cards as I write (waiting for unit to boot.) I ended up creating a series of directories with empty files in each. This makes it really easy to adjust the free size by the sector.

  24. Nik says:

    > NTFS compression make file fragmented. If you have uncompresed file in 1 continuous space like DDDDDDDD and you enable NTFS compression, the data will look like: CFCFCFCF (D=data,C=compressed data,F=free space). Is is just a bug or is this behaviour forced by some NTFS feature ?

    This is a feature. The sum of the physical extents does not change. It uses compression units to compress one set of clusters to something smaller. Then the freed clusters get marked as "sparse" and do not occupy space on the disk. Thsi is so taht the sum of all the extents is >= uncompressedfile size. Depending on things, the actual data clusters might be contiguous. So the file only "looks" fragmented. brilliant, actually

  25. Nik says:

    Volume Shadow Copies also will take up size; data is need to track changes like name and timestamp modifications. On that note, on a 100% full NTFS volume you might not be able to rename files into something longer.

  26. Cesar says:

    The opposite could also happen: the filesystem tells you it has X free bytes, you try to create a file with exactly X bytes, and the filesystem still has some free space left after that. For instance, if the filesystem compressed the file you just created so it used less space, or if part of the file you just created was inlined in the MFT or similar, or if something noticed the filesystem was getting full and autonomously removed old saved snapshots or something like that.

  27. foo says:

    > If you want to get fancy, you could grow the file by large chunks until you can't add large chunks any more, and then switch to smaller and smaller chunks.

    I guess there's no chance of raising Zeno's paradox since computer data has a discreet base size.

  28. Medinoc says:

    [Would you prefer that Explorer show 256 bytes for the size of a 0-byte file? (To account for the metadata. And how do you charge for single-instance-stored ACLs?) Anyway, it doesn't matter; Explorer doesn't have insight into NTFS's internal data organization. It just calls GetCompressedFileSize(). -Raymond]

    Either "256 bytes", or "X bytes + metadata" where X is the actual size outside of the MFT (meaning a 1-byte file with moderate metadata would be "0 bytes + metadata".

    I think either would be better than just doing the "round the data length up to cluster size" on a system where this isn't accurate.

    Of course, I'm aware this is just a "perfect world" scenario, since in the Real World™ Explorer isn't *supposed* to have access to the underlying structure of a file system.

    (Note that I'm talking about the "Size on disk" field, not the "Size" one which is fine as it is).

  29. Someone says:

    I think the "Size on disk" field gets added to give users a feeling how much space (if any) is saved by NTFS File Compression for selected files. It can't and don't need to account for each and every metadata.

  30. Voo says:

    @medinoc how would you account for data stored with the actual inode (I mean whatever the ntfs equivalent is) in the face of hard links?

    Face it, it's useless to try to be overly accurate and will only cause confusion for the user.

  31. Anonymous Cow Herd says:

    Yes, "size on disk" would ideally take into account metadata (and therefore display 256 bytes, or whatever the exact number is, for an empty file). That's the whole purpose of the "size"/"size on disk" distinction; "size" is the number of bytes of file content, and "size on disk" is the total number of bytes allocated for this file.

    @Someone: if it was purely for that purpose, it wouldn't round up to a multiple of 4K.

  32. Gabe says:

    Anonymous Cow Herd: Think of "size on disk" as the amount of space that would be freed up by deleting the file. It doesn't take into account MFT records, directory entries, security descriptors, or anything else that is part of some other file.

  33. Someone says:

    @Anonymous Cow Herd "if it was purely for that purpose, it wouldn't round up to a multiple of 4K"

    Why not? Compresssed files still occupy a multiple of the cluster size. As a consequence, files smaller than this will still use 1 cluster, regardless of the compression ratio.

  34. Medinoc says:

    @Gabe: Ooh, that makes sense.

    @Someone: +1

  35. S says:

    This Size on disk field is the only one that shows you that your 50mb archived redo log file gets reduced to something like 20mb inside a directory with NTFS compression. That's important if your O***** database spits them out like crazy.

Comments are closed.

Skip to main content