Building the next generation file system for Windows: ReFS


We wanted to continue our dialog about data storage by talking about the next generation file system being introduced in Windows 8.  Today, NTFS is the most widely used, advanced, and feature rich file system in broad use. But when you’re reimagining Windows, as we are for Windows 8, we don’t rest on past successes, and so with Windows 8 we are also introducing a newly engineered file system. ReFS, (which stands for Resilient File System), is built on the foundations of NTFS, so it maintains crucial compatibility while at the same time it has been architected and engineered for a new generation of storage technologies and scenarios. In Windows 8, ReFS will be introduced only as part of Windows Server 8, which is the same approach we have used for each and every file system introduction. Of course at the application level, ReFS stored data will be accessible from clients just as NTFS data would be. As you read this, let’s not forget that NTFS is by far the industry’s leading technology for file systems on PCs.

This detailed architectural post was authored by Surendra Verma, a development manager on our Storage and File System team, though, as with every feature, a lot of folks contributed. We have also used the FAQ approach again in this post.
–Steven

PS: Don’t forget to track us on @buildwindows8 where we were providing some updates from CES. 


In this blog post I’d like to talk about a new file system for Windows. This file system, which we call ReFS, has been designed from the ground up to meet a broad set of customer requirements, both today’s and tomorrow’s, for all the different ways that Windows is deployed.

The key goals of ReFS are:

  • Maintain a high degree of compatibility with a subset of NTFS features that are widely adopted while deprecating others that provide limited value at the cost of system complexity and footprint.
  • Verify and auto-correct data. Data can get corrupted due to a number of reasons and therefore must be verified and, when possible, corrected automatically. Metadata must not be written in place to avoid the possibility of “torn writes,” which we will talk about in more detail below.
  • Optimize for extreme scale. Use scalable structures for everything. Don’t assume that disk-checking algorithms, in particular, can scale to the size of the entire file system.
  • Never take the file system offline. Assume that in the event of corruptions, it is advantageous to isolate the fault while allowing access to the rest of the volume. This is done while salvaging the maximum amount of data possible, all done live.
  • Provide a full end-to-end resiliency architecture when used in conjunction with the Storage Spaces feature, which was co-designed and built in conjunction with ReFS.

The key features of ReFS are as follows (note that some of these features are provided in conjunction with Storage Spaces).

  • Metadata integrity with checksums
  • Integrity streams providing optional user data integrity
  • Allocate on write transactional model for robust disk updates (also known as copy on write)
  • Large volume, file and directory sizes
  • Storage pooling and virtualization makes file system creation and management easy
  • Data striping for performance (bandwidth can be managed) and redundancy for fault tolerance
  • Disk scrubbing for protection against latent disk errors
  • Resiliency to corruptions with “salvage” for maximum volume availability in all cases
  • Shared storage pools across machines for additional failure tolerance and load balancing

In addition, ReFS inherits the features and semantics from NTFS including BitLocker encryption, access-control lists for security, USN journal, change notifications, symbolic links, junction points, mount points, reparse points, volume snapshots, file IDs, and oplocks.

And of course, data stored on ReFS is accessible through the same file access APIs on clients that are used on any operating system that can access today’s NTFS volumes.

Key design attributes and features

Our design attributes are closely related to our goals. As we go through these attributes, keep in mind the history of producing file systems used by hundreds of millions of devices scaling from the smallest footprint machines to the largest data centers, from the smallest storage format to the largest multi-spindle format, from solid state storage to the largest drives and storage systems available. Yet at the same time, Windows file systems are accessed by the widest array of application and system software anywhere. ReFS takes that learning and builds on it. We didn’t start from scratch, but reimagined it where it made sense and built on the right parts of NTFS where that made sense. Above all, we are delivering this in a pragmatic manner consistent with the delivery of a major file system—something only Microsoft has done at this scale.

Code reuse and compatibility

When we look at the file system API, this is the area where compatibility is the most critical and technically, the most challenging. Rewriting the code that implements file system semantics would not lead to the right level of compatibility and the issues introduced would be highly dependent on application code, call timing, and hardware. Therefore in building ReFS, we reused the code responsible for implementing the Windows file system semantics. This code implements the file system interface (read, write, open, close, change notification, etc.), maintains in-memory file and volume state, enforces security, and maintains memory caching and synchronization for file data. This reuse ensures a high degree of compatibility with the features of NTFS that we’re carrying forward.

Underneath this reused portion, the NTFS version of the code-base uses a newly architected engine that implements on-disk structures such as the Master File Table (MFT) to represent files and directories. ReFS combines this reused code with a brand-new engine, where a significant portion of the innovation behind ReFS lies. Graphically, it looks like this:

NTFS.SYS = NTFS upper layer API/semantics engine / NTFS on-disk store engine; ReFS.SYS = Upper layer engine inherited from NTFS / New on-disk store engine

Reliable and scalable on-disk structures

On-disk structures and their manipulation are handled by the on-disk storage engine. This exposes a generic key-value interface, which the layer above leverages to implement files, directories, etc. For its own implementation, the storage engine uses B+ trees exclusively. In fact, we utilize B+ trees as the single common on-disk structure to represent all information on the disk. Trees can be embedded within other trees (a child tree’s root is stored within the row of a parent tree). On the disk, trees can be very large and multi-level or really compact with just a few keys and embedded in another structure. This ensures extreme scalability up and down for all aspects of the file system. Having a single structure significantly simplifies the system and reduces code. The new engine interface includes the notion of “tables” that are enumerable sets of key-value pairs. Most tables have a unique ID (called the object ID) by which they can be referenced. A special object table indexes all such tables in the system.

Now, let’s look at how the common file system abstractions are constructed using tables.

Object table: Object ID, Disk Offset & Checksum. Arrow to Directory: File name, File metadata; File Metadata: Key, Value; File extents: 0-07894, Disk offset and checksums; 7895-10000, Disk offset and checksums; 10001-57742, Disk offset and checksums; 57743-9002722, Disk offset and checksums

File structures

As shown in the diagram above, directories are represented as tables. Because we implement tables using B+ trees, directories can scale efficiently, becoming very large. Files are implemented as tables embedded within a row of the parent directory, itself a table (represented as File Metadata in the diagram above). The rows within the File Metadata table represent the various file attributes. The file data extent locations are represented by an embedded stream table, which is a table of offset mappings (and, optionally, checksums). This means that the files and directories can be very large without a performance impact, eclipsing the limitations found in NTFS.

As expected, other global structures within the file system such ACLs (Access Control Lists) are represented as tables rooted within the object table.

All disk space allocation is managed by a hierarchical allocator, which represents free space by tables of free space ranges. For scalability, there are three such tables – the large, medium and small allocators. These differ in the granularity of space they manage: for example, a medium allocator manages medium-sized chunks allocated from the large allocator. This makes disk allocation algorithms scale very well, and allows us the benefit of naturally collocating related metadata for better performance. The roots of these allocators as well as that of the object table are reachable from a well-known location on the disk. Some tables have allocators that are private to them, reducing contention and encouraging better allocation locality.

Apart from global system metadata tables, the entries in the object table refer to directories, since files are embedded within directories.

Robust disk update strategy

Updating the disk reliably and efficiently is one of the most important and challenging aspects of a file system design. We spent a lot of time evaluating various approaches. One of the approaches we considered and rejected was to implement a log structured file system. This approach is unsuitable for the type of general-purpose file system required by Windows. NTFS relies on a journal of transactions to ensure consistency on the disk. That approach updates metadata in-place on the disk and uses a journal on the side to keep track of changes that can be rolled back on errors and during recovery from a power loss. One of the benefits of this approach is that it maintains the metadata layout in place, which can be advantageous for read performance. The main disadvantages of a journaling system are that writes can get randomized and, more importantly, the act of updating the disk can corrupt previously written metadata if power is lost at the time of the write, a problem commonly known as torn write.

To maximize reliability and eliminate torn writes, we chose an allocate-on-write approach that never updates metadata in-place, but rather writes it to a different location in an atomic fashion. In some ways this borrows from a very old notion of “shadow paging” that is used to reliably update structures on the disk. Transactions are built on top of this allocate-on-write approach. Since the upper layer of ReFS is derived from NTFS, the new transaction model seamlessly leverages failure recovery logic already present, which has been tested and stabilized over many releases.

ReFS allocates metadata in a way that allows writes to be combined for related parts (for example, stream allocation, file attributes, file names, and directory pages) in fewer, larger I/Os, which is great for both spinning media and flash. At the same time a measure of read contiguity is maintained. The hierarchical allocation scheme is leveraged heavily here.

We perform significant testing where power is withdrawn from the system while the system is under extreme stress, and once the system is back up, all structures are examined for correctness. This testing is the ultimate measure of our success. We have achieved an unprecedented level of robustness in this test for Microsoft file systems. We believe this is industry-leading and fulfills our key design goals.

Resiliency to disk corruptions

As mentioned previously, one of our design goals was to detect and correct corruption. This not only ensures data integrity, but also improves system availability and online operation. Thus, all ReFS metadata is check-summed at the level of a B+ tree page, and the checksum is stored independently from the page itself. This allows us to detect all forms of disk corruption, including lost and misdirected writes and bit rot (degradation of data on the media). In addition, we have added an option where the contents of a file are check-summed as well. When this option, known as “integrity streams,” is enabled, ReFS always writes the file changes to a location different from the original one. This allocate-on-write technique ensures that pre-existing data is not lost due to the new write. The checksum update is done atomically with the data write, so that if power is lost during the write, we always have a consistently verifiable version of the file available whereby corruptions can be detected authoritatively.

We blogged about Storage Spaces a couple of weeks ago. We designed ReFS and Storage Spaces to complement each other, as two components of a complete storage system. We are making Storage Spaces available for NTFS (and client PCs) because there is great utility in that; the architectural layering supports this client-side approach while we adapt ReFS for usage on clients so that ultimately you’ll be able to use ReFS across both clients and servers.

In addition to improved performance, Storage Spaces protects data from partial and complete disk failures by maintaining copies on multiple disks. On read failures, Storage Spaces is able to read alternate copies, and on write failures (as well as complete media loss on read/write) it is able to reallocate data transparently. Many failures don’t involve media failure, but happen due to data corruptions, or lost and misdirected writes.

These are exactly the failures that ReFS can detect using checksums. Once ReFS detects such a failure, it interfaces with Storage Spaces to read all available copies of data and chooses the correct one based on checksum validation. It then tells Storage Spaces to fix the bad copies based on the good copies. All of this happens transparently from the point of view of the application. If ReFS is not running on top of a mirrored Storage Space, then it has no means to automatically repair the corruption. In that case it will simply log an event indicating that corruption was detected and fail the read if it is for file data. I’ll talk more about the impact of this on metadata later.

Checksums (64-bit) are always turned on for ReFS metadata, and assuming that the volume is hosted on a mirrored Storage Space, automatic correction is also always turned on. All integrity streams (see below) are protected in the same way. This creates an end-to-end high integrity solution for the customer, where relatively unreliable storage can be made highly reliable.

Integrity streams

Integrity streams protect file content against all forms of data corruption. Although this feature is valuable for many scenarios, it is not appropriate for some. For example, some applications prefer to manage their file storage carefully and rely on a particular file layout on the disk. Since integrity streams reallocate blocks every time file content is changed, the file layout is too unpredictable for these applications. Database systems are excellent examples of this. Such applications also typically maintain their own checksums of file content and are able to verify and correct data by direct interaction with Storage Spaces APIs.

For those cases where a particular file layout is required, we provide mechanisms and APIs to control this setting at various levels of granularity.

At the most basic level, integrity is an attribute of a file (FILE_ATTRIBUTE_INTEGRITY_STREAM). It is also an attribute of a directory. When present in a directory, it is inherited by all files and directories created inside the directory. For convenience, you can use the “format” command to specify this for the root directory of a volume at format time. Setting it on the root ensures that it propagates by default to every file and directory on the volume. For example:

D:\>format /fs:refs /q /i:enable <volume>

D:\>format /fs:refs /q /i:disable <volume>

By default, when the /i switch is not specified, the behavior that the system chooses depends on whether the volume resides on a mirrored space. On a mirrored space, integrity is enabled because we expect the benefits to significantly outweigh the costs. Applications can always override this programmatically for individual files.

Battling “bit rot”

As we described earlier, the combination of ReFS and Storage Spaces provides a high degree of data resiliency in the presence of disk corruptions and storage failures. A form of data loss that is harder to detect and deal with happens due to ­­­“bit rot,” where parts of the disk develop corruptions over time that go largely undetected since those parts are not read frequently. By the time they are read and detected, the alternate copies may have also been corrupted or lost due to other failures.

In order to deal with bit rot, we have added a system task that periodically scrubs all metadata and Integrity Stream data on a ReFS volume residing on a mirrored Storage Space. Scrubbing involves reading all the redundant copies and validating their correctness using the ReFS checksums. If checksums mismatch, bad copies are fixed using good ones.

The file attribute FILE_ATTRIBUTE_NO_SCRUB_DATA indicates that the scrubber should skip the file. This attribute is useful for those applications that maintain their own integrity information, when the application developer wants tighter control over when and how those files are scrubbed.

The Integrity.exe command line tool is a powerful way to manage the integrity and scrubbing policies.

When all else fails…continued volume availability

We expect many customers to use ReFS in conjunction with mirrored Storage Spaces, in which case corruptions will be automatically and transparently fixed. But there are cases, admittedly rare, when even a volume on a mirrored space can get corrupted – for example faulty system memory can corrupt data, which can then find its way to the disk and corrupt all redundant copies. In addition, some customers may not choose to use a mirrored storage space underneath ReFS.

For these cases where the volume gets corrupted, ReFS implements “salvage,” a feature that removes the corrupt data from the namespace on a live volume. The intention behind this feature is to ensure that non-repairable corruption does not adversely affect the availability of good data. If, for example, a single file in a directory were to become corrupt and could not be automatically repaired, ReFS will remove that file from the file system namespace while salvaging the rest of the volume. This operation can typically be completed in under a second.

Normally, the file system cannot open or delete a corrupt file, making it impossible for an administrator to respond. But because ReFS can still salvage the corrupt data, the administrator is able to recover that file from a backup or have the application re-create it without taking the file system offline. This key innovation ensures that we do not need to run an expensive offline disk checking and correcting tool, and allows for very large data volumes to be deployed without risking large offline periods due to corruption.

A clean fit into the Windows storage stack

We knew we had to design for maximum flexibility and compatibility. We designed ReFS to plug into the storage stack just like another file system, to maximize compatibility with the other layers around it. For example, it can seamlessly leverage BitLocker encryption, Access Control Lists for security, USN journal, change notifications, symbolic links, junction points, mount points, reparse points, volume snapshots, file IDs, and oplocks. We expect most file system filters to work seamlessly with ReFS with little or no modification. Our testing bore this out; for example, we were able to validate the functionality of the existing Forefront antivirus solution.

Some filters that depend on the NTFS physical format will need greater modification. We run an extensive compatibility program where we test our file systems with third-party antivirus, backup, and other such software. We are doing the same with ReFS and will work with our key partners to address any incompatibilities that we discover. This is something we have done before and is not unique to ReFS.

An aspect of flexibility worth noting is that although ReFS and Storage Spaces work well together, they are designed to run independently of each other. This provides maximum deployment flexibility for both components without unnecessarily limiting each other. Or said another way, there are reliability and performance tradeoffs that can be made in choosing a complete storage solution, including deploying ReFS with underlying storage from our partners.

With Storage Spaces, a storage pool can be shared by multiple machines and the virtual disks can seamlessly transition between them, providing additional resiliency to failures. Because of the way we have architected the system, ReFS can seamlessly take advantage of this.

Usage

We have tested ReFS using a sophisticated and vast set of tens of thousands of tests that have been developed over two decades for NTFS. These tests simulate and exceed the requirements of the deployments we expect in terms of stress on the system, failures such as power loss, scalability, and performance. Therefore, ReFS is ready to be deployment-tested in a managed environment. Being the first version of a major file system, we do suggest just a bit of caution. We do not characterize ReFS in Windows 8 as a “beta” feature. It will be a production-ready release when Windows 8 comes out of beta, with the caveat that nothing is more important than the reliability of data. So, unlike any other aspect of a system, this is one where a conservative approach to initial deployment and testing is mandatory.

With this in mind, we will implement ReFS in a staged evolution of the feature: first as a storage system for Windows Server, then as storage for clients, and then ultimately as a boot volume. This is the same approach we have used with new file systems in the past.

Initially, our primary test focus will be running ReFS as a file server. We expect customers to benefit from using it as a file server, especially on a mirrored Storage Space. We also plan to work with our storage partners to integrate it with their storage solutions.

Conclusion

Along with Storage Spaces, ReFS forms the foundation of storage on Windows for the next decade or more. We believe this significantly advances our state of the art for storage. Together, Storage Spaces and ReFS have been architected with headroom to innovate further, and we expect that we will see ReFS as the next massively deployed file system.

— Surendra

FAQ:

Q) Why is it named ReFS?

ReFS stands for Resilient File System. Although it is designed to be better in many dimensions, resiliency stands out as one of its most prominent features.

Q) What are the capacity limits of ReFS?

The table below shows the capacity limits of the on-disk format. Other concerns may determine some practical limits, such as the system configuration (for example, the amount of memory), limits set by various system components, as well as time taken to populate data sets, backup times, etc.

Attribute

Limit based on the on-disk format

Maximum size of a single file

2^64-1 bytes

Maximum size of a single volume

Format supports 2^78 bytes with 16KB cluster size (2^64 * 16 * 2^10). Windows stack addressing allows 2^64 bytes

Maximum number of files in a directory

2^64

Maximum number of directories in a volume

2^64

Maximum file name length

32K 255 unicode characters (for compatibility this was made consistent with NTFS for the RTM product)

Maximum path length

32K

Maximum size of any storage pool

4 PB

Maximum number of storage pools in a system

No limit

Maximum number of spaces in a storage pool

No limit


Q) Can I convert data between NTFS and ReFS?

In Windows 8 there is no way to convert data in place. Data can be copied. This was an intentional design decision given the size of data sets that we see today and how impractical it would be to do this conversion in place, in addition to the likely change in architected approach before and after conversion.

Q) Can I boot from ReFS in Windows Server 8?

No, this is not implemented or supported.

Q) Can ReFS be used on removable media or drives?

No, this is not implemented or supported.

Q) What semantics or features of NTFS are no longer supported on ReFS?

The NTFS features we have chosen to not support in ReFS are: named streams, object IDs, short names, compression, file level encryption (EFS), user data transactions, sparse, hard-links, extended attributes, and quotas.

Q) What about parity spaces and ReFS?

ReFS is supported on the fault resiliency options provided by Storage Spaces. In Windows Server 8, automatic data correction is implemented for mirrored spaces only.

Q) Is clustering supported?

Failover clustering is supported, whereby individual volumes can failover across machines. In addition, shared storage pools in a cluster are supported.

Q) What about RAID? How do I use ReFS capabilities of striping, mirroring, or other forms of RAID? Does ReFS deliver the read performance needed for video, for example?

ReFS leverages the data redundancy capabilities of Storage Spaces, which include striped mirrors and parity. The read performance of ReFS is expected to be similar to that of NTFS, with which it shares a lot of the relevant code. It will be great at streaming data.

Q) How come ReFS does not have deduplication, second level caching between DRAM & storage, and writable snapshots?

ReFS does not itself offer deduplication. One side effect of its familiar, pluggable, file system architecture is that other deduplication products will be able to plug into ReFS the same way they do with NTFS.

ReFS does not explicitly implement a second-level cache, but customers can use third-party solutions for this.

ReFS and VSS work together to provide snapshots in a manner consistent with NTFS in Windows environments. For now, they don’t support writable snapshots or snapshots larger than 64TB.

Comments (167)

  1. jepessen says:

    Nice.. I'm really curious… The change of filesystem is a milestone in the windows developing…

  2. This sounds interesting (and I could update my TechNet Wiki article about Windows file systems): is there any kind of documentation available yet?

  3. MegaZone75 says:

    Woohoo, way above my head this material. My question: if it improces streaming, does that mean it also works excellent or better with cloud services?

  4. abdo says:

    oh thats really great to change filesystem…i think this step give more advetage for every user and will let all poeple like windows 8 more

    http://www.full-windows8.com/

  5. Though initially this will only ship with Windows 8 Server, when can we expect it with Windows 8 and can Windows 8 access a ReFS volume from a Windows 8 Server?

  6. pmbAustin says:

    Are things like Compression (which I use a fair amount) and Quotas likely to be implemented in the future?

  7. Cakes says:

    Please add support for HFS+ and ext4 in Windows 8. Thank you.

  8. Shredda says:

    WinFS 2.0?  Just saying…

    en.wikipedia.org/…/WinFS

  9. Mike S says:

    Very nice.  A few questions:

    1) You say that while ReFS does not provide 2nd level caching, but third parties can provide that feature.  What API are they using, and is that caching protected from errors?  It seems a little odd to have 3rd party code in the middle of a very critical part of a filesystem.  

    2) You said that correction is only available on mirrored spaces not parity spaces.  Is that just for ReFS, and that storage spaces is autocorrecting errors on disk that ReFS finds, or only errors that are found doing a parity space scrub, or not at all?

    3) What management tools are being created to administer these new features?

  10. I have a few questions:

    1. Will Windows 7 be able to read drives formatted with ReFS?

    2. Do you intend for ReFS to eventually repace NTFS on everything, with the Windows Server 8 version being similar to a pilot program?

    3. Will this be like the transition from FAT32 to NTFS where some operating systems can't read drives formatted by newer versions of Windows?

    4. Is this the rumored "Protogon" file system that you are now calling ReFS?

  11. Raphael says:

    I want that on my desktop machine too!!!

    I'm also not a fan of the 100 different Windows versions (Home/Premium/Professional/Ultimate)

    The ultimate thing would be, if there is only an ultimate version (and one server version)

  12. JohannesB says:

    The one thing I was looking for: "Maximum file name length 32K unicode characters". Solves the problem with the current limit of 255 characters. Very good job!

  13. Dan Bugglin says:

    "The NTFS features we have chosen to not support in ReFS are: named streams, object IDs, short names, compression, file level encryption (EFS), user data transactions, sparse, hard-links, extended attributes, and quotas."

    Oof, I know a number of high-profile applications that use some of these.  DOSBox and Steam use short file names and sparse files, respectively.  Steam can live without sparse files though (it supports FAT32), but it would likely slow down some disk operations by a ton. Compression is very useful to me since some file types I use read-only and have a high compression ratio, so it makes sense to keep them compressed, but the application I use them with does not support a compressed file format.

  14. TheKernel says:

    @WindowsVista567

    All answers are in the article, a quick read would reveal most answers either directly or indirectly:

    1. Will Windows 7 be able to read drives formatted with ReFS? Yes

    2. Do you intend for ReFS to eventually repace NTFS on everything, with the Windows Server 8 version being similar to a pilot program? Yes

    3. Will this be like the transition from FAT32 to NTFS where some operating systems can't read drives formatted by newer versions of Windows? If the system can read NTFS then it can read ReFS

    4. Is this the rumored "Protogon" file system that you are now calling ReFS? Probably

  15. Rand says:

    TheKernel, where are you reading that REFS will succeed NTFS?

    I don't believe that's stated either directly or indirectly, clearly the inability to boot from RTFS or use it for external storage strictly prevents it from replacing NTFS right now, and they make no statements whether either limitation will be changed.

    Without that it would be clearly impossible for it to replace NTFS.

  16. JohannesB says:

    @Rand

    "With this in mind, we will implement ReFS in a staged evolution of the feature: first as a storage system for Windows Server, then as storage for clients, and then ultimately as a boot volume. This is the same approach we have used with new file systems in the past."

  17. Bitcrazed says:

    @Cakes – if you want an OS that supports Linux filesystems and file formats, then go use Linux.

  18. de@iru.ch says:

    First of all: THANK YOU VERY MUCH for this very interesting article. No marketing blah blah, just the interesting stuff.

    After reading the article two questions remain:

    1. You write that you found Log Structured Filesystems to be "… unsuitable for the type of general-purpose file system required by Windows.". Do you plan to publish your results? I couldn't find any papers regarding this problem. With the Microsoft's scale, technology and battery of tests I'm sure you could give the file system community valuable insights.

    2. Why didn't you store the file data right in the tree?

    I'm looking forward to see real world benchmarks and detailed design decisions of this file system. Thanks a lot!

  19. Josh says:

    I'm confused. If a storage pool can only be 2^52, yet a volume can be 2^78… where are those extra bytes going to be stored? How can you have a volume that is larger than the pool that presumably supports it?

    Also, I second pmbAustin's question: We curently use Quotas, Compression and hard-links; are those going to be implemented? We've saved 100s of GBs every day with our usage of hard links.

  20. NotMe says:

    @WindowsVista567

    @TheKernel

    No, I don't think windows 7 will read ReFS unless they release a driver for it (I doubt that happeneing).  

    If a computer can read NTFS has nothing to do with reading ReFS.  The API used to access the files are the same, but not how they are stored on disk (Think of it as the implementation of the API).  It's the same problem as a Microsoft operating system reading a linux drive.  It needs file system drivers to be loaded.

    But overall, this combined with Storage Spaces sounds like ZFS for windows.  And that makes me happy.

  21. Daniel Neely says:

    @JohannesB  I'm not sure it will.  NTFS already supports 256 char file names, but you have to use alternate versions of all the win32 file APIs in order to take advantage of it.  The default versions don't support super long paths because of the buffer overflow potential in huge numbers of badly written programs.  

    The only 3rd party userspace app I've seen which (partially) supported it is IBM Rational Clearcase (source control app), which will write >256 paths when copying files from the server to your computer.  It's check for changes and update routine however barfed on them. (This might be fixed in a newer version than I've used; my employer has deprecated CC, so I might never have gotten the latest and greatest.)

  22. @NotMe, WindowsVista567

    A ReFS volume cannot be read by a Windows 7 PC. However, the Windows 7 PC will recognize that the volume is formatted with ReFS and will display an appropriate message indicating there's a valid file system (as opposed to treating the volume as if it were raw.)

    For more details, see: msdn.microsoft.com/…/dd442652(v=vs.85).aspx

    @Mike S

    The combination of ReFS and Storage Spaces will do automatic correction only on a mirrored Storage Space. ReFS on a Simple Space or a Parity Space or any 3rd party storage subsystem will not do any automatic correction; however, it will log detected errors. (Note that detection and correction of errors for user data requires integrity streams.)

    Additionally, when the scrubber runs, it will trigger automatic correction only on a mirrored Storage Space.

  23. Mike S says:

    Matt, thanks for the quick reply.  I guess I don't understand exactly what parity is supposed to do.  If I lose a disk completely, I can replace the dead disk, and the parity will then be recalculated across the space,  correct?

    if that's true, why doesn't the same logic apply to a slab?  I can just as easily reconstruct the correct data for a slab, and then correct the bad data, and maybe not use that sector on the disk again, etc…  

    If it doesn't do this, then if I get a bad sector that corrupts a slab, the slab still has good data because of the parity calculation, so a read works fine.  But how do I clear the known fault?  Do I have to get a bunch of new disks, create a new space, and then copy from the old space to the new space, and then go back and reclaim the old space's storage?

    In other words, I am trying to understand how as an administrator I deal with a disk that has has some sectors that are bad on it.

  24. Jason Stangroome says:

    "The NTFS features we have chosen to not support in ReFS are: named streams" – does this impact the way files downloaded from the Internet get marked as Blocked (ie Zone.Identifier alternate data streams) on an ReFS file system?

    Regards,

    Jason

  25. Karl says:

    Will you be releasing the source to ReFS? Filesystem readability on different operating systems is a major issue these days and opening the source under a non-restrictive license would help tremendously.

  26. soreil says:

    Will this be an open filesystem I could read/write to from other OSses like OSX Linux and BSD? That is the only thing I care for at the moment since otherwise it just throws up too many hurdles in my daily use.

  27. Vinay says:

    While your at it, can you replace the kernel too.

  28. @Matt Garson [MSFT]

    I was afraid of that. It looks like the FAT32 problem all over again.

  29. Finally a new file system. hopefully this one will be faster and more secure than NTFS. and when will Windows 8 be able to boot from ReFS? if Windows is going to have a new file system, i want to use it now!

  30. "ReFS, (which stands for Resilient File System), is built on the foundations of NTFS, so it maintains crucial compatibility…"

    Very good.

    "This file system, which we call ReFS, has been designed from the ground up…"

    …Wait, what? Clearly it's not designed from the ground up if it's built on the foundations of NTFS maintaining compatibility. What are you trying to say here?

  31. Ed says:

    Can't properly read a ReFS file system with an older OS? That's going to suck….

  32. sreesiv says:

    So sad this is not coming to the client now. 🙂

    Some comparison between NTFS <–> SS (client) and

    ReFS <–> SS (server) would have made it more interesting. Although it's not considered or planned to be competing technologies, it would have created some excitement for folks planning to deploy ReFS and SS for file server scenarios.

  33. @WindowsVista567

    Would you mind clarifying what you mean by the "FAT32 problem"? It sounds like you've got some past experience in this area and I'd like to understand better before responding.

  34. ZipZapRap says:

    Sorry, but what a snooze-fest. There's a reason why the "other guys" are eating your marketshare and Windows 8 in the consumer space is not in the front of people's minds.

    Another post about file systems? Jesus, do you guys GET consumers?

  35. Thank you for all your comments and feedback so far. Really appreciate it.

    @MikeS

    You have outlined some important failure cases for drives – i.e., Drives can fail entirely or develop bad sectors. Both types of storage spaces (parity and mirroring) are resilient to these kinds of failures. In the case of Parity spaces, parity is used to recompute the original data and heal the storage as you guessed, and the bad sector is not used from that point on, again as you guessed. I'd like to add that from the administrator's point of view all of this happens automatically – i.e., new blocks are allocated, the reconstructed data is copied over and the fault is effectively healed. The new space is allocated from spare capacity in your pool, including any hot spare disks. The applications are not aware of the faults as the system does this healing transparently.

    There is a separate class of errors for which support is currently present for mirrored spaces only. This has to do with automatic correction of checksum errors found in data (these errors are not detected by the disk).

    Hope this helps.

    @c_barth, @WindowsVista567, @NotMe, @TheKernel

    Regarding reading ReFS formatted partitions from Windows 7 — There are two ways to access the data stored in ReFS: The first one is to use Windows 8 Server that includes support for reading the disk format directly using the new filesystem driver we have created. The second one is to read it over the network by sharing out a folder from a Windows 8 Server. Once this folder is shared out as a file share, Windows 7 (and Windows 8) clients can access this data.

    Note that support for NTFS is going to be present in Windows for the foreseeable future, so you should always be able to access all your NTFS data across versions without any problems.

  36. Ivan Zhakov says:

    I'm really disappointed why ReFS doesn't support second level caching/hybrid storage using SSD.

  37. sean.e says:

    Great news.  I run Win 7 Ultimate as a home server.  Will there be a Foundation edition of Win 8 Server?  It may be time to switch from a client SKU to Foundation for the home file server when Win 8 Server is released.

  38. @JohannesB, @Daniel Neely

    Regarding path length limits, the following MSDN article provides excellent information which applies to ReFS just like it does to NTFS.

    msdn.microsoft.com/…/aa365247(v=vs.85).aspx

    @deiruch

    Regarding log structured filesystems – We want to optimize reads very carefully since they far outnumber writes and they affect response time more directly. This is better done by careful placement. We do want to reliably update metadata and checksum protected user data and for that we do write them to new locations like a log structured filesystem does. Ultimately, the difference can be narrowed down to allocation and defrag policies.

    Regarding storing file data directly in the tree – there is a tradeoff here. Storing data in the tree gets access to the part of the file that fits within the leaf page in the same I/O as the original lookup, but the rest of the data is on other nodes which causes extra IOs (similar to when the whole file is stored separately). It may also increase the depth of the tree. This is exactly the type of issue that makes designing filesystems challenging.

  39. B8Blog says:

    @ZipZapRap – "Consumers" generally don't read technical blogs about how products are built so we have chosen to focus on the details of building and engineering the product — "behind the scenes".  This isn't really a place where we are marketing the product to consumers.

    Since we've received feedback (and tried to respond to it) that we have not focused enough on the more foundational elements of Windows, we did a few posts on the file system.  What would you like to hear more about?

    THere are lots of ways to potentially classify posts, but we are working to be balanced.  We've done about 40 posts on engineering details with about 25 of them focused on "end-user" details (though still throuhg an engineering lens) and about 15 focused on what I think of as "core" or "foundational" topics.  It is not clear how to categorize some topics though like boot and setup, which can be viewed through either lens.

    Your feedback is welcome as to where you think we should best go.

  40. grumpy says:

    but it's not going to do anything about the miserable performance when accessing files, is it?

    It really is ridiculous how slow operations that touch lots of files are when compared to non-Windows filesystems.

  41. TheChucklesStart says:

    When is this coming to WHS?  I am most excited about this, was sad it was scrapped from WHS 2011.

    Will you ever support a raid-6 like scheme to support multiple drive failure for read only data?

  42. Willem says:

    With Mobile being the next big thing: why haven't we got a sophisticated filesystem that combines cloud and local storage in 1? It sucks for average consumers to have to think about Skydrive/PC/local cached stuff.

  43. It's great to see that you even reimagine the filesysteme in Windows 8!

    But when can i expect ReFS as a storage filesysteme for clients? You said it will come later, because of caution. Thats fully understandable, but i think professional users can't wait to use it on the client too!

    Is there a chance to get it in the client after a Service Pack release (SP1 or SP2) or do i have to wait future Windows releases?

  44. No sparse and hard-links? No-no-no, we need these. At least hard-links are a must.

  45. Hi,

    I still do not uderstand a lot of jargon said above..so pardon me.

    *. On Laptops and handhelds and tablets, the raid like behaviour (striping/mirroring) cannot be had as easily. how do they work on a single drive.

    *. on a Laptop..  ReFS can only be had on the second partion which is not a boot volume. no external connected  2tb drives, no boot volume… etc

    *. is there any SSD related improvements

    *. Is there any performance level improvements over NTFS?

    *. Please add a little support to recognise other Operating systems Names and preserve boot menu/order etc

  46. Chris says:

    @ thechuckesstart i completely agree – i want a storage pool with raid 6 / dual parity single parity just isnt good enough when you have 10+ disks!

  47. Lionel says:

    Nice!

    It's a shame there probably won't be a Server Core edition priced cheaply enough to be used for a home NAS.

    >does this impact the way files downloaded from the Internet get marked as Blocked (ie Zone.Identifier alternate data streams) on an ReFS file system?

    >but it's not going to do anything about the miserable performance when accessing files, is it?

    I would like to read your answer to that, too.

  48. Joao M Correia says:

    Just like the storage pools, this is another interesting feature.

    The only drawback i see is, as has been pointed out by others, the lack of some features that are heavily used (in my corporate environment at least), like quotas, hard links (junctions) and compression. Although i can quite happily live without compression on the file system (given deduplication is in and the data is appropriate), i can not quite get around losing the other two. Are these specs final?

    Joao

  49. ravloony says:

    "nothing is more important than the reliability of data"

    I agree. You need to release the spec for this so that the data is accessible from any OS.

  50. Lewis says:

    Am I correct in understanding that when using Integrity streams just the changed blocks will be written out to disk and checksummed?

  51. Asbjørn says:

    While you are at it, will you please finally get rid of that stupid 260 character limit for paths. NTFS supports much longer paths, and has for ages, but the shell is still limited to 260 characters. Come on, this is 2012, not 1995. Honestly, the first time I ran into that limit (and when doing programming, that happens easily!) I thought it was some kind of joke, but alas not.

  52. Alex says:

    What about performance of this file system?

  53. Alex says:

    I have found info about performance info Thanks

  54. FZB says:

    very nice article. removing the chkdsk outage windows will be nice

  55. Windows is an endangered species says:

    Still NTFS for all the consumers… ReFS only for server version…  =  goodbye Microsoft…

    Office maybe will remain for some years… it depends on the next versions (for windows and for Mac).

    Too much "touch" features and no other advancements in desktop features = few years left, even for Office…

  56. far says:

    nice post! a little dissapointed abt not able to boot using that, still will wait

  57. > Q) How come ReFS does not have deduplication, second level caching between DRAM & storage, and writable snapshots?

    > ReFS does not itself offer deduplication. One side effect of its familiar, pluggable, file system architecture is that other deduplication products will be able to plug into ReFS the same way they do with NTFS.

    But in Windows 8 server dev build deduplication exists. It will be removed in beta version or not?  Or it will be available only on NTFS volumes?

    Sorry if comment appears twice.

  58. Sinnfrei says:

    You should really consider to get this in the client versions fast. I would like to use this with Storage Spaces, but I'm not buying a server version for my home server. So I will wait with buying Windows 8 until it's working …

    By the way: Will all versions of Windows 8 eventually get full featured Storage Spaces and ReFS?

    And you really should get the auto correction working with parity, and implement multiple parity in Storage Spaces.

  59. Daniel Neely says:

    @Asbjørn, as pointed out by Surendra Verma above, win32-unicode does support 32k path lengths.  However the existing non-unicode APIs can't safely be changed to support longer paths because it would result in buffer overflows for any program that has the maximum length hardcoded at compile time.

  60. Alex says:

    nice post! a little dissapointed about the performance trade-offs

  61. One other question, that some have alluded to in previous comments: will this new file system be supported in Windows 8 Home Server? I currently use a Synology DiskStation because all the Windows Home Server implementations are quite lousy. In this day and age more and more people are getting servers in their home this is a major oversight and problem. NASs are very popular and they all pretty much run Linux.

    These home servers usually have multiple disks that have arrays that are auto-expanding. My Synology has an iTunes server, a photo server, a web file server, works as a TimeMachine backup point, supports AD and ACLS, can record webcams, even recording only when there is motion in the frame. There are a ton of other features. Then I have a netbook hooked to my TV to provide me with access to all sorts of entertainment. So I now have two devices running all the time in my house and neither can be used in both scenarios.

    We really need a version of Windows that can be used to fill both of these roles, coupled with hardware that is very low power such that it can be on all the time and still be relatively 'green'. I can't believe that I need to run Linux in my home to satisfy reasonable consumer needs when I'm otherwise a 100% Windows shop, even waiting patiently for a convertible tablet/netbook running Windows 8.

    I think it would be quite a service to the community if you talked about your plans for Windows 8 Home Server and how it can be used in the role that I've addressed above.

  62. ReFS seems to be a milestone in further improving data integrity, especially challenges like bit rot. I think MS has taken the right approach to implementing it in phases. There is no hurry to replace NTFS on the client as it does its job exceptionally well. But good heavens I hope, with the next Windows release at least following Windows 8, the deal breaking limitations in ReFS will be addressed like lack of support for object identifiers!, file compression and sparse files, encryption, transactions, Disk Quotas, alternate streams etc You have had the time to enable BitLocker on ReFS but not compression, EFS, quotas, ADS and object IDs?? I can't imagine using ReFS on the client with these limitations! Lack of these on the server is also rather lame as many of these NTFS features are used heavily by enterprises in server scenarios. I hope these limitations are only for the time being due to ReFS being in its first release and every single one of them will be implemented over time.

    Also, note that whenever ReFS comes to Windows client, full read-write backward compatibility with previous OSes is paramount. If Microsoft is adamant about bringing such a core new feature for free to older Windows releases, I request they make it available as a commercial addon IFS driver. Many customers may even be ready to pay extra to add ReFS read-write support to previous versions of Windows.

    Is there a performance overhead of enabling integrity streams?

  63. Nick Lowe says:

    "The NTFS features we have chosen to not support in ReFS are: named streams, object IDs, short names, compression, file level encryption (EFS), user data transactions, sparse, hard-links, extended attributes, and quotas."

    By "user data transactions", I'm assuming you mean Transactional NTFS (TxF)? This has seriously been discontinued? it was highly touted back when Vista came out.

    This is a highly breaking change to existing code and throughly pulls the rug… Ouch!

    Please don't release something so half baked and unfinished!

  64. ShaneM says:

    @StevenSinofsky

    Since you raised the question of what we might like to hear more about, I thought I'd answer.  I have been reading these posts since the beginning, and find them very informative.  The constant rant about the W8 UI hasn't really been addressed.  I think the best way to answer these criticisms is to provide a fictional enterprise user walkthrough.  

    For instance, let us say we have a high end accounting type person.  They like to use three displays (as I do).  The left is often fullscreen excel, the right is fullscreen accounting app.  The middle screen often has a small email window showing an email used as reference when working on screen 1 or 3. The middle screen also has Live Messenger open, as it is constanlty used to chat with branch offices.  Additionally, there may be a small IE window open to help with arcane Excel functions he is using.

    While the details may change, this is a fairly common setup (at least with two screens) for many people.  If you showed how someone would do some useful (but fictional) work with Windows 8 in this scenario, it would set quite a few minds at ease.  

    I commonly find that I have need to see all of the information mentioned in my example above.  Even as I write this entry, on my left screen I have IE open full screen, in my right screen I have Outlook open to monitor incomming emails, and in my center screen I have:

     notepad open with a few key points in a small window,

     the cricket open media center in a small window (yes I'm an Aussie!) ,

     Windows Messenger open in a small window.

    The only thing I can think of is that in Windows 8, I will spend almost all on my time in traditional desktop apps.  What about Messenger?  It would be great as a Metro app, but I need it to only take up a small amount of my valuable screen space.

    Sorry for the long, off-topic post, but I believe you can address peoples concerns with some carefully considered workflow examples.

  65. Only one word!

    "YEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEAH!"

  66. Rosch says:

    Hi Microsoft!

    Please don't drop the support for hardlinks! We use them DAILY to deduplicate hard disks.

    They are vital! If you really remove them I have to switch to Linux and ext4 as filesystem.

    Roland

  67. Klimax says:

    @those who miss hardlinks:

    You don't need hardlinks when you have full softlinks available. (Since Vista)

    Anyway, I'd like too ask which features are hardkilled. (i.e.: Are dropped permanently, right now it is bit uncertain)

    @ShaneM :

    Just wait for beta and you'll see how it'll work out for you. (Am too waiting with judgement for beta)

  68. Klimax says:

    Also, I'd like to see evidence for NTFS is "slow". So far didn't see any tests, just assertions….

    (And AFAIK no SSD/HDD review mentioned anything about this supposed "problem")

  69. Pusher Robot says:

    All you people complaining about the (currently) missing features: NTFS is not going away.  You won't have to use ReFS if it doesn't meet your needs, just continue doing what you are doing.

  70. Asbjørn says:

    @Daniel Neely: Yes, they do, but only via convoluted workarounds. To quote from MSDN (msdn.microsoft.com/…/aa365247%28v=vs.85%29.aspx):

    "In the Windows API (with some exceptions discussed in the following paragraphs), the maximum length for a path is MAX_PATH, which is defined as 260 characters. […] These prefixes are not used as part of the path itself. They indicate that the path should be passed to the system with minimal modification, which means that you cannot use forward slashes to represent path separators, or a period to represent the current directory, or double dots to represent the parent directory. Because you cannot use the "\?" prefix with a relative path, relative paths are always limited to a total of MAX_PATH characters."

    So basically, it is theoretically supported, but only just, and many things won't work. This is what needs to change. Windows needs to support longer paths *by default* and *everywhere*, and then use a compatibility shim for programs that don't work. It is especially embarrassing that Explorer is still limited to 260 characters. I don't want these limits forced on me, if I don't use any programs that have problems with long paths.

  71. rms says:

    Another proprietary and closed file system! Just what we needed!

  72. Nick Lowe says:

    I have just posted about this on my blog:

    http://www.nicklowe.org/…/pulling-the-rug-microsofts-refs

    Mapped to file system attribute flags, the following are not going to be supported:

       FILE_NAMED_STREAMS

       The file system supports named streams.

       FILE_SUPPORTS_OBJECT_IDS

       The file system supports object identifiers.

       FILE_FILE_COMPRESSION

       The file system supports file-based compression.

       FILE_SUPPORTS_ENCRYPTION

       The file system supports the Encrypted File System (EFS).

       FILE_SUPPORTS_TRANSACTIONS

       The file system supports transactions.

       FILE_SUPPORTS_SPARSE_FILES

       The file system supports sparse files.

       FILE_SUPPORTS_HARD_LINKS

       The file system supports hard links.

       FILE_SUPPORTS_EXTENDED_ATTRIBUTES

       The file system supports extended attributes.

       FILE_VOLUME_QUOTAS

       The file system supports per-user quotas.

    This is an incredible list of features that are not going to be supported in ReFS, at least in its initial implementation. It is likely to have huge compatibility implications and to break many of the existing applications that are in use today.

  73. I second the need for an article on a typical multi-monitor usage scenario with screen shots and video.

    We also need some clarity on development. Can WinRT be used to program windowed (desktop-based) applications? Can a single code-based be used for desktop and Metro-style applications? Can Metro-style applications be run on the desktop in a window? Can desktop applications be distributed as an appx file? Will these appx files then be automatically managed when Windows is Reset or Refreshed (per your last article on Windows recovery options)? Can a single application be distributed with the option to execute as a desktop or Metro-style application? If not, how will programs like notepad and calculator work? Will there be two copies of each program, one for desktop and the other for Metro? If so, how will I tell them apart on the start screen?

  74. @ShaneM ( @StevenSinofsky)

    I sort of agree with you ShaneM, and I also have some interesting ideas for Enterprise scenarios that make use of Metro.

    I've even emailed StevenSinofsky in the hopes that someone could put together a scenario/demo when the beta launches.

    What I can imagine in the scenario that you offer is that your mail app will exist as a metro app, messenger will exist with a metro app, and possibly better due to Messenger now supporting xmpp enabling support for Messenger, Yahoo, FB, AND Lotus Sametime.

    Your reliance on traditional desktop apps will wane whe you start tiling metro apps across your many screens.

    I would conceive that one of your desktops may have excel (with PowerPivot) and another your accounting application, while the third will have the metro apps side by side by side showing you the content that is important to a knowledge worker such as yourself, such as the Cricket match and your emails and chats.

    In the scenario I gave to the Windows 8 development team I also outlined creating Metro 'wrappers' around existing LoB portals (ERP/CRM/etc.) in order to enable share and seach charms across those legacy server applications.

    The value proposition in data integration at the deskktop is HUGE.  Average users will be able to move data between these systems reliably, with IT confidence, as the data will be encapsulated by WinRT and not haphazardly copied and pasted, or dare I say, rekeyed.  The other alternative would be long development cycles trying to create rich integrations on the server side, which can be quite costly and rather limited in their scope due to complexity.

    The Windows Runtime will be the core of every windows OS you use in the future due to it's elegance and simplicity.  The amount of plumbing or boilerplate code has been reduced to something even the most novice programmer can manage.

    With that said, here's hoping such a demo can be put together.

    And now, something that is not off topic.

    I am looking forward to the ReFS.  Looks like a lot of time and engergy went into the design.

    I also enjoy how the windows team this time view Windows 8 and all its subsystems as a reimagining.  It tells me there are juices of creativity flowing which will spark some interesting innovation once vendors and OEMs get to do their part.  Here's hoping the ecosystem becomes as colourful as these windows 8 blog posts.

    Windows 8 is not your father's Windows.  For that I will be eternally grateful.  Now who wants to earn a Ship It award??????

  75. Andre says:

    @ Steven Sinofsky, some very important issues:

    1-what will be done to improve performance?

    2-what will be done to reduce the fragmentation of files?

    3-What is the forecast for refs on Windows 8 client?

    4-Why not stop with the SKUs? Stop, this only makes the Microsoft lose customers. Copying Apple.

    Love the Windows, but it needs to improve.

  76. jader3rd says:

    "The Integrity.exe command line tool is a powerful way to manage the integrity and scrubbing policies."

    No! Some people on the Windows team understand, but others do not. Do not use executables to manage, maintain or configure Windows, use PowerShell cmdlets. I wish a bunch of Windows exe's were deprecated, and certainly new ones never introduced. Cmd line tools will never be as powerful or as easy to use as PowerShell cmdlets for managing Widnows. Please don't introduce more exe's. Every time I have to deal with an exe while managing Windows it's a pain compared to PowerShell cmdlets.

    On a different note, is you can't make the OS disk ReFS it sounds like it still has a way to go. What rarely used NTFS feature does Windows use, preventing Windows from running on ReFS?

  77. I guess when ReFS is brought to the Windows client or when ReFS supports booting, many limitations will be addressed. Few examples: The Distributed Link Tracking service in NT depends on Object IDs. The servicing stack depends on hard links. IE adds a named stream to every file it downloads. MS can't possibly take away the Encrypting File System feature out of Windows. Or compression or sparse files or quotas or transactions?

  78. Nick Lowe says:

    @xpclient – File system transactions (TxF) are also extensively used:

    "As for platform stability, this is achieved by some of the ways that Microsoft is using TxF in its own technologies. There are three core features in Windows Vista and Windows Server "Longhorn" that now make use of Transactional NTFS: Windows Update, System Restore, and Task Scheduler. All of these use TxF to write files to the file system within the scope of a transaction in order to handle rollback/commit in case of any exceptions, such as a system reboot due to a loss of power. If you’ve ever experienced a loss of power right in the middle of Windows Update, you know this can leave your system dead and in need of a fresh installation. But now that Windows Update uses TxF, if this same situation occurs on your Windows Vista system, the computer can recover by rolling back the transaction when the system reboots. By adopting TxF internally, Microsoft has helped make its own operating system more stable."

    msdn.microsoft.com/…/cc163388.aspx

  79. What i do not understand is the fact that NTFS supports 32k file names but the Windows Explorer in Windows 7 is not able to manage such files? Why this limitation? It is quite nasty to get error messages when working with rather large file names.

    Can you give some details whether the performance for basic tasks  like copy / delete / create is better comparted to NTFS or is it likely to be the same?

  80. asdf says:

    @Surendra Verma [MSFT]: Does ReFS have automatic sparse files? e.g. the file extents table looks like it can check if an entry contains a block of all 0's during checksumming and decide not to store it.

  81. Dave says:

    No more file locking of open files, please??  If you're going to break backwards compatibility, really do it on that one.  I want to be able to delete a file that is currently open by another process and have things work like Linux.

    Your current semantics of locking open files makes windows more costly to support in a heterogeneous environment!  Just do it!  Promise us you will!!!

  82. Hermann Schinagl says:

    Dropping Hardlinks and Named Streams *really* hurts.

    Hardlinks are the core of any incremental backup aka timemachine, and symbolic links can not be used as a fill in here. I am the author of a little freeware tool, which does incremental backup and this really smashes any 'Delorean Copy' aka timemachine like backups. See schinagl.priv.at/…/ln.html

    On the other hand we heavily use named streams, to provide checksums to arbitrary files, and give them kind of 'free xml based attributes' in this alternate named stream. How do we transport this without streams?

    When do you ship ReFS for the first time? Can we have this in a normal windows8 beta(not server) just to try out file system features?

    Maybe you rethink your decision with respect to hardlinks and named streams, but I guess it is too late for the first version

    To be honest I am little bit disappointed.

      BR Hermann

  83. Mike Cramer says:

    For those that are doing backup operations, are there any reasons you're not using shadow copies to perform these functions?

  84. Mike Cramer says:

    My only request though I know it will go unheard:

    ReFS should be made available to Linux in some shape or form where the fiasco that has been NTFS drivers should go away. This should NEVER have been an issue. Please do something about it. You don't even have to GPL the code, make a proprietary but 100% compatible kernel module.

  85. Steve H says:

    Would there be any downside for me using Windows 8 Server as my desktop OS?

  86. Alvaro says:

    I agree with @jader3rd new Command Line exe's should't be introduced … PowerShell 3 it's comming

    @StevenSinofsky: Me too, i want to see some multitasking and/or multimonitor use examples, to set the record straigh about one (and the biggest) shortcommings of w8 (at least apparently)

    Alvaro 😉

  87. @Matt Garson [MSFT]

    "FAT32 problem" is probably not the best way to phrase my statement. I am referring to the fact that no Windows 9x operating system can read NTFS drives. Not even Windows ME, which was released after Windows 2000, can read NTFS drives. It might be better to call it the "NTFS problem," but NTFS is superior to FAT32 and I consider the problem to be related to the fact that Windows 9x must use a FAT file system to store data, and that disks often have to be specially formatted to work with Windows 9x computers. In the same way, once ReFS is expanded and becomes more common, we may see a point where there are disk compatibility problems on certain computers because those computers cannot read the file system of some drives.

  88. What about MinWin? says:

    Building the next generation kernel for Windows: MinWin would be also a good post and your work on MinWin!

  89. Klimax says:

    @Asbjørn 17 Jan 2012 7:27 AM:

    Shim every single application which used ANSI version of api or which compiles with Unicode support yet hardocodes max length? Remember, in headers many functions are compile time defined either to A or W versions, so many programms use W version yet they have original limit.

    I don't think it would be efficient use of space…

    However so far no such limitation is mentioned for new CreateFile2 (msdn.microsoft.com/…/hh449422(VS.85).aspx).

  90. Thanks for all your comments and feedback so far.  More responses below.

    @Bastien92, @Sinnfrei, @Andre, @xpclient,

    Regarding when ReFS will be available on the client, we’re following a deliberate approach where we evolve it in a staged manner starting with Server first, then on the Client and ultimately on the boot volume. We expect NTFS to be supported and available with all of its features. We do like to hear feedback about features, so that is highly appreciated.

    @ravloony,

    Regarding accessing ReFS from OS’s other than Windows 8 Server – the data is accessible via a network file share over both SMB and NFS protocols that are very widely adopted. You would have to expose the ReFS volume as a file share, which is then visible to the other OS as a folder to work on.

    @Lewis

    “Am I correct in understanding that when using Integrity streams just the changed blocks will be written out to disk and checksummed?”

    Yes that is correct.

    @Alex

    “What about performance of this file system?”

    We believe that it will meet the performance requirements of general file server scenarios. We would be interested in learning about the performance of your specific scenario, as such feedback is very valuable to us.

    @FZB

    “very nice article. removing the chkdsk outage windows will be nice”

    Thank you.

    @xpclient

    “Is there a performance overhead of enabling integrity streams?”

    There is a small CPU cost in computing checksums. Other than that, there is an additional cost to store the updated checksums with the new data. We expect these costs to be acceptable given the benefit provided by checksums.

  91. Mike says:

    @Surendra Verma [MSFT]

    So ReFS will be in the Client in the RTM release?

    I am confused, your statement is not clear for me.

  92. @Mike

    I'm sorry I wasn't clearer. I meant to say that we will evolve across releases of Windows. This gives us a chance to get valuable feedback.

  93. Bugs says:

    Would windows 8 have multiple desktops support like ubuntu and mac?

  94. @Surendra Verma [MSFT]

    Ahh.thx I get it!

  95. Alvaro says:

    @Surendra Verma

    What about  Integrity.exe command line tool, any plans on fully integrating ReFS with powerShell ????

    "Windows PowerShell cmdlets allow administrators automate management tasks …"

  96. … So those filenames can now be long like on Unix based OSs?

    and in Windows 9, I assume we'll be allowed to boot off this OS?

    how does ReFS handle fragmentation?

  97. You guys really have no clue how happy I am. this is like an amazing Christmas present. you guys rawk.

  98. Natalia Portillo says:

    girlyngeek.blogspot.com/…/microsoft-returns-customers-to-past.html my comparison of features of ReFS with other filesystems.

  99. WinFS never was a filesystem tho bro…

    anywho, I thought up this idea lastnight, here it is. hopefully you'll implement this one too (;

    Microsoft, you should give up on constantly indexing the HDD/SDDs and instead save advanced metadate in teh MFT, and when a search is performed, you simply consult the MFT, *** you have to anyway. and it'd free u[ resources as well.

  100. @Willem

    We don't have device and cloud storage unified like that because bandwidth just isn't there yet.

  101. Mike P says:

    Although the newer system has advanages, would love simple features like hot spare for Servers and think it would be a great marketing to allow "Professional" verions of OS's support a basic RAID 1 Mirror. Practically every mother board can do it, why cant Windows?

  102. Why don't you all just do something like. three versions of Windows 8?

    Server (You can strip it down however you need with AIK) Home, and Ultimate.

    discuss?

  103. Whoa. Disturbance in the force… A girl has arrived… Hide your motherboards boys!

  104. Axel says:

    No-boot, no-sparse, no-hardlinks, no-EFS, no-tx, NO GO!

    The maximum path length has been 30K Unicode chars in NTFS since 1992, and NTFS was available on workstation and server AND bootable from day one in the very first NT beta. Where were you Surendra? 🙂

    Maybe 5 years from now when your telemetry tells you no one uses ReFS (mostly because the forgotten features breaks far more apps than you thought, including many not released yet) we'll get a proper "reimagined" NTFS 8 with most of the touted advantages of ReFS. *THAT* happened before…

  105. LTD says:

    @MSFT

    Does REFS deal with the issue of a corrupted boot sector or damage to the B+ trees due to bad sectors etc?

    Are there any improvements to the file system that removes the need to defrag or will that still be handled (for non-SSD) by 3rd party apps?

    With the removal of hard links will there be a facility to relocate user profiles and documents to a non-boot device (don't need most of it on an SSD)

    @ShaneM

    Metro is a fisaco and useless for creation or management of content and data. Using a 30" monitor for a calculator is just one example of the stupidity. We're going to have to live with it and hack the registry to work around it, or simply don't upgrade.

    There are a lot of posts about metro and it’s lack of usability on non-touch devices, MS knows it’s generally hated but they need a “tablet interface” so the useful windowed environment must die for MS BOB 2.0 (we all know how well THAT was received) to live.  Next they’ll bring back “clippy”.

  106. I'm really surprised to see that compression is not going to be implemented on the ground of "complexity and limited value". I just don't understand the complexity argument.

    I agree that the compression algorithm in NTFS was too slow for today's HDD (especially with interleaving properties of Pool Spaces), but then, there are newer algorithms out there which are much faster (a few hundred of MB/s per core !) and compress better. In fact, a compressor which decodes very fast will even **improve** HDD read performance as perceived by the application. And since this is a classical bottleneck, such property will have a positive impact on user experience….

  107. Nigel says:

    For Windows 8 news I have created an aggregator that you guys can follow:

    win8tabs.com/windows-8-news-reviews

  108. The article made it clear that ReFS and NTFS do not require Storage Spaces (although there are benefits to deploying them together), but I am wondering if the reverse is true. Does a storage space care if it is formatted NTFS/ReFS, or does it function at the block-level only? In other words, will a storage space (whether simple or resilient) be seen as a literal virtual hard drive that can be formatted with any file system? Can you format a storage space with FAT32, ext3, or any other arbitrary file system that a 3rd party program running on Windows was written to create/handle? or must you absolutely use either NTFS or ReFS if you are using storage spaces?

  109. Mubeen M says:

    I'm amazed to see this as MS came up with WinFX couple of years back.

  110. Nick Lowe says:

    @Mubeen M

    WinFX ran on top of NTFS, it was never a file system.

  111. @IsoUnit

    Regarding storage space being used by file-systems other than ReFS/NTFS – yes, a key point about our architecture is that any arbitrary filesystem can be formatted on storage spaces.

    @LTD

    “Does REFS deal with the issue of a corrupted boot sector or damage to the B+ trees due to bad sectors etc?”

    If ReFS is running on top of Storage Spaces, then bad sectors detected by the disk are healed by Storage Spaces. Other corruptions get detected by ReFS which can interact with Storage Spaces to heal. When ReFS is not running on a mirrored space it can heal B+ trees by scavenging out the bad elements. In addition, ReFS also stores a copy of the boot block that can help recover from a corrupt boot block.

    @Mike P

    We do support hot spares for our Storage Space pools.

    @Andre, @LTD

    Regarding file fragmentation –

    The hierarchical allocators in ReFS allow it to find better allocation more quickly. For example, related metadata blocks are naturally placed closer to each other. Another example is large allocations for large files: it is easy for ReFS to scan for excellent placement of large allocations by consulting the proper layer in the allocator hierarchy.

  112. Thom Denholm says:

    A focus on Data Integrity, B+ trees for file system metadata and allocations – it's nice to see that Microsoft is learning! Glad also to (finally) see solid support for Flash media and handling for power interruptions, though "Torn Write" is not a term I've often heard. Count me in for more news on file systems for sure!

    Of course, anyone who wants this support today can get that and more from Datalight's Reliance File System, protecting user data on flash-based devices since 2003 on mobile and embedded platforms.

  113. ac says:

    Hello. Will this filesystem finally support API's to split large files into two or more parts at arbitrary offsets without copying? I know that in the context of existing file format, it may not make sense or be very appealing. But it can be very appealing for developers of new file formats and editing applications.

  114. database software solutions says:

    Very informative. Thanks for sharing.

    http://www.sevenit.com/Software-Solutions.php

  115. Mike S says:

    @Surendra Verma [MSFT]

    Thanks for the reply.  Let me push on this a little bit more if I might.

    I understand that the filesystem can only repair corruption detected in a space that is mirrored.  But can ReFS report up to the administrator when corruption is detected in a parity volume, even if it can't be repaired?

    For example, if the file with the corrupted extent was a video file, an error may be only slightly noticeable on playback, so you wouldn't necessarily want to remove the file.   If the file was an executable, the error would be fatal, but the administrator might be able to restore it from install media or a download, or from another system.

    If the detection of the corruption happens relatively quickly, then it may be possible to recover the good file from backup.

    All of these examples make me think there is significant value in checksumming all file in all spaces, not just mirrored ones.  If you could support this and alert administrators when corruption is detected for action, that would be a lot better than just ignoring the problem or not allowing checksumming at all.

    Thank you for your posts – I am a longtime unix admin, but I appreciate the scale and GUI management utilities that windows has always done better in.  If you can deliver this kind of functiionality and performance with robustness, I think you will win quite a few accounts from oracle and other platforms.  

    Mike S

  116. B8Blog says:

    @Firemoon, @c_barth, @ShaneM

    Thank you for the suggestion on multi-mon.  We’ll work on this one.  

    In the meantime, this is something you can experience in the desktop preview and something I demonstrated at the //build/ conference.  If you are only running desktop apps then everything works just as it does today.  You can set things up and operate just as you do today.  As you might have noticed, we improved the taskbar so it stretches across the screen now and can be customized in terms of where running programs are shown.

    @Firemoon, your scenarios around the enterprise are very much in line with what we designed and why we see the advantages of Metro style apps in the enterprise.  

    Keep in mind that this all will take time.  You can transition right away and run just as you always have and gradually add Metro style apps that match your workload.  Or you can hold off and wait until you can make the switch all at once.  My personal view is that this sort of switch is already underway as more and more people in the enterprise are using computing devices that embrace a focused and immersive experience.  As we have said regarding touch, we don’t think the PC is the place to sit out these technology changes and rather we see the Windows PC as embracing and innovating around them—bringing the best of all these scenarios.  But you can go at your own pace and when we say "no compromise" what we mean is exactly that–you don't have to give something up to get something new, or give something up at all.

    Multimon is a scenario we planned for and thought deeply about, both in terms of improvements in the desktop and in how the new start screen and Metro style apps work.  Just to make sure we're all on the same page, the use of multiple monitors is not something we see growing at a fast rate.  Among stationary PCs (no battery) we see 7% of the machines with more than one monitor and of that about 95% have just a second monitor.  In this statistically significant sampling across all types of users about 0.3% had 3 monitors and 0.04% had 4 monitors.  Let's not break off into a debate about the telemetry and if we are "using" it too much — I just want to offer some real world data that shows what we see broadly as it is different than I think most folks experience individually where we (I mean that literally) all run multiple monitors on most of our machines.

  117. It allows 32K characters in file path as opposed to just 256 in NTFS…………Great

    .

    .

    .Congrats!

  118. offtopic says:

    @Firemoon

    You are forgetting that you can't tile more than 2 metro apps (or 1 metro app + Desktop). So it's not like you can tile 3 different metro apps across your 2 monitors, or have 1 monitor for desktop and 2 for metro apps.

    When you press Win all your monitors are populated with Metro start screen.

    @Steven Sinofsky

    Thank you very much for replying even though it is an offtopic.

    I understand that only a minority of users has more than 1 monitor, but most of those who do are in corporate environments, where CEP is disabled by administator's policy.

    I understand that Windows 8 is consumer-oriented OS and I like what it is doing for touch devices. However, it seems that it's hard to justify it for corporate environment.

    As an enthusiast gamer I would probably get it for my Desktop, but not for the reasons of metro style applications. If it provided better multi-monitor support like an ability to tile more than 2 apps on multi-monitor system or have an ability to select only specific monitor(s) to be occupied by Metro interface (with ability to tile applications on/between them) while leaving 1 monitor for Desktop – that would have been a tremendous improvement.

  119. JohannesB says:

    I'm curious what the difference between NTFS and ReFS is as described by the two pictures? More precisely the wavy line in the left picture as opposed to the straight line in the right picture. Has the architecture been cleaned up in some way?

  120. @Sunil Jadhav: As i have understood there is no reason to be hapy about that because there is a limitation build into the Windows Explorer because also NTFS can handle such a long file path.

  121. B8Blog says:

    @offtopic —  Things don't work quite the way you say at the start of your post (start takes over all monitors) and work precisely as you say at the end (dedicate one monitor for metro style apps).  This was demonstrated at //build/.  The multimon demo is 1:45-1:47 into the main keynote from day 1 (channel9.msdn.com/…/KEY-0001).

    As we've said before quite a bit, the opt-in rates of CEIP do not impact the statistical significance across segments/customer types.  We would not cite the data as we do if there was a bias like you describe.

  122. W T says:

    One more comment: Thank you for a factual article without the hype (aka B.S.) typical of most info about Windows 8.  That is MUCH appreciated.

  123. W T says:

    As I started reading this post I was actually excited – finally something that might give Windows 8 some noticeable advantage over other OSes and previous Windows versions.  I got more and more depressed the farther I read.  I'm really surprised by some of the NTFS features not supported by ReFS: compression (seems like it should go hand in hand with checksums/ECC and is still useful even on large file stores), file-level encryption (necessary for many applications in addition to drive encryption), extended attributes (custom metadata without sidecar files), and quotas just for starters.  I think Natalia Portillo's blog post summed it up well — despite some improvements, this mostly seems like a step into the past and very problematic.  Is support of these or other unsupported NTFS features planned for future versions?  If not, to be perfectly blunt, what century are you you guys living in?

    (And yes, NTFS came out full bore with NT, so this incremental approach does NOT follow that pattern.  But given the problems you guys are introducing, it shouldn't!)

    To Steven Sinofsky:

    I've looked into Metro-lating of apps, and I don't ever want to see or use one.  Just try to get a decent list or tree view where you can see more than 4 entries on the screen at a time.  Or, Try changing sort order of a table.  It sucks.  And it's not something anyone with any sense will be "migrating to" over time.  

    The firewall between the Tablet side and the Desktop side makes them almost unusable together — can't share local files, no file explorer on the tablet, no plugins in the tablet browser (not even Silverlight), etc.  The reference above to "Microsoft Bob" is right on — your approach to how to deal with users seems to consider them (us) dumb and treat them (us) that way, and in so doing remove the functionality and capability that people need for good business applications and efficient business use.  And it's business use that should be your target, not the "idiot consumer" space you seem to be going after.

  124. offtopic says:

    @Steven Sinofsky

    Thank you for the answer. I'm sorry I didn't wach the keynote – only reading this blog and comments on other sites. I should have, maybe it would have saved me the embarassment. This is certainly better for most users with multi-monitor setup.

    I join the request for the post about multi-monitor support then. If possible, please provide "user stories" of how you see the switching between Desktop and Metro apps.

    As other posters said, I will reserve my judgement till the release or at least beta version, seeing as my current understanding of how it works is not anywhere near complete.

  125. Daniel Armstrong says:

    I use compression on my SSD's daily, in fact all the computers I manage use compression on their main drives and this not only gets them (and their expensive SSD's) ~30% more space but also improves read speed (which far outnumber writes) and it uses 1-2% cpu on a modern system.

    Please add support for compression (perhaps multiple types?) to ReFS as it helps in so many ways… Hell I would enable it by default if I were you. I haven’t seen a situation where it works badly and with CPU power being so cheap and SSD's costing heaps you would be crazy not to. It would even reduce writes to SSD’s reducing wear on the drive…

    While you are at it you could add compression levels and improve the algorithm, or perhaps use something like bZip or LZMA2 or even a good deflate if you were to aim for speed.

    I love the features of ReFS but cannot all my file servers to it without compression support. And if you are worried about already compressed data being on the rise just allow an exclusion list (and perhaps pre include .zip, .rar, .7zip, .docx, .avi etc…) this would let only things that need compression get compressed and further boost performance of the system.

    Please let me know what you are doing about this, my last comment went un answered.

  126. @StevenSinosky, Why don't you allow more than 2 apps to be present on a metro interface?

  127. Dong Xie says:

    That "File Structures" picture reminds me of Windows Azure. Any plan to merge them, with a clever Network stack?

  128. Jason says:

    So we are being shown a new FS that we won't be able to use on Windows 8 Client? when do we get to use it? if not in Win 8 are there any NTFS improvments in the mean time?

    Can you do a post on kernel improvments please.

  129. Misplaced Error Checking says:

    Error checking and correction needs to placed at the drive level and not the File system level.  This is because drive technology can change making File System error correction pointless and obsolete (a file system that mirrors data on a drive that provides internal deduplication).  The error checking needs to be dependent on the technology of the hardware.

  130. Stefan says:

    Wow ! This sounds great. Finally something that did wake me up about Windows 8. But i think there must be updates to all other Windows systems still used; 2000, XP, Vista and Windows 7. They must be able to read the new filesystem as well. I mention the older versions of Windows because people and corporations do still use them. If there should be a smooth upgrade from the old Windows versions to Windows 8 or later i can think of a scenario where the upgrade goes step by step. If You do as You think i guess many users and corporations will run in to problem with incompability between different Windows versions. That is not very smart. Don't say it is impossible to upgrade the older operatingsystems, because i can read filesystems in Linux, Mac and more in my old Windows 2000 and XP. If they can, then Microsoft can.

  131. Stefan says:

    Should add that it is software installed that makes it possible to read from Linux , Mac and so on. in my old Windows.

  132. oti@live.ru says:

    "In Windows 8, you cannot boot from a space. As an alternative, you can continue to use dynamic volumes for booting." blogs.msdn.com/…/virtualizing-storage-for-scale-resiliency-and-efficiency.aspx

    Can I install Windows 8 on a dynamic disk?

  133. Alvaro says:

    I haven't read any response about why the File System team deem a command line tools more appropriate than a Power Shell cmmdlt.  For those rambling against PS > ; consider that you cant show ALL the advanced features in a GUI (usually u show only the most common ones) nor make automated, repetitive or complex task, plus PS is available in Core Editions 🙂

    I'm with @Daniel Armstrong with the need to support drive compression; especially if that read speed boost it's true ( haven't heard about that before)

    About dynamic volume booting, i haven't found anything on it. In the previous post you promise us a follow up about dynamic volume booting. Would be great to see a how-to similar to those about installing w7 into a vhd.

    >Send-Greetings

    Alvaro

  134. jamome says:

    Excellent post – one of this blog's best!   Please more frequent technical posts like this, and fewer on the smartphone/tablet UI for the PC (Metro=massive barf!!!).

    Also – please consider a 2nd post comparing/contrasting NTFS and ReFS: scalability upper limits, performance deltas on microsoft tests, estimated scenario improvements, etc

  135. @JohannesB

    The diagrams are meant to provide an approximation of the architecture of NTFS and ReFS. For discussion purposes, we thought the diagrams helpful as it illustrates how ReFS uses a new architected on-disk engine while re-using most of the NTFS code responsible for file system semantics. The wavy vs. straight line simply represents that ReFS has a clearer separation between the two layers than NTFS.

    Hope this helps clarify.

  136. MiMiC says:

    No hardlinks, eh?

    Any idea how that will impact WinSxS (As far as I know side by side is built around hard links?

    (Sorry if this is a double post, I cannot see my first attempt in the thread)

  137. Daniel T says:

    New FS?  Looks like NILFS, BtrFS, or ZFS.  This is NOT suggesting most users would use ReFS.  Most users would stick with NTFS, and enterprise users with many hard drives and clusters to manage would find ReFS useful.  The boot loader won't even know how to access ReFS at first.  I do wish Microsoft would cite that this technology has been in use for a number of years in other systems, and all these ideas can be found in other file systems (BtrFS for instance).

  138. @Mike S

    Regarding checksums on non-mirrored Storage Spaces – ReFS supports this. In fact integrity can be enabled even when not running on Storage Spaces. When integrity is enabled and we detect a checksum mismatch, we log an event and, by default, fail the read operation. You can change this behavior for a file if you don’t want the read to fail (the event will still be logged so you’ll know about the corruption).

    @Misplaced Error Checking

    Our architecture allows running ReFS directly on disks (i.e., without Storage Spaces). If you know that your storage does a good job of mirroring, deduplication, etc., then you have that choice. ReFS will still employ techniques like not writing metadata in place, using checksums on metadata (and optionally on user data) for end-to-end robustness to guard against firmware and hardware faults.

    @Oti5 @Alvaro

    Yes you can boot from dynamic volumes. More details to follow.

    @Jason

    We have improved NTFS disk checking and availability quite significantly in Windows 8, along with making it more resilient on SATA drives that are very common on clients. For more details you can watch this BUILD talk starting at 25 minutes channel9.msdn.com/…/SAC-446T

  139. @Steven Sinofsky

    In many ways, it's the "immersive" part of Metro that bothers me and the reason why I don't believe your "no compromises" slogan is really true – desktop computers were not designed to run immersive apps. Smartphones and tablets use immersive apps because that paradigm works well with touch. I once briefly posted something on the forums (now deleted) called "The Truth about Smartphones Applied to Windows 8," basically saying that I don't see why phone-style paradigms could make sense on the desktop and that "immersive" isn't really superior to a standard windowed environment. The true "best of everything" scenario would be to use a desktop PC with a WinRT-style application model powering Aero-style desktop apps (as I have previously said), not an immersive phone-style UI.

    I get annoyed when people react to my criticism with comments like "You don't like Metro, which means you want Windows to be outdated," or "Windows can't have a powerful, next-generation app platform without Metro," or things like that. Those two statements are simply not true, and I fear that even the Windows team itself does not understand this.

    To further explain my ideas, I have created a mock-up of Windows 8, posted here: social.msdn.microsoft.com/…/12345671-f635-4b29-a1d5-1fa2a8ca0e99

  140. Aethec says:

    @Steven Sinofsky

    I agree with @WindowsVista567; a WinRT desktop app model is needed. At the very least, allow Metro apps to be contained in desktop windows.

    IIRC, some guy in the MDL forums showed that it is possible to do it in some old leaked build, but I don't think it's a supported scenario right not.

    From a developer's point of view, WinRT apps are awesome because of WinRT, not because they're full-screen. Bring WinRT to the desktop, add an option for dual-mode apps (install both versions, share the same settings), add live tiles as desktop gadgets, and you've got the most awesome version of Windows ever.

    PS: When will you stop this commenting system PostBack madness?

  141. John says:

    The headline is misleading "Building the next generation file system for Windows: ReFS". Should be "Building the next generation file system for Windows Server: ReFS".

    Does it still get fragmented with times with ReFS?

  142. Rumple says:

    Does ReFS provide support for "secure deletion"?  For example, will one be able to SecureDelete("c:/CitiAcctXYZ.dat") and rely on the filesystem doing whatever is necessary to wipe/eradicate the data associated with that file (file contents, file name, file size, etc)?  I think such a feature "needs" to be built into modern file systems and perhaps also passed down to the drivers that would know about wear leveling, etc.  

  143. Arun says:

    Is there any kind of specification/documentation available yet? If not, are there any plans to release those in near future?

  144. Love ReFS says:

    Is this file system going to be RAM hungry like ZFS?

  145. de@iru.ch says:

    @MSFT: Impressive that you engage in direct consumer discussions. That's great to see. Reminds me of the pre-Vista blog days where every feature was discussed on some Microsoft blog and the people responsible for a feature would discuss and explain their reasoning.

    As Steven Sinofsky is already discussing offtopic stuff: I'm running the dev preview on my main machine since it came out and I'm quite happy with it. I even got accustomed to the new start menu. If I had a single wish it would be this: Give me a "Never combine, hide labels" option for the taskbar icons.

  146. Professor Ramalho says:

    Realmente isto é um marco na tecnologia Microsoft.

    Temos que acompanhar este processo e entender como o mesmo será migrado para as versões do Windows 2008 e Windows 7.

  147. GregH says:

    I don’t believe you still have that big grey crappy scrollbar in the Beta for mouse control – Steven were you not listening! We do actually care about aesthetics and design! It looks worse than a Windows 95 scrollbar!

  148. GregH says:

    Get down to the design team office Steven – and ask them “Do they really believe that the crappy grey scrollbars are good enough?” why are there arrows on it – you just need thin sophisticated metro like transparent bars to appear for mouse activity – not bullish big left and right arrow boxes that literally look like the worse theme from Windows 2000 – trust me! Your design team are doing a terrible job! They don’t have a clue – literally, not a clue, sack them quickly and get some designers who care!

  149. Susan says:

    Looks promising–should improve performance!

  150. Mark says:

    If you can't disable the Metro interface, we will definitely be skipping Windows 8. An interface designed for tiny touch screens just *doesn't* work on a desktop with a large screen and a mouse. You can either give us an option to disable it, or have the company's biggest flop ever on your hands.

  151. Marten says:

    Mark speaks true.

    Metro is a big bag of desktop poo placed on your door mat, set on fire. Then microsoft rings your door bell.

    Windows 8 has a lot going for it. The underlying tech updates are awesome.

    UI is what everyone fronts 24/7 and it is totally screwed for desktop PC users. I run 2 screens one 42in and the other 24 in in portrait mode. Metro is an abomination wiping my current functioning windows to a blocky montage of app links I don't need to see.

    Please die metro.

  152. Metro is not going die  people. i love metro can wait for the beta. If apple were the ones trying to give the world this combo of metro/desktop these guys will be singing their praises. i believe some people are scared  microsoft might pull it off. I am ready for the brand new world. @StevenSinosky

  153. Marcin says:

    No quotas? Is this a joke? Filesystem without quotas is good for desktops or pendrives but not for [storage] servers.

    You're kidding, right?

  154. Windows7 says:

    It's sad but true, Microsoft are fighting a losing battle with this Metro nonsense.

    It's been said that Windows 8 is the biggest leap since Windows 3. I think the basic problem is that Windows is now no longer Windows, and it's just become the world's biggest phone app.

    I want Windows on my PC driven by keyboard and mouse but with Windows 8/Metro 1, this is no longer possible. I don't use a touchy-feely hand-held device nor do I want to.

    The basic problems in Windows 8/Metro 1 for PC users are:

    – No multi-tasking. Having a maximum of 2 apps running is just a joke

    – Now I have to force my way into the Desktop UI, which increases the number of clicks needed and wastes time and user effort. When I finally manage; whenever I click the "Start" button in the bottom-left corner, it flips me back to Metro!!!

    You have added a lot of nice new features for Windows 8, but it's all completely meaningless if the core UI is just so unusable.

  155. Chen says:

    How about displaying folder-size in explorer just like file-size is shown instantly?

  156. Tim Cooker says:

    Our legal team working very hard to find some of our patents in ReFS to sue you guys. Happy New Year.

  157. Multi desktop says:

    Please implement multi desktop support in windows 8

    Because desktop is the most used feature and it usaully gets crowded so multi desktop like ubuntu and mac would be very usful

  158. Sinnfrei says:

    I agree, Metro for desktops is a fail by design. But to be honest, designing UIs never was one of Microsoft's strong sides. Windows 8 could be a real win, if they would simply make the old UI (the standard Windows Desktop) fully customizable.

  159. Alvaro says:

    It's time for a new post, maybe about multitasking, multimonitor or other user related topic (to mix it up a little).

    At least clos comment here. The latest comments are beggining to derail from OT to simply trolling.

       (im still waiting for a post about dynamic disk booting, in the future)

    Alvaro:)

  160. Haven't yet commented on a Building Windows 8 blog entry (although I've read most of them).  Felt a need to comment on this one.  I've been working with Windows and storage in some capacity for almost 15 years and a lot of ReFS seems like major steps backwards.  A treatise in 3 points:

    1. What are we solving for? – Dependability, while important, really hasn't been much of an issue with NTFS since journaling was added.  What customers have been clamoring for is better metadata handling and more robust tools to handle permissions beyond the archaic ACLs.  For all its warts, Sharepoint handles both better than what I just read about ReFS.  Why didn't the MS file system look to the Sharepoint team for inspiration?

    2. Missing and just plain axed features – There's no good reason why deduplication shouldn't be included in a new file system.  None.  It's a game changer.  I have a DataDomain appliance that lets me store 750 TB+ of data on 30 TB raw.  Imagine something similar built into the file system of every Windows machine on the planet.  If you can't do block-level dedupe, at least include single instance storage.  Now to the discarded features.  Quotas?  Compression?  Did you guys really reach out to your customers for this information?  The built-in quota system sucks, but using File Server Resource Manager is really no better.  This kind of thing belongs in the file system, not in userland.  Compression has saved more than a few admin's bacon when they quickly need to regain some space.  Your own updating system uses it on some versions of Windows.  Granted, it's not nearly as powerful as some of the compression apps out there, but at least it's something.  It's a hell of a lot easier to turn on compression for a folder than build an archiving system which runs ZIP/RAR/etc over and over.

    3. The name – As soon as I saw the name it bothered me.  Now I know why.  ReFS.  Sounds an awful lot like a shortened version of ReiserFS.  You know, the guy who killed his wife.  Did you consult with your marketing folks at all on this one?  It makes about as much sense as "Windows Phone 7" (average cell phone buyer asks "What happened with Windows Phone 1-6?")  Call your Xbox guys up.  I'm sure they can come up with a better name in 5 minutes.

  161. Stephen says:

    I'd also like to echo many of the comments from this post in regards to both Metro and ReFS.

    With ReFS, there seems to be so many major features that have been removed that I don't see how it would improve our organisation by using it over either a third-party file system or NTFS itself. If you want a system to replace NTFS, which ReFS appears to be, it needs to support the same features. Could we see some statistics in regards to the use of NTFS features such as Quotas and Hard Links, because I would think most organisations rely on these and won't accept a file system that doesn't support them?

    In regards to Metro, I can't come up with any use case in which user's productivity will be improved on the desktop by using it. I would hope the team that suggested adding it to Windows 8 had to present some serious use cases to Microsoft's management before it was approved. I'd like to see a blog post detailing some of these. Also I don't think you'd find many OH&S people who would say that people sitting at their desks all day touching a vertically positioned is a good thing.

  162. I'd like to thank everyone for their comments and feedback. As we stated in the blog, we believe we've advanced the state of the art for Microsoft file systems in multiple dimensions, even though we currently don't support some of the features of NTFS. We have a lot of headroom for innovation in the design, and we will evolve it in stages. As usual we'll look for feedback on functionality as we do that.

  163. I'd like to see it on the client-side, to be honest. I don't want to download an eval server OS to play with a potential feature. Sure, make us jump through hoops to enable it… but put it on the client OS… please!

    And while you're at it, stop crippling other apps like VSSADMIN and the likes. You make my life hell by having a different server and client application/utility, with the same name, with wildly different functionality.

  164. John says:

    Off-topic

    Please fix the "Folders views" settings.

    I've had my settings to details with customize and sort type (also make it applies to all folders in the settings). When open files with media like (jpg, png, avi etc.. ). It's random change the views to big thumbnail.

    Annoying as hell.

  165. A few belated points and questions.

    "• Maintain a high degree of compatibility with a subset of NTFS features that are widely adopted while deprecating others that provide limited value at the cost of system complexity and footprint."

    How does the ReFS footprint compare to NTFS? Could you compare in terms of a ratio or percent of volume used in each case?

    "• Verify and auto-correct data. … • Optimize for extreme scale. Use scalable structures for everything. Don’t assume that disk-checking algorithms, in particular, can scale to the size of the entire file system. • Provide a full end-to-end resiliency architecture when used in conjunction with the Storage Spaces feature, which was co-designed and built in conjunction with ReFS."

    Considered against the first point, these three points hint that Microsoft engineers are having problems getting NTFS (and chkdsk) to scale to very large volumes with the extreme reliability required. Why go to the enormous effort of building even a semi-new FS, with a substantially reduced feature set, if that were not the case? Particularly so, given that NTFS is Storage Space compatible. Furthermore, if ReFS does not get hardlink capability, but is intended in the future to be used on boot volumes, the Windows servicing stack will have to be rearchitected to cope, presumably a major effort in its own right (or is a reworking of the notoriously slow servicing system occuring anyway?). Note that i don't intend this as a criticism – if NTFS is to be regarded as the more general-purpose FS, and ReFS as the simpler structured, ultra-scalable/reliable FS but with reduced feature set – that's fine by me. 'General-purpose' is synonymous with some sort of tradeoff.

    "Being the first version of a major file system, we do suggest just a bit of caution. We do not characterize ReFS in Windows 8 as a “beta” feature. It will be a production-ready release when Windows 8 comes out of beta, with the caveat that nothing is more important than the reliability of data."

    Very forthright and honest. Much appreciated. However, releasing a new, boot volume incompatible FS with a reduced feature set, when the existing FS is used nowhere near its theoretical limits, is surely begging the question. Is there a problem with NTFS scalability on very large volumes – yes or no? I suppose the question is complicated by hardware issues like 'bit rot', so the question should really be rephrased by assuming 100% reliable hardware.

    "We have tested ReFS using a sophisticated and vast set of tens of thousands of tests that have been developed over two decades for NTFS. These tests simulate and exceed the requirements of the deployments we expect in terms of stress on the system, failures such as power loss, scalability, and performance."

    I would like to see a blog post on MSFT's internal testing. Not regarding anything you might call 'insider information', but an overview of testing procedures with a few juicy stats and a 'factory tour' sort of feel to it, with some good pics, quotes and of course a video. Not asking for much i know, but something that gives the readers' a sense of the scale of Windows development and testing might be beneficial.

    "Initially, our primary test focus will be running ReFS as a file server. We expect customers to benefit from using it as a file server, especially on a mirrored Storage Space. We also plan to work with our storage partners to integrate it with their storage solutions."

    I have noticed a few comments about the perceived 'excessive' number of Windows SKUs. I tend to agree there are too many, and moving from system to system and missing/gaining features/commands/switches can be annoying. Having read the requests to get ReFS in the Win8 client, and also thinking about why the Ultimate SKU was a bit lacking in ultimate features, my (serious) suggestion is that at least one client SKU gets a limited form of Active Directory. I don't see why server capabilities should be limited to special-purpose machines, either in the home or in very small business environments. With features like Away power mode and Wake-on-Lan, i think AD on a client is a workable option. It would also be a great opportunity for young people to get some AD exposure. Suggested Windows 8 SKU set:

    • Premium (64bit only) ≈ Win7 Ultimate + Active Directory (10 users) + ReFS

    • Standard (32/64bit) ≈ Win7 Enterprise

    • Basic (32bit only) ≈ Win7 Basic

    "The Integrity.exe command line tool is a powerful way to manage the integrity and scrubbing policies."

    Why didn't you extend the existing attrib.exe command for this purpose, rather than creating a new command? Presumably a cmd.exe compatible command was developed (rather than a PS cmdlet), so that like the new format /i switch, it can be used in an offline environment like WinPE?

    "…all ReFS metadata is check-summed at the level of a B+ tree page, and the checksum is stored independently from the page itself. This allows us to detect all forms of disk corruption…" "Integrity streams protect file content against all forms of data corruption."

    Are there security implications for all this low-level activity? I'm thinking in particular of rootkits and the ability to detect these. Will ReFS be more resilient to some forms of malware? Also, will ReFS make command switches like Imagex's /check and /integrity redundant? Have you made a FS so reliable that network hardware will become the weakest link in the data transmission chain? Quoting from technet.microsoft.com/…/2007.10.desktopfiles.aspx which states the WIM file format is…

    "…susceptible to corruption. Most often this corruption occurs when a WIM file is transmitted over the network. Early on in Windows Vista, we saw specific problems with specific network cards—if they dropped packets, the WIM file was corrupted…"

    Are network hardware vendors working towards the same levels of reliability that you are with ReFS?

    "For these cases where the volume gets corrupted, ReFS implements “salvage,” a feature that removes the corrupt data from the namespace on a live volume. The intention behind this feature is to ensure that non-repairable corruption does not adversely affect the availability of good data."

    This is an incredible level of reliability sophistication. Just a query on how it works –

    "If, for example, a single file in a directory were to become corrupt and could not be automatically repaired, ReFS will remove that file from the file system namespace while salvaging the rest of the volume."

    Does this work at the sector-by-sector level (if necessary), like the *nix DD Rescue command?

    Regarding shadow paging, does the use of this mechanism (as opposed to the journaling mechanism used by NTFS) more or less preclude the implementation of hardlink functionality in ReFS? Quoting the Wikipedia link: "When the page is ready to become durable, all pages that referred to the original are updated to refer to the new replacement page instead. Because the page is "activated" only when it is ready, it is atomic." Are you also wanting the reference updating to also be atomic, again for reliability and simplicity reasons? (Hopefully that question makes sense given my limited knowledge of filesystems).

    The section 'Reliable and scalable on-disk structures' is fascinating. Hope to see some more info on this, perhaps in Technet Magazine. A question regarding file metadata; how extensible will this be for developers and perhaps even end-users? Here are two things i would like to see supported in the new FS.

    1. The notion of in-context and out-of-context filenames. An in-context filename is the normal filename supplied by the creator/owner when doing say, a Save-As. The out-of-context or floating filename, is a metadata field the user might add to a files' properties dialog. The floating filename can be used, or is used, for example, when emailing the file as an attachment. Ex:

    • InContextPath = %userprofile%DocumentsSpreadsheets<Company>Sales<YYYY><MM>

    • InContextName = summary.xlsx

    • FloatingName = "<Company> Sales Summary for <month>, <YYYY>.xlsx"

    Hope you like my reimagining of short & long filenames!

    2. A user-defined file attribute: U. Something more robust and that doesn't interfere with the A(rchive) attribute. My driver collection is organized like this:

    <path>DriversOSIdentifierClassVendorPackageName

    I want to do this with the attrib command:

    > attrib +U "<path>Drivers**<Vendor>" /S /D

    Then this with Robocopy:

    > robocopy "<path>Drivers" "<anotherPath>" /MIR /IA:U /A-:U

    Is something like that conceivable with ReFS?

    Sorry so late with this comment. Could you please extend the commenting window to 10 days?