Virtualizing storage for scale, resiliency, and efficiency


In this post, we are going to dive into a feature in the Windows 8 Developer Preview. Storage Spaces are going to dramatically improve how you manage large volumes of storage at home (and work). We’ve all tried the gamut of storage solutions—from JBOD arrays, to RAID boxes, or NAS boxes. Many of us have been using Windows Home Server Drive Extender and have been hoping for an approach architected more closely as part of NTFS and integrated with Windows more directly. In building the Windows 8 storage improvements, we set out to do just that and developed Storage Spaces. Of course, the existing solutions you already use will continue to work fine in Windows 8, but we think you will appreciate this new feature and the flexible architecture. As we talk all about consumer electronics next week, thinking about all the media we all have in photos (especially huge digital negatives) and videos, this feature is sure to come in handy. In this post, Rajeev Nagar, a group program manager on our Storage and File System team, details this new feature.

In previous posts we’ve seen folks jump to try to identify edge cases or debug the designs. We’re trying an FAQ approach at the end of this post to see if we can focus the dialog a bit 🙂 The FAQ also talks about the numerous opportunities to use PowerShell as a management tool for Storage Spaces. 

–Steven


By my own admission, I am a digital packrat. My data collection continues to expand and includes some of my most precious memories, including irreplaceable photos and home videos of my children since their birth. For quite some time now, I have sought a dependable, expandable, and easy to use solution that maximizes utilization of my ever-growing collection of USB drives. Further, I want guarantees that my data will always be protected despite the occasional hardware failure.

Windows 8 provides a new capability called Storage Spaces enabling just that. In a nutshell, Storage Spaces allow:

  • Organization of physical disks into storage pools, which can be easily expanded by simply adding disks. These disks can be connected either through USB, SATA (Serial ATA), or SAS (Serial Attached SCSI). A storage pool can be composed of heterogeneous physical disks – different sized physical disks accessible via different storage interconnects.
  • Usage of virtual disks (also known as spaces), which behave just like physical disks for all purposes. However, spaces also have powerful new capabilities associated with them such as thin provisioning (more about that later), as well as resiliency to failures of underlying physical media.

Before we start exploring Storage Spaces in more detail, I will digress briefly to give you a little more context: some of us have used (or are still using), the Windows Home Server Drive Extender technology which was deprecated. Storage Spaces is not intended to be a feature-by-feature replacement for that specialized solution, but it does deliver on many of its core requirements. It is also a fundamental enhancement to the Windows storage platform, which starts with NTFS. Storage Spaces delivers on diverse requirements that can span deployments ranging from a single PC in the home, up to a very large-scale enterprise datacenter.

Pools and spaces

The figure below illustrates the concept of a storage pool. As you can see, we have taken a pair of 2TB (note we use byte measurements as you see in marketing) USB disks and “pooled them” (logically speaking) for subsequent usage.

Two 2-TB discs

From this storage pool, we are free to create one or multiple spaces. Note that once physical disks have been added to a pool, they are no longer directly usable by the rest of Windows – they have been virtualized, that is, dedicated to the pool in their entirety. And although we call this “virtualized,” the storage and reliability provided is very real. The available storage capacity can be utilized though creation of spaces from this pool. In the illustration below, we have carved out one such space from the “My Home Storagepool.Illustraton comparing logical view (10 TB "Documents" space, thinly provisioned, mirrored resiliency) with Physical view ("my home storage pool" 2 2-TB disks)

This virtual disk is usable just like a regular physical disk – you can partition it, format it, and start copying data to it. You will notice, however, that the space has a couple of interesting properties:

  • Its logical capacity is listed as 10TB although the underlying physical disks in the pool have only 4TB of total raw capacity. As a result, you no longer need to worry up-front about the size.
  • Resiliency is built in by associating the mirrored attribute, which means that there are at least two copies of all data contained within the space on at least two different physical disks. Because the space is mirrored, it will continue to work even if one of the physical disks within the pool fails.

The magic that allows us to create a 10TB mirrored space on 4TB of total raw capacity is called thin provisioning. Thin provisioning ensures that actual capacity is reserved for the space only when you decide to use it, for example, when you copy some files to the volume on the space. Previously allocated physical capacity can be reclaimed safely whenever files are deleted, or whenever an application decides that such capacity is no longer needed. This reclaimed capacity is subsequently available for usage by either the same space, or by some other space that is carved out from the same pool. We achieve all of this through architected cooperation between the underlying file-system (NTFS) and Storage Spaces.

With thin provisioning, you can augment physical capacity within the pool on an as-needed basis. As you copy more files and approach the limit of available physical capacity within the pool, Storage Spaces will pop up a notification telling you that you need to add more capacity. You can do so very simply by purchasing additional disks and adding them to your existing pool.

Illustration showing 6 disks of the following capacities: 2 TB, 2 TB, 3 TB, 3 TB, 3 TB, 3 TB

As you see in the illustration above, we have expanded the raw capacity of the “My Home Storage” pool by purchasing and adding four 3TB disks – of course, you could just as well connect SATA and/or SAS storage in conjunction with USB-connected physical disks, and, grow your pool capacity that way. Once we have added this physical capacity, we don’t need to do anything more to consume it. We can simply keep copying files or other data to the space within the pool and this space will automatically grow to utilize all available capacity within the containing pool, subject to its maximum logical size of 10TB. If needed, you can certainly also increase the maximum logical size of a space.

You do not need to explicitly inform Storage Spaces which of your USB disks should be used for each of the spaces you have created. Behind the scenes, Storage Spaces optimally manages the capacity of each of the physical disks within the storage pool, for all the spaces carved out from the pool.

Another core (also optional) capability associated with a space is resiliency to failure of the physical disks comprising the storage pool. For example, the space we’ve illustrated above is a mirrored space (in other words, it has the mirrored resiliency attribute associated with it). This mirrored setting ensures that we always store at least two (and optionally three) complete copies of data on different physical disks within the pool. This way, despite partial or complete disk failure, you’ll never need to worry about loss of data. As a matter of fact, the physical disks comprising the pool are typically not even visible to other components within Windows or to applications running on your PC. By extension, the fact that some physical disks within the pool have failed, is completely shielded from other Windows components or applications. They continue to operate on the space, completely oblivious to the fact that Storage Spaces is working quietly in the background to maintain data availability. Additionally, upon disk failure, Storage Spaces automatically regenerates data copies for all affected spaces as long as sufficient alternate physical disks are available within the pool.

Resiliency through mirroring

It might be interesting to more closely examine how your data is mirrored on different disks. The illustration below shows how a (two-copy) mirrored space is constructed from a two-disk pool:

Mirrored space in a two-disk pool

In this case, Storage Spaces has allocated physical capacity for the mirrored space in what we call “slabs”, which are multiples of 256MB. Also, for this particular example, half of each slab is mirrored on 2 separate disks. Even if one of the two disks fails, Storage Spaces can continue to deliver your data because at least one copy exists on a non-failed physical disk. When multiple disks are available, Storage Spaces spreads slabs across suitable disks as shown in the six-disk pool below:

Mirrored space in a six-disk pool

When a pool disk fails, Storage Spaces identifies the impacted slabs for all spaces utilizing the failed disk, and reallocates them to any available hot-spare disk or to any other suitable disk within the pool (hot-spares are reserved disks within the pool, only to be used as automatic replacements for failed disks). This self-healing is done automatically and transparently so as to minimize the need for manual intervention. We’ve also optimized for speed to prevent data loss from multiple hardware failures at the same time.

Resiliency through parity

There’s another resiliency attribute, called parity, which directs Storage Spaces to store some redundancy information alongside user data contained within the space, thereby enabling automatic data reconstruction in the event of physical disk failure. While conceptually similar to mirroring, parity-based resiliency utilizes capacity more efficiently than mirrored spaces do, but with higher random I/O overhead. Parity spaces are well suited for storing data such as large home videos, which have large capacity requirements, large sequential (predominantly append) write requests, and an infrequent-to-minimal need to update existing content.

Akin to mirrored spaces, slabs for parity spaces are strewn across available disks (with capacity utilized for parity information) as shown below for a parity space contained within a six-disk pool:

Parity space in a six-disk pool

When a disk fails, the parity space recovers equally transparently and automatically as does the mirrored space. For parity spaces, Storage Spaces utilizes the parity information to reconstruct affected slabs for all affected spaces, and then automatically reallocates the slab to utilize any available hot-spare disk or any other suitable disk within the pool (just as it does for mirrored spaces)

The illustration below shows two spaces – one with mirrored resiliency and the other with parity resiliency – carved out from the same pool:

Logical view (10 TB space and 50 TB space pooled) compared with physical view (2 2-TB disks and 4 3-TB disks pooled)

Obviously, both spaces above are thinly provisioned and share the same backing pool (physical disks). Slabs for both spaces are intermingled, and optimally spread over all available physical disks, although each space uses different mechanisms to recover from physical disk failure.

You can access spaces contained within a pool, as long as a simple majority of physical disks comprising the pool are healthy and connected to your PC, a concept called quorum. For example, you will need four of the six disks comprising the My Home Storage pool to be healthy and physically connected to the PC in order to access either the Documents or the Multimedia space. Of course, as previously stated, the resiliency attribute associated with the space determines degree of data availability in the presence of physical disk failure – for example, if the Documents space is three-way mirrored and allowed to use all disks within the pool, you can continue accessing data despite the loss of any two disks.

I’ll explain the virtualization capabilities of Storage Spaces by walking you through a common usage scenario. Imagine that you have just purchased a Windows 8 PC and wish to use this machine as a central repository for much of the digital content in your home or small business. A reasonable setup would involve creation of two resilient spaces – one is a mirrored space for your important documents and the like (these are typically modified more often), and the other is a parity space, for your large multi-media content like home videos and family pictures, which you typically update less often, but view more often. By using the appropriate resiliency scheme, you can optimize for both capacity utilization as well as for best performance.

Logically, your storage configuration would look exactly like the illustration provided above, wherein two spaces with different resiliency attributes are carved out from a single pool. Achieving this is quite simple:

  1. Connect your physical disks to your PC via USB
  2. Create your pool and the two spaces

You can invoke powershell to create the pool and spaces, as well as to complete more advanced tasks. In our example, we have purchased and connected six physical disks to our PC. Below are the simple PowerShell commands to set up our pool and two spaces:

(a) To create a storage pool:

>$pd = Get-PhysicalDisk
>New-StoragePool -PhysicalDisks $pd –StorageSubSystemFriendlyName *Spaces* -FriendlyName “My Home Storage”

(b) To create the two spaces:

>New-VirtualDisk –StoragePoolFriendlyName “My Home Storage” –ResiliencySettingName Mirror –Size 10TB –Provisioningtype Thin –FriendlyName “Documents”
>New-VirtualDisk –StoragePoolFriendlyName “My Home Storage” –ResiliencySettingName Parity –Size 50TB –Provisioningtype Thin –FriendlyName “Multimedia”

Note that the above commands will only work on the forthcoming Windows 8 Beta and subsequent releases. A preliminary version of Storage Spaces is available in the Windows 8 Developer Preview, but the above PowerShell commands will not work in that build. If you are curious to try out Storage Spaces on the Developer Preview build, you can use the below alternative commands:

(a) To create a storage pool:

>$pd = Get-PhysicalDisk
>New-StoragePool -PhysicalDisks $pd –FriendlyName “My Home Storage” –StorageSubSystemFriendlyName *Spaces*

(b) To create the two spaces:

>New-VirtualDisk –StoragePoolFriendlyName “My Home Storage” –StorageAttributesName Mirror –Size 2TB –ProvisioningScheme Sparse –FriendlyName “Documents”
>New-VirtualDisk –StoragePoolFriendlyName “My Home Storage” –StorageAttributesName Parity –Size 1TB –ProvisioningScheme Sparse –FriendlyName “Multimedia”

Also note that, in the Developer Preview, space sizes were limited to < 2TB. That limitation will be removed in the Beta release. Since the availability of the WDP release, we have also activated many additional features within Storage Spaces.

We now get to take a sneak peak at an alternative easy-to-use tool to configure pools and spaces. Beginning with the forthcoming Windows 8 Beta, you can simply go to Control Panel and walk through the sequence below:

(a) To create our pool and a mirrored space, go to Control Panel, click System and Security, and then Storage Spaces.

Click Create a new pool and storage space.

Shows option to "Create a new pool and storage space"

Select the drives you want to add to the new pool.

Select drives to create a storage pool. Image shows two drives selected.

Select your resiliency mechanism and other options.

Select a name, drive letter, layout (two-way mirror selected), and maximum size.

Note that you can assign a drive letter and format the resultant volume as part of creating the space.

Storage Space "Documents" (D:), Two-way mirror, using 2 GB, 10 TB maximum size

(b) To add a couple of disks to an existing pool, select the drives you want to add.

2 drives selected

Storage pool now has 4 physical drives

(c) To create an additional parity space, click Create a storage space, and then select Parity from the layout options.

Selecting parity from the layout options

Parity space consists of 2 storage spaces, and 4 physical drives

(d) In the event you start running out of capacity, expect a notification like this:

"Check Storage Spaces for problems / Open Action Center"

Click the notification to see information about the problem and how to fix it.

Warning on the "Multimedia Parity" storage space: "Low capacity; add drives"

That’s all you need to do to start using Storage Spaces. Once the spaces have been created, you can utilize them just like any other “disk.” For example, you can turn on BitLocker for the spaces you have created, as shown below.

Bitlocker drive encryption page: Windows (C) Bitlocker on, Documents (D:) Bitlocker on, Multimedia (E:) BitLocker on

Options for managing Storage Spaces: Rename pool, Create a storage space, Add drives; Options for individual spaces: View files, Rename, Delete

There is a lot more to say about the many capabilities of Storage Spaces and how other Windows technologies can also leverage these capabilities – we will continue with this discussion in subsequent write-ups.

I hope you find this new capability intriguing and encourage you to play with it. It will all be available to you as part of the Windows 8 Beta release in addition to the features available in the Developer Preview.

– Rajeev

Storage Spaces FAQ

We know that some of you will still have questions about Storage Spaces, so here is an FAQ that we hope will cover most of them. As we get more questions from you in the Comments, we will try to update this FAQ to be more complete.

Q) I use Windows Home Server with Drive Extender. Is there a tool to help me migrate data from the Drive Extender format to Storage Spaces?

No. You will need to create a pool on a Windows 8 PC with a fresh set of disks. Then, you can simply copy data over from your Drive Extender-based volumes to a space within your pool. The functionality delivered through Storage Spaces is more flexible and better integrated with NTFS, so it will generally be more reliable and useful.

Q) Are Storage Spaces some kind of RAID? If it is, what RAID versions do you implement?

Fundamentally, Storage Spaces virtualizes storage in order to be able to deliver a multitude of capabilities in a cost-effective and easy-to-use manner. Storage Spaces delivers resiliency to physical disk (and other similar) failures by maintaining multiple copies of data. To maximize performance, Storage Spaces always stripes data across multiple physical disks. While the RAID concepts of mirroring and striping are used within Storage Spaces, the implementation is optimized for minimized user complexity, maximized flexibility in physical disk utilization and allocation, and fast recovery from physical disk failures. Given these significant differences in objectives and implementation between Storage Spaces and traditional inflexible RAID implementations, the RAID nomenclature is not used by Storage Spaces.

Q) How does the read performance of a space compare to RAID 0 or RAID 10?

For both mirrored and striped spaces, read performance is very competitive with optimized RAID 0 or RAID 10 implementations.

Q) Can I use a RAID enclosure with Storage Spaces for additional reliability and/or performance? Is that a good idea?

We don’t recommend it. Storage Spaces were designed to work with off-the-shelf commodity disks. This feature delivers easy-to-use resiliency to disk failures, and optimizes concurrent usage of all available disks within the pool. Using a RAID enclosure with Storage Spaces adds complexity and a performance penalty that does not provide any improvement in reliability.

Q) Can I boot from a space?

In Windows 8, you cannot boot from a space. As an alternative, you can continue to use dynamic volumes for booting. At release, we will offer guidance on how you can add appropriately partitioned system/boot disks (with dynamic volumes) to a pool.

Q) What is the minimum number of disks I can use to create a pool? What is the maximum?

You can create a pool with only one disk. However, such a pool cannot contain any resilient spaces (i.e. mirrored or parity spaces). It can only contain a simple space which does not provide resiliency to failures. We do test pools comprising multiple hundreds of disks – such as you might see in a datacenter. There is no architectural limit to the number of disks comprising a pool.

Q) How can I know which physical disk a space is on?

Through PowerShell, you can query the set of physical disks backing a particular space. Since all data is striped across all physical disks backing the space, you have this information.

Q) How will I know when a physical disk fails? How do I replace a failed disk?

If the physical disk is contained within an enclosure that supports the SCSI Enclosure Services protocol, we will activate a red LED (if present) next to the failed physical disk. A standard notification will pop up in the desktop. You can also see information about the failure in the Storage Spaces applet in Control Panel. Here is what that looks like:

Check Storage Spaces for problems. We found one or more possible problems with your storage. To address these problems, check your Storage Spaces. Button: Open Storage Spaces

On a parity disk: "Degraded: check drive health"; On a two-way mirror disk: "Degraded: check drive health"; On one of 4 physical disks: "Disconnected: reconnect drive"

Through PowerShell, you can also query disk health to determine if a disk has failed.

Once you’ve detected the failed disk, you can physically disconnect it at any time. Replacing a failed physical disk is easy – after removing the failed disk you simply connect the replacement disk to the PC, and then add the disk to the pool either via PowerShell or via Control Panel .

Q) How do I replace a working drive with a bigger one (or just cycle drives)? Does it require a “rebuild”?

As long as you have created mirrored or parity spaces, you can always simply remove a physical disk within the pool, and add a different (perhaps larger) one. Within a short period of time, the impacted spaces will automatically be resynchronized (the Storage Spaces design optimizes this operation to be faster than traditional RAID rebuilds). You can determine whether all spaces are healthy – i.e. data has been resynchronized so as to maintain the designated number of copies –either via Control Panel or via PowerShell commands.

Q) Can I trigger resynchronization myself?

Yes. If you don’t want to wait for automatic resynchronization to start, you can choose the Repair command via PowerShell, which will initiate resynchronization so long as suitable replacement disks and/or spare capacity is available.

Q) What kind of disks will Storage Spaces work with? Are there any special requirements? What about custom enclosures housing these disks?

You can use Storage Spaces with any physical disk that otherwise works with Windows, connected via USB, SATA, or SAS. If the physical disks are connected via some custom enclosure (e.g. in JBOD configurations), Storage Spaces will utilize the SES protocol (if supported by the enclosure), to identify physical slots where the disks are located. When needed, Storage Spaces will also use SES to light up failure LEDs associated with physical disks (assuming that the enclosure has such LEDs). For Storage Spaces to use enclosure capabilities, the enclosure must conform to the Windows logo requirements. Enclosure vendors have been made aware of these requirements and we expect increasing conformance over time.

If your disks are housed within an enclosure, and if Storage Spaces either does not provide you with slot information associated with the physical disks or does not light up LEDs on the enclosure, you can assume that the enclosure does not conform to Windows logo requirements.

Q) Is there a defrag or CHKDSK equivalent for pools?

No. Storage Spaces optimally utilizes all physical disks. In the event that Storage Spaces metadata on a physical disk becomes corrupt (which will be obvious since the disk health will indicate a problem with the physical disk), you can treat the disk just as you would any other failed disk – simply remove it from the pool. If the physical disk is healthy, you can subsequently re-add it to the pool.

Q) How do I know how many mirrors a given file has?

If your file resides within a NTFS volume on a two-way mirrored space, two copies of all your file data will be maintained. If you configure a three-way mirrored space, there will be three copies.

Q) Can I pick which drive to use for mirrors? For example, if I know a particular disk is faster/better/newer?

Yes. In typical deployments, Storage Spaces will automatically select physical disks from the pool to back your spaces. However, if you so desire, you can manually specify a specific set of physical disks within your pool to back a particular space and thereby control allocation. You can do this via PowerShell options at the time you create the space.

Q) Can I change the maximum size of a space? Are there advantages or disadvantages to just making every space 50TB?

You can increase the logical size of a space at any time via Control Panel or PowerShell. Decreasing the logical size is not supported (or needed), given thin provisioning. It makes no difference whether you specify the initial logical size to be a smaller number (say 1TB) and grow it as needed, or set it to a very large number (say 50TB) right from the beginning. The latter may save you time and effort later.

Q) Can I change the slab size to something other than a multiple of 256MB?

No. The slab size is automatically determined by Storage Spaces based on a multitude of factors to deliver an optimal experience in terms of performance and availability.

Q) Does the pre-defined slab size result in sub-optimal utilization of capacity? For example, what if most of my files are very small? What if they’re all large video files?

The slab size is an internal unit of capacity that we use for provisioning across multiple spaces within the same pool. Its value has no bearing on optimal storage of files, regardless of file size.

Q) Can I move a storage pool from one PC to another, once created? For example, if I have a cage with 6 removable drives?

Yes. Just connect the physical disks comprising the pool to the new PC.

Q) Say I have 3 external enclosures and I remove them one at a time. I then plug them into another Windows 8 PC in reverse order.  Will the new PC think I have a broken pool or will it eventually catch up? What if I never plug in one of the enclosures?

You can plug enclosures back in in any order. When Storage Spaces detects a sufficient number of disks for quorum, it activates the pool and contained spaces. You can plug in more enclosures later. If the data on any disks becomes out of sync, Storage Spaces will automatically sync them. Even if you never plug in some enclosures, as long as Storage Spaces detects the minimum number of disks needed, you can continue working with your data. Both via PowerShell and via Control Panel, Storage Spaces informs you that a few physical disks are missing, thereby encouraging you to plug them back in.

Q) You mentioned that quorum for the pool requires a simple majority of healthy and connected physical disks. Does that mean I always need to have an even number of physical disks in the pool? Or do I need an odd number of physical disks? What about two-disk pools?

There is no requirement for an even or odd number of physical disks. Storage Spaces correctly handles two-disk pools and continues delivering resiliency to failures for a two-way mirrored space contained within such a pool, even if one physical disk fails or is disconnected.

Q) What happens when I plug physical disks comprising a pool into a Windows 7 machine?

Windows 7 does not support Storage Spaces and will treat the physical disks just as it would any disk with an unfamiliar partitioning scheme.

Comments (181)

  1. After launch of Windows developer preview MS has updated the builds so many times. dev. pre has a build no 8102 and MS has updated it to 8149 so why don't MS gives the update to testers via windows update or web download so they can see the new improvements in the new builds and also give a blog or webpage platform to share the new ideas by testers and windows users

  2. sgy says:

    newest build is 8176 and is from 3rd january 2012  

  3. JJCrandall says:

    Is this also going to be a feature of Server 8?   What is the parity write performance like?  Is is similar to existing soft-raid5 in windows, or closer to LVM in linux?

  4. Alvaro says:

    So far im loving this!!!… I was just thinking about makinga RAID 0 or RAID 10 at home (The windows Experience Index said mi Disk I/O its my botleneck).

    Now ill wait a bit, and see if this new feature can be a better option 🙂

  5. Great feature! Do you plan to include this feature in all SKU's of Windows 8 or is it only for professional use? And the idea of a FAQ is perfect!

  6. konstantin says:

    Looking forward to it! 2012 will be an exciting year – ultrabooks with Windows 8 on them!

  7. Paulos says:

    There's one thing I actually don't understand about setting the logical size pool to (example) 20TB when I only have 5TB of physical storage. I hope it doesn't mean that I will see 10TB of free space when only 5TB is available, so am I right that I will only see the amount of space that's physically present?

  8. deiruch says:

    Storage Spaces is the *SINGLE BEST* feature of Windows Server 8. Can't wait to use it. Replication over LAN is the last piece I'm missing in all of this. That way I could use Storage Spaces as HA storage and could throw away all the redundant ond overpriced storage solutions.

  9. nedim says:

    maybe it's off topic, but will be possible to use "metro" on desktop as "widget"?

  10. Neticis says:

    How long disk full path with disk drives in Windows will start with letter. Today it has not any meaningful relation to the real disk storage. Isn't time to wipe out this stupidity?

  11. Euphoric says:

    Sounds awesome for home server.

    Will this feature be present in any other Windows version?

  12. jps says:

    @Neticis I have no problem with using letters. In fact, I can select a letter as a mnemonic like S: for source. Better than using UNC paths to go places IMO.

  13. Cant wait to see this in action!!

    Good Job!! Guys!!

  14. TheRackow says:

    This looks awesome! One question though…

    So if I have an existing PC that I want to upgrade to Win8 and use the storage pool, how do I create the storage pool without losing all of my data? Do I have to back it all up somewhere first?

    Thanks

  15. Killswitch says:

    @Euphoric it would make sense to have this feature in a server version..  

    @nedim dont think so, i think i read it on <a href="http://www.winbeta.org">winbeta</a&gt; that it wasnt possible..

  16. Noel says:

    Is windows installed in one of these spaces? and if not, can the un-used space on the drive with windows be allocated to a pool? then what happens as windows starts to fill up that drive and the available space for the pool gets reduced? Also what about appdata, alot of applications including Office (think Outlook) bury appdata into instalation folders and hidden folders, will it be easy to use spaces for this data?

  17. Alxandr says:

    A feature I would also like, that I have to utilize an ugly hack to use in win7 is the ability to move a user-foler, or the entire users-folder to another physical disk (or in this case space). My computer consists of a small SSD only large enough for windows and a few applications that I need to run fast (like visual studio and browsers), but I want the rest of the data on my 2TB hdd. Today I need to manually move the user-folder using cmd from the Administrator account (which first needs to be activated) using xcopy or robocopy (don't remember). Then I have to create a junction from C:UsersAlxandr to D:UsersAlxandr. This should be automatic and a posibility in windows 8.

  18. Euphoric says:

    Is this server-only feature or will it be present on desktop version too?

  19. domenicoav says:

    Good Job

  20. Great post, and loving the FAQ idea.

    This is a really great feature that would be useful for just about anyone. Has there been any thought given to allow OEM home server solutions to provide a plug-and-play pool functionality (bypassing the need to manually set up the pool) for the less tech-savvy consumers, maybe also hooking up to the user's libraries?

    For example, this is something my parents would love to safely keep both family photos and important business data, but setting up a pool and the concept of "drives" is far beyond them – they just want to plug a system in that will ensure their data is safe.

  21. Joao M Correia says:

    This is the absolute best feature discussed so far. Being a user of open-source alternatives that do the same parity protection (flexraid, disparity), having this incorporated into the OS is a great move.

    The proverbial cherry on top would be to have access to data deduplication on the client as well, on top of this. Will it be possible?

  22. Indeed, this is an excellent feature! Like that storage space becomes something abstract, somehow disconnected to physical hard drives. I already imagine scenarios, like: "Dude, can you borrow me one or two hard drives for a few days? My pool is a bit tight right now…"

    @arrow22

    If I understood the blog post above correctly, thats possible. The OEM just needs to set up a pool (and virtual storages spaces if desired) and sell the hard drives as one package. As soon as the buyers plug it into their Windows 8 machine, they will have access to the pre-formatted storage spaces…

    At least I think it works like that…

  23. mobeen says:

    Driver Extender Lives!

    The only reason i never got WHS 2011

    CAn't wait for Windows 8 🙂

  24. Is this available in the client or the server version of 8?  I assume both.

  25. Jote says:

    Wow, this is really cool. I hope it won't be just for the Server version…

  26. Sounds good with regard to an integrated business solution. Perhaps a bit out of context in this blog, but I hope I can leave a message here hoping that one big item I wish to be further improved for the windows 8 platform: inking, handwriting and ink-to-text recognition. It is an underapreciated function since its improvement in the windows 7 OS. What is needed is improved handwriting recogniton. We have come so far now with technologies like Tell Me and Siri for voice, why not handwriting as well. Missing this function a lot, especially with all the critique I get with my sloppy and irregible doctor's handwriting.  Secondly the handwriting the multilangual handwriting recognition library needs to be vastly improved and thirdly a more handwriting friendly ecosystem needs to be developed, ranging from a standardised file format for saving and sharing to a handwriting friendly end user screen experience. I'm talking about a real life A4 or note sized window on a tablet screen.

    Thanks for all the effort so far. Windows 8 looks to become great!

  27. Windows7FAQ says:

    Please be more specific about this – i.e. what will Windows do? "What it usually does, wink wink" is the sort of answer I expect from my grandfather when I ask about sex, not when I ask a reasonable question about a mechanical operation. Oh, right.

    Q) What happens when I plug physical disks comprising a pool into a Windows 7 machine?

    Windows 7 does not support Storage Spaces and will treat the physical disks just as it would any disk with an unfamiliar partitioning scheme.

  28. Gavin Greenwalt says:

    Will there be a way in Windows 8 to more intelligently use different speed drives?   e.g. Hybrid between SSDs and Spinning disks?   Could I have a SSD that is mirrored to a spinning disk but transparently behind the scenes?

  29. winnerd says:

    @Alxandr

    In Windows 7 you don't have to use cmd or anything to crazy complicated to accomplish what you want when it comes to moving a user folder. What i do is go to your user folder "c:usersname" and what ever you want to move (for example My Documents) right click > properties > Location > Move and tell it where to move it to… its not super easy as a one click option "move user folder" as I would like it to be but this solution is the best way of moving your personal data to another drive. This also maps it so that any future programs that are installed and all default save points are moved to this new location… You will still have your "user" folder with your name on the C drive but al the data stored in there (documents, pictures, movies, music etc) are moved to the new location provided. 🙂 Hope this helps for now and hopefully something easier will come with Win 8 but i havn't seen anything yet.

  30. Lance says:

    @Alxandr

    I had sent an email to Steven before this blog had any posts and it isn't planned and they recommend against it. I'd like this feature as well, but it would be a fundamental change and they didn't plan to put it in windows 8.  That said it would be nice.

    @winnerd

    I store my actual data on another drive, but windows still keeps a lot of junk in the user directory.

  31. Amazing technology. I love Storage Pool provided by my Windows Home Server and its absence was reason not to upgrade to WHS 2011. With Windows 8 all my storage problems may be solved in pretty nice and reliable way.

    Add to Windows 8 new backup features like Windows Home Server has and there is no need for it anymore.

    Is there any information about new version of WHS built on top of Windows 8?

    Great job.

  32. Dennis Martijn says:

    Since moving a pool to another device requires the movement of the physical disks to that device, what happens to other pools crafted out of these disks?

    Say, I have 4 physical disks, 1, 2, 3, 4:

    Pool 1: 1, 3, 4.

    Pool 2: 2, 3, 4.

    I want to move data from pool 1 to a different device.

    I disconnect 1, 3, 4 and attack it to the different device.

    Pool 2 has 3 and 4 missing, only 2 left for use.

    Now what? Doesn't this corrupt the 'partition'? What about the files that are stored physically on 3 and 4?

    Or, do these pools require physical disk alignment so that a combination of physical disks once in a pool can only have these same physical disks used for another pool? 1, 3, 4 for both pools, for example?

  33. Stefan says:

    @Dennis

    It wouldn't corrupt Pool 2, it would simply be unusable on the first machine until you added either drive 3, or 4 back. Pool 2 would however be usable on the second machine since you have access to 2 of the 3 drives that make up pool 2.

  34. Stefan says:

    Now the questions.

    Mirrored Resiliency results in a storage reduction of either 50% or 67%, what is the reduction for Parity Resiliency ?

    The FAQ says this feature doesn't play nice with RAID, how do you recommend using PCIe SATA expansion cards? JBOD? or something else.

    This looks like a killer feature, great job to all involved.

  35. Mike says:

    I'm also interested in how SSDs will be handled by this.

    Since you can't boot from a storage space, many people will just have Windows installed on a SSD, but then you can't add that to the pool, which means if you want resiliency for your docs, you need to keep them on the (slow, spinning disk) drives in the pool, and if you want speed, you keep them on the SSD with Windows, like people normally do.

    Surely there's a better, simpler way?

  36. "Mirrored Resiliency results in a storage reduction of either 50% or 67%, what is the reduction for Parity Resiliency ?"

    Yeah, I'd definitely like to know the answer to that as well.

    Also, from the post, it sounds like the Storage Pools are independent of the Windows install. So if the system drive fails, our storage pools remain in tact? We can just replace the system drive, setup windows and continue as if nothing happened?

    Also, can you change a pool type? Say you make a simple one, can you make it mirrored ot parity later? Or can you make change a mirrored one to a parity one?

  37. If the logical pool size does not matter, why bother with one at all?

    It will be adding complexity when the pool grows, and there will probably be no possible way to match exactly the logical and physical sizes when you add discs, so you're forever doomed to set the logical to oversize. And you risk wasting precious disk space if undersized.

    I personally would prefer not having to deal with logical sizes at all, at least as an user option, but rather have Storage Spaces warn me when free space approaches the size of the largest disk, say 3TB + 10% = 3.3TB in your example, so that even if a large disk fails, it will still automatically mirror to the remaining drives.

    You mentioned hotspares in the article, but nowhere in the screenshots does it show how you can add a disk to a pool but keep it unused as hotspare, or at least it's not obvious if it's a matter of unselecting some checkbox. I'm not even sure if the concept of hotspare is useful in this Storage Spaces approach, as long as total free space is larger than the largest disk in the pool, or if it's even desirable, as it's probably better to prevent bitrot and mechanical/electronic defects to cycle all the existing disks in the pool instead of dedicating disks to storing nothing for months until needed. That's typically when you need it that you find out it does not work anymore.

    If there are dedicated hotspares, can they be unallocated but available, so they can be allocated automatically to the first pool that needs it? Otherwise users would be forced to maintain as many hotspares as pools.

    Just a matter of preferences, but the representation of the pools and disks in Storage Spaces and Windows Explorer looks very bland. I know, evolution, not revolution, but I would like to hide the pool's drive letter in Windows Explorer and see the pools only listed by their names, such as "Movies", "Television series", "Music", etc. (the drive letter could still be visible in the properties), and in Storage Spaces, a more graphic representation of the physical drives that form the logical pool would be welcome. Not only it would look very cool, but it would also allow easier maintenance if the drives are sorted by port, protocol, internal/external type, USB hub, front or rear USB/eSATA, enclosure, etc. There does not seem to be many properties to easily identify disks, even for the systems that support SES, or to visually see the health and size information of the pools and drives. It's all too textual and complex for most end users.

    The example in this article of a messy cluster of different USB drives really does not seem nice, it would be great if Microsoft could release specs for enclosures that work specifically with Storage Spaces, so people could purchase compatible computer cases and empty enclosures. I am not sure for instance that SES works for internal drives or through USB, so an alternate system to light up the internal or USB drive fail LED would be great for USB enclosures or internal drives that are plugged into externally accessible trays.

  38. AndrewDover says:

    How about integrating this with Skydrive ?

    It would preserve the data in case of flood or fire since the data would end up on the Skydrive,

    and fast read storage for local access.  

    People would probably pay something to store large amounts of data.

  39. Salty says:

    Looks like my previous comment was lost – the main point was. using the parity resiliency option, this solution seems to provide a similar resiliency level to RAID 5 (if i interpreted the information correctly). Is there a "partial recovery" story for Storage spaces? If as user was to lose two drives simultaneously for example, it seems that with the concept of Slabs, a user's ability to recover the data from the remaining drives is bordering on "impossible" or at the very least, costly.

    Is there a configuration option which does not involve slabs and provides drive pooling but stores data onto a standard NTFS volume? Using this sort of approach, I can separate data from parity using solutions like FlexRAID. In the event of multiple drive failure at least i have the data of the remaining healthy drives.

    Thoughts anyone?

  40. Josh says:

    I can't see any advantages of this over Intel chipset hardware and its manager software that has supported migration and upsizing RAID arrays for years.

    But there are many disadvantages – bad performance and latency compared to hardware level, and cannot make it bootable partition to start with.

    The only user value I can see is laptop only users, that have USB drives and won't invest in a NAS. Then rather than currently trowing out old small drives and replaced it with a big single new one, they can have a bunch of small old drives being utilized for redundancy.

  41. Is the number of parity units limited to 1, or can it support more than that (like RAID6 or FlexRAID's Tx engine)?  I like the idea of surviving multiple device failures.

  42. TheCyberKnight says:

    Very nice. Good job.

    This single feature is the final argument in the Drive Extender debate. I am now glad Microsoft killed it.

    BTW, I assume that "Previous Versions" works correctly with Storage Spaces. Isn't it?

  43. Chris says:

    Any plans to implement some sort of checksumming?  I saw in another post that there is an offline deduplication feature available – would the checksumming in this be usable for data integrity verification?

  44. commongenius says:

    "Decreasing the logical size is not supported (or needed), given thin provisioning."

    Decreasing goes hand in hand with increasing; if decreasing is not needed, then increasing is not needed either. If neither are needed, then why bother having a logical size at all? If increasing actually does serve a purpose, then that means having a limit serves a purpose, which means you have to provide a way to decrease the limit also.

  45. Gigaplex says:

    Having a desktop notification pop up on something that effectively acts as a server is not really going to help much unless someone frequently logs in to manually check the system health. Is there a simple way to get the system to send an email alert when a space has a problem (ie low capacity or hardware failure)?

  46. HandNF says:

    It looks like a really neat feature for Windows 8! I just hope that it is built to be for desktop as well as servers. (A few articles said it was server only, which would be really disappointing.)

    Like a few others have stated, a cloud solution would work great for backing up really important files. In doing so, you could essentially reduce the logical size to infinity.

    I do have a question with regards to backing up data. I use BackBlaze, which goes through my drives and backs up data in the background. If it was backing up my pools, would it pick up Drive D, where all my files are organized and solitary, or is it going to back up the copies of files too.

    What I'm asking is if pools will be the same architecture as normal drives for programs.

  47. ZipZapRap says:

    One question – if I have an internal 2TB drive, and an external 2TB drive as my pool, if I decided to take the external drive away (say for storage), what happens to the data on the internal drive? Is it accessible.

    Here's why I'm asking. All of my holiday pictures, photography stuff and videos of my daughter are stored on a 2TB internal drive, which I use SyncToy to back up to two seperate external 2TB drives. One gets stored at my parents house, one gets stored with me.

    If I create a "pool" of these three drives, can I still give the second drive to my parents for safe-keeping, keep the other unplugged, and have access to the data on the internal drive?

    Thanks!

  48. Format? says:

    If i have 4 disks, and all 4 disks are independent disk (not pool) and contains full data in it. Can I create a pool out of this 4 disks while keeping the data?

  49. GS says:

    This is the reason I didn't move from WHS v1 to v2. Easily half the users of wHS on various forums were stunned at leaving drive extender out of WHS v2. I will hold onto v1 and change my servers to w8 as soon as it hits.

  50. Thank you for the interest, kudos, and feedback. Some responses are below:

    Storage Spaces is available on both Windows 8 client as well as Windows 8 server.

    @Paulos – A pool does not have a logical size associated with it – each space does. As demonstrated in the screenshots, the current pool capacity is always displayed as is the amount of used physical capacity so you always know the amount of free physical capacity available for all spaces contained within the pool.

    @TheRackow – When you upgrade your PC, Windows 8 will continue working normally (as expected) with your existing disks. If, however, you want to create a pool from your existing disks, you much back up your data elsewhere prior to adding these existing disks to a pool.

    @arrow22 and @plot-paris – OEMs/partners can certainly choose to offer preconfigured multi-disk packages. Storage Spaces will immediately allow access to spaces within a preconfigured pool as soon as a quorum of physical disks are connected to the PC.

    @Windows7FAQ – Since Windows 7 does not support Storage Spaces, you will be unable to access the data contained on the underlying physical disks from the Windows 7 PC.

    @Gavin Greenwalt – Pools can be comprised of a combination of SSDs and HDDs. Further, using PowerShell, you can certainly configure a two-way mirrored space such that data is replicated across a combination of SSDs and HDDs. That said, Storage Spaces does not attempt to “automatically migrate” data between SSDs and HDDs to (as an example) deliver better performance for frequently accessed data.

    @Dennis Martijn – Each physical disk can only belong to a single/unique pool. Your example is not valid since physical disks 3 and 4 cannot simultaneously belong to both pool 1 and pool 2.

    More responses shortly. Thanks again.

  51. Derix says:

    Q) How does the write performance of a space compare to RAID 0 or RAID 10?

    Can it be use for intensive write applications such as video recorder software?

  52. sreesiv says:

    This is a super feature.

    >> As we get more questions from you in the Comments, we will try to update this FAQ to be more complete.

    Please update the FAQ as a separate post by clubbing new questions and anwers. Not a good idea to update an already posted blog. Readers won't notice the changes or new additions.

  53. danielgr says:

    First, this looks to be an amazing feature I'm pretty much looking forward to (and I'm also glad there seems to be much more into Win8 than Metro …)

    Then few further questions (my apologies if they were already answered but I missed them in your very complete post):

    1) Am I right in assuming "defragmentation is no longer needed, nor an option, with Storage Spaces"?

    2) How do the "storage spaces" show in Explorer ? Is the real free space clearly visible? or only the logical one? If the later is true I think it'll be annoying for many people (not knowing how much disk they have left). Guess one could get used to the convenience of "not needing to know", but with so many years of inertia is going to be tough.

    3) Let's say I have a mirrored space with 2 drives:

    a) Can I take one of them to a different Win8 computer and still get access to all the data (without needing to move both drives?)

    b) Can I then rebuild the mirrored space on the other computer by adding a new drive?

    4) Same as 3) if I start from a "parity" storage space.

    5) Joining others question: will this be available on all editions of Win8? Or limited to Pro/Ultimate as in Win7 mirrored volumes?

    Thanks in advance… can't wait to get my hands on the Beta.

  54. Some additional responses follow:

    @Stefan – for parity spaces, the capacity used for storing parity information is 1/n where “n” is the number of columns configured for the parity space. “n” can range from 3 – 8. Therefore, with a minimum of 8 drives within the pool, the minimum amount of capacity reserved for maintaining parity information is approximately 12.5% of available physical capacity backing the parity space. As noted, parity spaces are suitable for data that is very sporadically updated.

    @Stefan – you can use any non-RAID disks that otherwise work with Windows.

    @Mike – you can create a pool with SSDs, HDDs, or any combination thereof.

    @Existentialism0 – yes, physical disks comprising the pool are unaffected by whatever happens to/on non-pool disks. For your example, you can replace the system drive, setup Windows, and continue. No – you cannot change the resiliency attribute associated with a space. Notice that pools do not have resiliency attributes – spaces do.

    @Chimel – pools do not have logical sizes. Each space has a logical size associated with it – this is the size of the “disk” that the file system (e.g. NTFS) works with. Storage Spaces will indeed notify you in low capacity conditions. You can designate physical disks as hot-spares via PowerShell. These hot-spares are on a per-pool basis. When a given physical disk fails, Storage Spaces can indeed utilize remaining capacity on existing pool disks even in the absence of hot-spares, to ensure redundancy of data for affected spaces. However, note that this requires sufficient capacity remaining on existing pool drives as well as at least one pool drive that was not being used to back the set of spaces that were impacted by the physical drive failure – you can even add additional physical drives to satisfy the latter constraint. Our partners already have access to documents describing expectations off enclosures. We will certainly make them publicly available. Appreciate the feedback.

    @Salty – two-way mirrored spaces will survive failure of any 2 disks backing the space. Parity spaces will only survive failure of a single physical disk backing the parity space – note, however, that concurrent failures of other disks within the pool not backing the parity space do not impact access to the parity space.

    @Hexland – with Windows 8, parity spaces tolerate failure of any single physical disk backing the specific parity space

    @TheCyberKnight – Yes, Previous Versions will work fine on volumes contained within any space.

    @Chris – Storage Spaces do not check-sum data. However, applications/consumers of spaces can maintain their own check-sums and then utilize the redundancy possible through Storage Spaces to deal with failures within any one copy of the data.

    @Gigaplex – yes, you can easily have an email notification sent to you. We will provide guidance on this.

    @HandNF – your backup application would only see the single set of your data. Spaces are regular drives to the rest of Windows and to Windows applications.

    Even more responses shortly. Please keep the feedback coming. Thank you for all the interest.

  55. Stefan says:

    @Rajeev Nagar Thanks for the responses. I think the board ate my first reply.

    My second question wasn't about the disks, but about PCIe controller cards, and how they might work with Storage Spaces.

  56. Moses says:

    Rajeev, quoting the below:

    'for parity spaces, the capacity used for storing parity information is 1/n where “n” is the number of columns configured for the parity space. “n” can range from 3 – 8. Therefore, with a minimum of 8 drives within the pool, the minimum amount of capacity reserved for maintaining parity information is approximately 12.5% of available physical capacity backing the parity space. As noted, parity spaces are suitable for data that is very sporadically updated.'

    Does it mean that for parity mode,  there's a limit of only up to 8 drives that can be used for a single storage pool? Is there also an 8 drive limit for mirror mode in this case?

  57. Mike M says:

    @Rajeev, this is great that you are answering these questions!  As a longtime linux user, I have to say I am very impressed by what you guys seem to be doing here.  Some questions:

    1) What kind of performance have you seen in an array of 8-10 modern SATA disks in a parity space?  I know you say it's comparable to RAID 0, but for large files, what kind of real world sequential transfer rates have you seen?

    2) Modern H/W RAID controllers come with battery backup units to assure that in a power fail, buffers that aren't yet written to the disk will be saved so when power is restored, datya will not have been lost.  How do you deal with this kind of resiliency requirement?  Can I use an SSD or non-volatile memory to assure filesystem consistency in a power outage?  What performance hit will be taken?

    3) ZFS has internal checksumming in the filesystem to deal with memory or odd I/O errors – is there something like that in storage spaces?

    4) My ZFS system has a SSD as a write buffer and a cache to improve performance, and I find it helps in a lot of circumstances – do you do anything like this to boost performance using SSDs?  Alternatively, if I had a lot of RAM, is there a way that storage spaces can use it to do large buffering to optimize transfers to the disks?

    5) What kind of tools will be available to optimize  performance and general tuning?  The description in the blog makes everything seem automatic, but in the circumstance where I may really care about sequential or random access performance, how I can tell where the bottlenecks are so I can address them?  for example, that some older disks are the bottleneck in a parity space or that I adding more disks would improve parallelization of I/O's?

    6) Is storage spaces designed to run well in a virtual machine?  Do you require physical disk passthru like on ESXi to the windows 8 VM or are virtual disks OK too?

    7) What kind of CPU horsepower will be necessary to handle a lot of physical disks in parity spaces and keep performance high?  Linux is pretty efficient here, but some idea of sizing for windows 8 configurations would be helpful.

    I find my ZFS servers under ESX to work very well, but Solaris has a lot of sharp edges in how you configure it and manage it.  Even if storage spaces isn't quite at feature parity, I'd cut over to it if I knew it was reliable and was easy to configure and manage and had great hardware support.  You guys can also get the enclosure vendors to make management a lot easier, and that alone would be a huge win if the fundamentals are very capable.  

    Are all the features going to be available when Windows 8 is released?

    Thanks!  This is the first Microsoft product in quite a while I have been excited to get my hands on!

  58. Lionel says:

    Storage Spaces look quite appealing.  Here are some more questions:

    * "However, applications/consumers of spaces can maintain their own check-sums and then utilize the redundancy possible through Storage Spaces to deal with failures within any one copy of the data." — Could you elaborate?  Will Windows include such a consumer, in order to provide (ZFS-like) protection from data corruption (e.g. if cosmic rays flip a bit in one of the slabs)?  (If not, you definitively should add this feature.)

    * Can I change the way a disk is connected?  (e.g. remove an internal SATA disk that is part of a Storage Space, put it into an USB enclosure, and plug in the enclosure; will the disk still be recognized as part of the Storage Space?)

    * Is it possible to use network drives (SMB, WebDAV, NFS, whatever) as part of a Storage Space?

    * Is it possible to use iSCSI drives in a Storage Space pool?

    * Does the Recycle.bin work on Storage Spaces?

    * Is it possible to create hardlinks/symlinks/alternate streams/EFS files/very long paths/other NTFS features, in a Storage Space?

    * Is it possible to store network drives (SMB/WebDAV/NFS/…) on a Storage Space?

    * Is it possible to encrypt a Storage Space?

    * Is it possible to mount VHDs from Storage Spaces?

    * Is it possible to use VHDs as part of a Storage Space pool?

    * Since there is no CHKDSK, and the data is broken into slabs, how should filesystem corruption (e.g. because cosmic rays flipped a bit in the wrong place) be handled, if the filesystem is no longer readable?

    * Will you document the underlying storage structures, so that Storage Spaces can be read from third party tools?

    * A disk was part of a Storage Space, it was disconnected and put away for a long time.  If I connect it, it will automatically be added to the Storage Space.  However, it has nothing but stale data, and I only want to format it as soon as possible. Is there a way to prevent it from being automatically detected as part of the Storage Space?

  59. Rabbit@Net says:

    Finally a technology to satisfy the storage needs of power users and small business alike. It seems easy to use and scalable. Great job!

    While this blog is primarely intended to be an exchange of thoughts on the comming desktop OS, I wonder about some enterprise freatures in the storage technology.

    @Rajeev: You mentioned that the storage pools will scale to hunderts of discs in a data center setup. This raises to questions for me:

    1.) Will it be possible sometime soon to have redundant storage pool managers (read: two or more Windows OS instances handling the same (!) storage pool) ???

    2.) Will the hosting of virtual machines within a storage space be supported?

    Thanks in advance for your reply!

  60. Some more responses:

    @Oren Deri – the write performance of mirrored spaces is competitive with optimized RAID 10 implementations. You can certainly use mirrored spaces for write intensive applications. You can also use parity spaces for storing sequentially written and infrequently updated data.

    @danielgr – Defragmentation occurs within a volume as opposed to directly on the space. With Windows 8, a storage optimizer task performs defragmentation as an automatic routine maintenance activity for NTFS volumes (more on that in a later write-up). Once you create and format a volume within a space, the volume shows up as just another drive in Explorer. To find out the actual free capacity for the pool (which applies to all spaces within the pool), you can use the control panel or PowerShell. Of course, spaces can also be “fixed provisioned” in which case the logical capacity specified for the space is pre-allocated to the space from the pool. For a mirrored space contained on a 2-drive pool, you can take one drive to another Windows 8 PC and access data there. You can also add a new drive to the original PC and Storage Spaces will automatically repair the mirrored space using the new drive. You cannot create a parity space on a 2-drive pool. You cannot take a single drive backing a parity space to a different machine and be able to access data off that drive.

    @Stefan – as long as disks connected through these cards are directly accessible to Windows, they can be added to a pool. However, we do not recommend RAID disks exposed through such cards to be added to a pool.

    @Moses – a pool can contain many drives (>100). However, a single parity space can have a maximum of 8 columns – note, however, that in a pool comprising > n disks (where “n” = number of columns for a given parity space), the slabs composed from these columns will be laid out flexibly over different combinations of “n” disks. Of course, you could construct multiple parity spaces from all disks comprising a pool, each of which will flexibly and optimally use all available physical disks within the pool. Mirrored spaces (both two-way and three-way mirrored) can also have >8 columns per mirror for increased performance. In short, your pool can certainly have more than 8 drives and you can create many spaces from each such pool with Storage Spaces optimally using all physical drives within the pool.

    More shortly. Please keep your questions, comments, and feedback coming. We appreciate the engagement and interest.

  61. eshan says:

    This is absolutely a killer app feature for me when considering whether to continue with Microsoft. I had started to look at osx server as an alternative to my current media server ( I've gone through WHS and now use window 7 as my media server environment.)

    Its a bit of an understatement to say Im pleased about this:)

  62. MikusR says:

    "At release, we will offer guidance on how you can add appropriately partitioned system/boot disks (with dynamic volumes) to a pool."

    Does that mean that you can add a part of dynamic disk to the pool. i.e. I have a 2 2TB dynamic disks and I can create a pool using 1TB volume of each.

  63. xpclient says:

    1.  Can I have a combination of mirrored and parity attributes in the same space and how many minimum physical disks will be required for that? Currently, I use Intel's Matrix RAID, which allows a combination of different levels of RAID on the same array in any assortment. e.g. I can combine RAID 0 and RAID 1 to get the benefits of both on just two physical disks. Or RAID 0 and RAID 5 with 4 disks. Plus, I can also boot from Matrix RAID array. What advantages do you get over a solution like Matrix RAID which also has very easy management?

    2.  Please expose all operations related to Storage Spaces in the Control Panel GUI. Don't keep some operations exclusive to PowerShell. That would reduce the user friendliness of this whole feature.

    3.  The question "Is there a defrag or CHKDSK equivalent for pools?" isn't clear. Does it automatically perform chkdsk like operations to maintain the health of the pool? Doesn't it have to take the physical disks offline?

    4.  Where is the pool and quorum information stored? On disk, on the file system or in the registry? How does moving it across Windows 8 PCs automatically make it work?

    The FAQ approach is nice. Wish you did these FAQs and added user FAQs for every post henceforth (and for older posts as well).

  64. blapp says:

    My head explodes by considering that anyone is trusing your bullshit enough to let it care for a single MB of data. Seriously. Every occurance of Windows is a plainly a cruel pain in the ass. Why don't you guys just stop torturing the world with your stuid bugware.

  65. Misha Pisha says:

    Windows 7 Service Pack 2? When? Give news about it.

  66. JoeH says:

    Simply awesome. I've wanted something like this for a long time. Thank you.

  67. @sgy newest build is 8176 and is from 3rd january 2012  but were I can get it tell me download link please

  68. AndyCadley says:

    This is, without doubt, one of the most exciting new features for Windows 8, it's going to be extremely useful in many scenarios.

  69. Christian says:

    Dear Rajeev Nagar [MSFT]

    Do you have any plans for implementing a Unraid/flexraid solution ?

    X+1 Drives, with the +1 used as as Parity (as long as it's bigger than the Xs).

    (I dislike striping because its very unreliable (lose 2 drives, you've lost everything)

    Having a "pool" that i can expand at will with 1 parity drive would be awesome…

    Specially if i know that "at worse" i'll lose the data on the dead drives (if 2 or more die).

  70. Carsten says:

    I've some questions:

    – Would it be possible to run third party tools/filter drivers like Truecrypt between a Space and the file system?

    – What about checksums, deduplication, etc?

    – Is the parity/mirror information saved on sector level or only for allocated data?

  71. Weslee db says:

    This really is a great feature.

    I think this + the integrated personal archive in exchange 2010 = easy and cheap JBOD solution.

  72. Interesting idea but just like raid it will probably make life more complicated not less!

    Can everything you describe be organised directly from control panel or do you constantly have to delve into powershell as you seem to suggest?

    Sounds like a "feature" that might be beta but not quite make it to RTM! 🙂

  73. folken says:

    Good. Finally welcome to this century of IT. (Actually last century, when we talk about HPUX LVM)

    The Following would make it a killer solution:

    * Add support for network storage (ISCSI, Physical Devices over Ethernet, maybe even CIFS, or a really nice protocol by which you can access physical disks hosted on other machines.)

    * Add support for shrinking spaces.

    * LOSE THE DRIVE LETTERS PLEASE! (What if I have more than 26 spaces?)

    * Provide a way to cache files over several drives, according to the performance of the drive.

       – E.g. you have a fast SSD and large conventional spindle drives. Use the SSD to its fullest for frequently accessed data, and for large files that are infrequently accessed move them to the large drives.

    Reinventing UNIX one step at the time.

    – Folken

  74. See it in action! says:

    This feature would be cool to see in action! Why no video? maybe we see more about Win8 at CES!

  75. I still think there should be one single panel for all those features discussed in this post and in the previous one, something like a “Backup and Restore Manager”. Honestly, the current way those features are presented lacks coordination, friendliness, and aesthetics. The common user will probably miss the big picture, won’t even realize that all those features are there and in which way they differ from each other. And, by the way, Search doesn’t help that much. For example, when I search for “Restore”, there are up to 10 relevant results out of which I have to choose. Search experience does make it less messy only if one already knows what he is looking for.

    There should be one unifying panel, one default tile for this panel on the Start page, and one clear link to it in both control panels. And this panel should look like this:

    Backup and Restore Manager

    Option 1. Backup and restore through restore points

    [Explanation: what it does? when to opt for it?]

    File History

    [Explanation] [Link for more information] [Button]

    Open System Restore

    [Explanation] [Button]

    Configure System Restore

    [Explanation] [Button]

    Option 2. Backup and restore through Windows Backup

    [Explanation: what it does? when to opt for it?]

    Option 3. Backup and restore through Storage Spaces

    [Explanation: what it does? when to opt for it?]

    Option 4. Backup and restore through system image

    [Explanation: what it does? when to opt for it?]

    Option 5. Refresh

    [Explanation: what it does? when to opt for it?]

    Option 6. Reset

    [Explanation: what it does? when to opt for it?]

    Option 7. Clean Reinstall

    [Explanation: what it does? when to opt for it?]

    Windows Easy Transfer

    [Explanation] [Button]

  76. There are 2 questions I would like to ask, the answers may be mentioned before:

    1. If I create my storage space, there is no way how to change its storage layout? After some time I will find out, that my pool is running out of space, so I would like to switch my Photos space from 3-way mirror to 2-way mirror or parity. Will it be possible? I understand it is quite complex operation, but if data are distributed in slabs, this operation means just deleting additional slabs or building new slabs with parity data.

    2. As mentioned in parity mode about 8 columns, does it mean, that my parity storage space is limited by size of 8 drives from the pool or is it just my misunderstanding?

    Thank you

  77. Ashwin Nanjappa says:

    The ugly Up button is appearing everywhere! Either remove it or at least *please* integrate it more elegantly into the Explorer UI!

  78. sainik biswas says:

    Is it possible to say insert two usb drives of  1tb each create a storage pool of 2tb then dump data to it and unmount the storage pool and when required i can just plug back both my drives and get my storage pool back something like RAID over USB.

  79. Tom Servo says:

    Controlled drive replacement would be favorable. The post seems to imply that redundant spaces are a requirement. That's kinda uncool. Because it sounds like if I have an non-redundant space in the pool, I can't replace drives.

  80. @ZipZapRap – although Storage Spaces were not designed for your particular mode of usage, you could achieve your goals by creating a two-disk pool (excluding the internal drive) and pretty much do as you described.

    @Mike M – thank you for the receptiveness and feedback. We will provide much more detail on performance in the near future. Mirrored spaces maintain information about I/O’s in flight and are able to rapidly resynchronize the minimum amount post unexpected power-loss. Parity spaces include a journal to ensure data integrity regardless of write size and in the presence of unexpected power loss. Stay tuned for more information on the work we have done with Windows file systems – this work builds on Storage Spaces. Parity spaces do use some memory caching to improve performance. Storage Spaces do not use a SSD as a write buffer although SSDs can be used to back the journal thereby helping performance for parity spaces. While our goals are to deliver very simplified, lights-out operation, we do enable you to set number of columns, stripe size, or even specify the exact backing media for specific spaces – which would address your specific example. There are additional knobs you can tune. Storage Spaces were designed to be deployed on the host (in virtualized deployments) thereby significantly lowering capital and operational costs for highly consolidated scenarios – we would appreciate your feedback and any suggestions for improvement once you have tried it out. We certainly recommend deploying VHDs on volumes contained in spaces. We have tuned our implementation to minimize CPU overhead and will provide detailed guidance/best-practices for large deployments. These capabilities are indeed part of Windows 8 and available with Windows 8.

    @Lionel – we have already publicly announced interfaces exposed through Storage Spaces that would allow applications to utilize their own check-sums in conjunction with Storage Spaces to auto-correct data. Stay tuned for more information on our file system enhancements. You can change the way a disk is connected as you describe and the disk will continue to work as part of your pool. Network drives cannot be part of a pool. We do not recommend iSCSI drives in a Storage Pool except for test deployments. The Recycle Bin works within the file system and hence, works just as today for a volume contained within a space. NTFS formatted volumes work just as expected when deployed on a space. As demonstrated in the blog, Storage Spaces are a fundamental platform technology and as such, integrated with BitLocker for encryption. You can definitely put VHDs in volumes contained within a space. We do not recommend using VHDs as part of a pool except for test deployments. A substantially revised CHKDSK (more on that later) continues to work as expected on NTFS volumes contained within the space. You can simply remove the disk from the pool and it is then available for any alternative usage.

    @Rabbit@Net – Yes, we support clustered pools which scale to multiple nodes and large collections of disks. Yes, for enterprise deployments, we encourage Hyper-V usage with Storage Spaces.

    More shortly. Please do keep the comments coming.

  81. azyr says:

    Hey, this is really interesting, i'm quite hooked.

    I'm a newbie on raid setups and everything generally associated with it (software drive extending or HW raid controllers).

    I've just got basic knowledge on what mirroring a drive is and i do understand what pooling is (combining 2+ drives to act as a unified space right?).

    What i want to know is this;

    I have 6 disks, I want to set these up so that they are combined as a unified drive (not that i need of a unified space as they are 1TB or so each but bear with me). Does the parity res. option allow me this ;

    Backup of data so that no matter WHICH 2 disks (<%50) fail, i just pop in 2 new ones (with the same or more capacity) and i'm good to go (with maybe some recovery time for the system to rebuild the structure for the new disks).

    This is essential to me, the data contained in my current disks beg for backup and the idea to just mirror them seems wastefull, but necessary if i can't find anything better.

    Can someone enlighten me on this topic?

  82. AEonCIpher says:

    Can you "convert" NTFS formated drives that currently have data on them to create a Windows Storage Space Pool? For example I have multiple 1.5TB drives with data already on them can i keep that data and add the drive to a pool or do I need to create a brand new pool with fresh drives then 1 by 1 copy the information over?

  83. Tom Servo says:

    Another one: Does the filesystem on top, which is assumed to be ReFS, if you are to believe the leaks, coordinate with the slab allocator to release slabs that have been cleared? It would be hilarious if frequent storing and deleting of large files would blow up the space, because there's tons of slabs allocated to it, which go unused. Similarly, cleaning up a lot of data from a space and then defragging it would also result in quite a few empty slabs.

  84. Single instancing? Data de-duplication ? … please tell me it's integrated. Please.

  85. @BranM: Some kind of data de-duplication was mentioned in Windows Server 8 Storage session at Build conference and was shown as very efficient way to store VHD files.

    Will you provide us please more information about mystical ReFS and its capabilities?

  86. Sergey says:

    Would Window detect data corruption in background?

    The typical scenario is that you store your old photos (with redundancy) and never access them. If system would not verify data integrity time to time it is question of time when you lose both copies.

  87. Those who have used WHS v1.0 will appreciate this question:  Will Storage Spaces have a UAC GUI similar to that found in the WHS1 Management console?

    The UAC tab in the WHS1 management console made it extremely easy to manage individual user access to each shared folder.  I sincerely hope they implement something similar in Win8, rather than the old-school method (ie. right-click -> properties -> Security -> Sharing -> etc etc).

  88. sean.e says:

    Sounds very interesting.  I want to echo calls for checksumming – this came up recently at slashdot:

    (url truncated in order to try to get the comment to post) story/11/12/26/0030241/ask-slashdot-best-kit-for-a-home-media-server?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+Slashdot%2Fslashdot+%28Slashdot%29

    I'm currently using Win7 Pro as a media server and want to add checksumming to my drive pool so that I can sleep better at night.

    Will there be a formal way to mirror a space for offline backup/storage or would that be simply via a standard backup or robocopy script?

    I appreciate that a drive from a mirrored space can be read when attached to another PC.  Since a parity space can not be read in the same way, I would probably use a mirror space for large media files.  Would there be an appreciable performance hit for doing so?

    Finally, are physical drives in a pool able to spin down independently of each other?

  89. sevenacids says:

    Please, either change the SI prefixes (MB, GB, TB…) to binary ones (MiB, GiB, TiB…), or display the correct values if you continue to use the SI prefixes (1 KB equals 1000 Byte, NOT 1024 Byte!).

  90. Okosisi says:

    This is excellent work, but 2 things:

    1. Give this a better name – Storage Pools and Storage Spaces are so computer science class. Windows is now an appliance that a billion people will use. No one names their toaster "heating element 2000". Have a Heart. Some off the cuff suggestions include "Infinite Disk", "Virtual Drawer", " Personal Storage". Its should be fairly trivial to boil this down into the core customer value propositions and distill a catchy colloquial name. Apple does this well – what does Time Machine mean? Or Siri for that matter? Yet millions say its with perfect cognitive clarity about what it does.

    2. I applaud how simple you've made a complex piece of technology in terms of configuration, but this could be simpler. The UI still looks inaccessible to the man on the street clutching on to misconceptions about Macbook vs. Windows. This UI is crying out for richer icons, a pie chart to show content distribution, or an artfully drawn cylinder indicator for fullness. The progress indicator UI just doesn't cut it.

    I know part of this is the control panel framework – everything goes in there. But its so limiting, so vertical linear, with terrible use of space. Imagine how much white space will be in that cpl on a 24 inch screen? Keep the entry point in the panel and design a really beautiful, information dense and drop dead simple configuration UI for this really important piece of technology; that launches from there. Posterity will reward you.

    I don't expect these 2 things to be done to everything. But it should be a consideration on important technology like this. Heck keep the enterprise name seperate from the consumer name if you want or find a name that is appropriate for both.

    Windows is a sink of value because of similar oversights. Its like an iceberg with millions playing in the 10% while the 90% (of stunning technology) remains undiscovered and unacclaimed – how many times do you have to say "but wait that was in windows 5 years ago!" to other OS zealots before you learn? Its a disservice all around to the creators, customers and the shareholders.

  91. Musafir_86 says:

    -For those who want a bit more technical details on this, you may refer to these TechNet articles/blog posts (authored by the very same Mr. Rajeev Nagar here):

    blogs.technet.com/…/windows-8-platform-storage-part-1.aspx

    blogs.technet.com/…/windows-8-platform-storage-part-2.aspx

    Regards,

    -Musafir_86.

  92. Julien Couvreur says:

    Why does the size of a Space have to be declared up-front? If the provisioning is "thin" or delayed, then why can't it be elastic and continue growing (if underlying storage pool has room)?

  93. Could you please explain the purpose of the quorum? As far as I know, it's a safety mechanism to prevent split-brain scenarios, especially in cluster systems using shared storage.

    This means, that you for (2 x n) disks or (2 x n + 1) disks you need at least n + 1 disks to access the pool.

    But, in my opinion, quorum is just a necessary, not a sufficient condition for a functional pool.

    Let's take your example, with 6 disks. You say you need 4 disks to achieve quorum and activate the pool.

    If we look in the table that shows how you mirror a space when you have a 6 disk pool, we can see that the data in the storage space between offsets 127.00G and 127.25G has two copies, on disks 6 and 7, between drive offsets 42.75G and 43.00G

    Let's say we have access only to disks 2, 3, 4, and 5, and disks 6 and 7 are missing. By having 4 disks, the quorum requirement is satisfied. But you still can't access the storage space data between offset 127.00G and 127.25G because both copies were stored on disks 6 and 7 that are missing.

    So, how can you say that 4 disks are enough to access the spaces from a 6 disk pool? This is clearly not the case, at least not for any 4 disks. Can you clarify exactly how the system works?

  94. Chris says:

    My own question – but backed up by vladimir:

    if you are only calculating single parity – and available space is n-1, then n-1 is the minimum number of disks you must have.

    in your example you say multimedia is available with 4 out of the 6 disks – surely it HAS to be 5 out of the 6 disks for the pairty protected space?

  95. Niels says:

    Sounds very interesting.

    Will this work with Intel Rapid Storage Technology that is provided with the Z68 chipset? What about hybrid harddrive technology?

    I like that these features become available in the OS, it provides a lot of flexibility. I am not sure when it makes sense to do this in HW/FW vs. SW.

    Will we see Microsoft Home Storage Servers that will work as a NAS with a home version of Active Directory linked to Windows Live accounts? One of the biggest issues that I have in our home network is to manage users (Homegroups dont work), most Network Attached Storage does not support kerberos security.

  96. AMicSys says:

    Awesome! Much awaited features in here. Kudos to author for the extensive post

  97. Is there any initiative in place or planned to encourage OEMs to include front-accessible, quick-access hard-drive slots in cases, with support for the hardware indication lights you mentioned? The reason that USB hard-drives are so popular is that they are easy for non-technical users to install, though they obviously have a significant performance penalty. Even technical users would greatly appreciate it, as accessory kits are already becoming more common on hardware enthusiast sites. Also, is it / would it be possible for the indication lights to inform you when you should consider adding additional storage? So Red for failure, Orange for Warning, Green for Operational and a Blue for recommending extra storage (next to an empty slot or wherever is deemed appropriate).

    I'd like to see quick-access slots on the front of cases with indication lights. For instance, earlier today I had an issue with a faulty SATA cable on one of my seven internal drives. To diagnose it I had to open both side panels on my case, disconnect each drive in turn and load up EFI/BIOS to see which drives were showing, do this multiple times to find the problem drive, then replace the cable, check that it was now working and then close up my case again. It would have been a lot easier to simply see a red light on the front of my case and to just reinsert my hard-drive (or if it had failed replace it), with no need to open up the case at all and with no need to disconnect a lot of drives unnecessarily.

    As for the implementation discussed here, I'm concerned about the idea of unrelated storage spaces being used. Why should it show 10TB if you only have 4TB connected? I'd much rather see an indication of how much space has been used – as is traditional – or to simply omit the size information altogether and the icon to change when an additional hard-drive is recommended. Making up fictional space is counter-intuitive. I understand the desire to disconnect users from reliance on the physical space to let them know the space can be expanded indefinitely but making up a fake physical space is not the answer. Other than that this is an excellent feature but I'm concerned about how it will be marketed to less technical people and to experienced users, who prefer folders to libraries and want to know which physical disk data is on – it doesn't seem like an easy feature to promote.

  98. Tom says:

    FFS, this damn thing seems to have lost my comment.

  99. sean.e says:

    @Tom – did you have a url in your comment?  seems it automatically rejects (without notification) any comment that has a url in it.

  100. sean.e says:

    note to self – verify before posting…

  101. To  me and to many others I am sure, this is probably one if not going to be the most important new feaure in Windows.

    Because what we have right now right here at home is a lot of files. And the PC or workstation is all about files.

    2 years ago I still had my file system partitioned into different uses or applications. In practice different folders.

    As I went into my user space it was all dissambled because I had to spread it out on many disks.

    Finally I put all that was around onto a piece of paper and did an empty box that said user space.

    I the planned how to move all those files logically into that space. Then I changed the storage to fit.

    Worked well, however as stuff continued to grow I rather quicly grew out of that changed or fitted storage.

    So this is just great new. Now the concern of storage capacity is finally separated from the file system. Thanks a lot.

    I also used RAID 1 or simply mirroring for many years and never suffered a data availbility  problem locally. I still did backup though. Mainly for the system – and for data recovery as well.

    The only problem would be if the house burned down or a lightning stroke blowing everything out or water floods due to environmental changes around here or something similar.

    In case of the fire scenario – it would be nice, if we could start mark a folders at some point for backup to Windows Live, our roaming profile or something similar.

    So how will the backup feature to Windows Live or similar online storage work with Storage Spaces? It would be nice if they are going to work closely togther – just like if the legacy concept of Backup (really Data Recovery and today's sync) will be just as much as part of Windows Storage Spaces as the other attributes current in there. So when doeing "backup" – get rid of the backup concept and just make it part of the Windows Storage Space component. Then we'll never need to worry about that any more :o)

    A complete capsule.

    But I guess I could just add Skydrive to my documents space and get it all sync'ed. But there we have it with some practical problem for online creeping back in if My Documents where like 1TB or if I have a lot of raw personal HD footage which I do. If the sync if case of online where to be at the folder level – then some attributes of the storage concept creep back into the file system.

    So in case of online WAN storage – if we could add a folder or really an online file as a disk – and then tell the storage system that in this case of the syncing ("thin" mirroring) a local resource (a Windows Storage Space, i.e. 10 TB) with an online WAN resource (i.e. 25 GB – so that has an impedeance mismatch with 10 TB) – we can in a managed way give priority to what should be "thinly" mirrored or synced online – i.e. by marking folders and files. To do that unmanged or automated could be an additonal otption – however, Windows would have to interpretate use of the file system then, i.e. what is virtually or acutall most important to me at any moment if I only had 25 GB cloud storage and wanted to maximize its value or effectivity adding it to a storage space – even though the impedeance mismatch would be big … but still as a consumer Windows would just know how to tkae advantage or in the manual option would support me in quickly taking advantage of any cloud storage in the data recovery case for home burned down to the ground.

    Storage Spaces – A very important feature to have. I think it is almost finalized. Thanks.

    P.s. What about Anti-Virus behavoiur with Storage Spaces? AV is file sysem level – but if I added some WAN resource to the storage space would that AV program not risk treating it as a local resource (like we saw in the old days with mapped network drives (looking to the AV as if such a drive was a local harddisk – and network performance went down). So hope there is something buld into Storage Space making them AV-like programs not having to worry about what "HD" is local and what "HD" is a networked resource.

    Looking forward to read much more about all this … :o)

  102. Everything is new in Windows 8 but this is just so basic – because this is about data, practically speaking today still mostly about files, which practically drives our use of computers – and in addition and still too about the storage capacity below.

    The big news it that storage capacity and harddisk are taking out of this equation. Practically speaking, now we don't have to move files around or break down the logic of the user concepts you have build into your folder hierarchy at home to get room for more files.

    File system and storage have been separated in an effective way – great value. As good as it gets.

    We've got plug'n'play storage.

    Please note. This post more or less states, that beginning with Windows 8 the concern of disk size is more or less gone. The post says you might as well let it just grow dynamically towards the current upper bound of 50TB or just set the logical disk size to 50TB.

    So now we can figure out there is really nothing left of user interesst concerned with disk size. Disk size is all narrowed down to a teoretical technical problem only of concern if use of some space is closing in on the 50TB upper limit potentially forcing a redistribution of files)

    So disk sizes just disappeared not just as a user problem – but completely for some years (the road is a bit long to 50TB – all though we're getting there – I'm at 4TB). When super highdefinition video comes around – we'll probably get there quickly.

  103. High resolution data samples of nature (with infinite resolution?) driving storage use up? I.e. video?

  104. Tom says:

    @sean.e, not that I recall. It seemed to just reload the page when I hit submit. What pisses me off more is probably that fact that web browser STILL do not store multi-line form data in history. What the hell is with that. It's the one thing you want in your History.

  105. Dr AJ says:

    no one has said it yet so I will:

    Poor man's Drobo

  106. Alistair says:

    Agreeing with some other comments and comments on various forums, I don't understand the approach to thin provisioning, if the maximum space is not real then why bother setting it at  anything below to 50TB maximum? Why bother even passing this information along to the end user when it reflects no real and thus useful value?

    This system proposes to be simple but I believe it could go a step further without sacrificing functionality.

  107. orcoipod says:

    It would be nice, in case of drive failure, the system will reports the S/N of the disk. Usefull for the internal drives

  108. SA says:

    "In Windows 8, you cannot boot from a space".  Does this mean that you cannot use Storage Spaces to backup the primary system disk that Windows and the applications reside on (i.e. the C: drive), or do you provide a work around?

    Also, does Storage Spaces support TRIM for SSDs?

  109. arianw says:

    Just one more vote for supporting network drives with this 🙂

  110. Dopefish says:

    Thanks for this feature, this is something I've been waiting for a long time!!!

  111. kman says:

    Rajeev,

    If the power is interrupted during a rebuild, does the rebuild continue once power is restored and the machine restarted automatically?  Or how will power failures be handled?  

    Thanks in advance.  I was going to buy a drobo, but I think I'll wait for windows 8 instead now.  Can't wait for 8!   🙂

  112. Carnitron says:

    This is absolutely brilliant.  Almost makes W8 a must have on its own, for me.  Finally, a seriously meaningful attempt to do away with the tyranny of the drive letter and all the needless complexity that causes for non-savvy users!

  113. John says:

    Win8 looks great, so long as you can disable that horrid metro crap.

  114. Tom says:

    Ok, I'll retype my comment because I think it's worth it.

    Rajeev, I appreciate the efforts put into Storage Spaces. It's certainly a step in the right direction. Backup is a critical aspect of any home desktop, and it's good to see Windows making it easier for the average joe to setup, and more flexible for the poweruser.

    As important as backup is though, it's rarely actually called on. It's really only in the event of a hardware failure that backup's like those provided by Storage Spaces are useful. The major cause of data loss is as a result of accidental deletion, and even more so, accidental overwriting of files. Volume Shadow Copy/Previous Versions/System Restore or whatever you want to call it, was a very welcome addition in Windows Vista, though it was a shame it was only available to Business and Ultimate users. Windows 7 thankfully brought this important feature to the home editions of windows. Previous Versions needs some attention though, as it's got a number of areas for improvement.

    Previous Versions is only enabled on the boot drive by default. Increasingly, users are storing their data on drives other than their boot drive. Previous Versions should therefore be enabled by default on all internal drives (anything connected over SATA/SAS or SCSI), and users should be prompted to enable it whenever they plug in an external hard-drive for the first time.

    Previous Version snapshots are also not taken regularly enough (only about once a day by default), and there's no easy way to increase the regularity. Adjusting the scheduled task in task schedular often breaks the schedule from my experience. I've had to resort to a VBScript to trigger off previous versions once an hour. It works, though my whole machine has a major hiccup lasting about 5 – 10 seconds after the job begins.

    Finally, and most importantly, Previous Versions should use ALL available disk space. One should not have to allocate a certain percentage of disk space to previous versions. It should just use whatever free space is available. As the user consumes more of their hard-drive, only the oldest previous versions should be removed to make room. This would make previous versions behave more like the Windows RAM caching machanism (e.g. SuperFetch), where all available RAM is used to speed up application boot times, but the RAM is quickly made available whenever it's needed. After all, free hard-drive space is absolutely useless unless it's being used.

    I really hope the Previous Versions feature is given more attention, as it's versioning that users demand more and more. Previous versions has saved my ass a number of times, but it could have saved my ass even more times my suggestions were implemented. I've seen too many people lose valuable data as a result of trivial mistakes, sometimes not even of their own fault.

  115. @Vladimir-Corneliu Nicolici:  According to the information in the article, having a quorum of disks allows you to see and have control over the spaces in that pool of disks. However, this does not mean the actual data is necessarily accessible. The resiliency attribute of a particular storage space (two-way mirrored, three-way mirrored, or parity) will determine how many disks are needed to actually access the data in the space. The example that you cite from the article is a two-way mirrored space, so you could only safely lose 1 disk (Rajeev misstated this in one of his answers, but the FAQ is correct). A parity space would be the same way. A three-way mirrored space could lose any 2 disks since each slab is always on at least 3 disks at a time.

  116. daviesow says:

    "Thin Provisioning" is a nice business buzzword but that's all it is.  It may have made sense compared to earlier fixed provisioning schemes but in this case it's a half-step short of a much better paradigm.

    I agree with others, why set a bogus number and maybe have to go back and bump it up later?  This leads to two scenarios:

    1) Your applications believe they are out of room when, in fact, more storage is available.

    2) Your applications falsely believe they have more space available when you run out of underlying physical storage.

    In Scenario 1, after your application runs aground, you simply need to go bump up your pretend size.  Presumably in Scenario 2 your Storage Spaces are sending emails, popping messages etc. in advance.  Hopefully they get your attention in time to avert disaster.

    In the best case you've added unnecessary administration tasks.  In the worst case you are causing applications to crash (probably in a controlled manner for Scenario 1 and an uncontrolled/catastrophic manner for Scenario 2).

    Why not just let each logical Space report that is has as much free storage capacity as it could possibly present given it's mirroring/parity schema and the actual free physical storage backing the Space?  For example:

    The physical pool is 4x 2 TB drives for (8 TB raw capacity).

    C: is a Space with no mirroring or parity

    D: is a 2-way mirrored Space

    E: is a 3-way mirrored Space

    F: is a Parity Space

    Initially, with no data stored on any of them they each report free space as:

    C: 8 TB free

    D: 4 TB free

    E: 2 TB free (Theoretically 2.66 TB but not unless we had 3X drives)

    F: 6 TB free

    This is a "thinly provisioned" picture as obviously you don't have 20TB but it is also a more realistic picture as each number is within the realm of possibility.

    Now say 1TB is written to Space F:.  That physically occupies .33 TB on each of 3 drives with .33 TB of parity data on the 4th.  Now free space would show as:

    C: 6.67 TB free

    D: 3.33 TB free

    E: 1.67 TB free

    F: 5 TB free

    A second example, same setup but this time 2TB is written to Space E: and nothing is written to C:, D: or F:.  This occupies all 2GB of three of the drives and none of the last drive.  Now free space is reported as:

    C: 2 TB free

    D: 0 TB free

    E: 0 TB free

    F: 0 TB free

    In this case the only "weirdness" for applications is that drive space can magically go away even when nothing is written to that drive.  I don't think this is a problem in most cases as applications don't assume they have complete control of the storage.  They must assume other applications could write to storage at any time, filling it up without their knowledge.  This is certainly less likely to lead to disaster than scenario 2 above where the drive reports free space but the application can't actually write anything more.

    Please consider revising Storage Spaces to eliminate unnecessary administrative overhead and present a truer picture of free capacity to all applications.

  117. Offtopic says:

    With the first public beta approaching and the release due this summer or Q3, the need for a place to collect never fixed Windows show stoppers is urgent. Windows 8 has been called "the most risky" by Balmer. It evidently is. Does MS need people saying "s..t, they would better fix…"

    For example, files are associated with applications. How come I cannot DELETE an association through GUI? Or how many DECADES does MS need to teach the Japanese IME to respect the specified default setting? If I say the input mode should be Hiragana, IT SHOULD BE HIRAGANA, not Half width alphanumeric!

  118. Meitzi says:

    Posting older blog is disabled so I will post here.

    I want Windows 8 to support "storage drivers fix on boot". (I'm talking mostly SCSI drivers here)

    If you move your Windows to new computer (lets say whole HDD) to save time to install everything again. I wont boot if new computer have different SATA/SCSI. You can manually fix that but its difficulty. (when it wont boot and you cant just istall drivers in old computer where that device does not exist)

    So, we need "install storate drivers" boot fix option which will install them for me.

    (couple of times I had fix servers too where drivers where not bootable)

  119. Nice job. I missed the whole WHS thing until the feature was removed (and was the whole reason I got excited about WHS – and is also the reason a few of my WHS users have switched recently to other technologies). Bringing this to W8 is a fantastic move. Congrats. My massive media centre dreams live on.

  120. I also would really like to know:

    Would Window detect data corruptions automatically and correct them?

    You say this behaviour can be achieved via application support (checksumming) and the Spaces APIs.

    For me, as a normal user, the "application" is Windows Explorer, Photo Gallery etc.

    How do I deal with silent data corruption?

    Otherwise, this sounds great!

    Will the on disk format be final in Win8 beta? How stable will the Spaces code be?

  121. Carnitron says:

    Now that I've digested a bit:

    1. Can someone confirm whether this is going to be in all versions of W8?  Please tell me it's not server only.

    2. I agree with the people who are asking: what is the point of sizing the space up front?  Why needlessly complicate things with this number, since it doesn't seem to have a concrete, useful meaning for the end user?  If you can't shrink it, and can just expand it later as it becomes necessary, what is the point?  Why doesn't windows just hide this and do it automatically?

    3. I agree with the people who think the UI could be made prettier, and more end-user friendly, particularly in terms of correlating the drive on the control panel to the actual physical drive you just installed.  I love how one of the drives I have shows up in Device Manager as "WDC WD1002FAEX-00Z3A0" instead of something a normal human would understand, like "Western Digital 120 GB HDD – Model 1002" (or whatever).  Heck, it'd be even cooler if OEMs could include pictures of their HD's like we get in the "Devices and Printers" window, so I could actually know the drive by sight when I'm looking in my machine.

    4. Ditto with the suggestions that MS work with OEMs to turn hot-swapping into a task that doesn't involve anything beyond opening a door and just pulling the drive out, like a game console cartridge.

    And now the biggie, as far as I'm concerned.

    "In Windows 8, you cannot boot from a space".

    I'm getting cold shivers from this, like maybe some of the others here.  What does this mean?

    All of my machines have Windows and all my apps on two RAID 0 SSDs, and all my documents, music, videos and pictures on large slow HDDs.

    How do I get all the juicy benefits of Storage Spaces on all the SSD stuff?  If Windows won't boot from a space, does that mean that Windows itself — and be extension, everything in Program Files — can't get the automatic resiliency benefits of Windows Spaces?

    I have to tell you, it is FAR more punishing to me when a drive with Windows on it goes down then when my data drives go down, b/c I back my data up pretty regularly.  But having to rebuild my machine from scratch is a total, weekend-destroying, pain in the rear.  Yeah, I know I should back an image of my main drive up, but the overhead of that is so much higher in some ways than just periodically dragging all my data files from one drive to another.  Heck, my kids' machines each have less than 1 MB each to back up.  (Low-tech I know, but super simple, almost zero upfront time investment, etc.)

    So basically, my concern is that if Windows 8 cannot boot from a space, it'll kill one of the most potent benefits of this technology for me:  not having to worry about Windows and my apps being trashed in a time-consuming-to-restore way if a drive goes down.

    Help me Rajeev, tell me this isn't so!

    (Tried posting this yesterday, but it seems like the blogging system ate it.)

  122. Carnitron says:

    Now that I've digested a bit:

    1. Can someone confirm whether this is going to be in all versions of W8?  Please tell me it's not server only.

    2. I agree with the people who are asking: what is the point of sizing the space up front?  Why needlessly complicate things with this number, since it doesn't seem to have a concrete, useful meaning for the end user?  If you can't shrink it, and can just expand it later as it becomes necessary, what is the point?  Why doesn't windows just hide this and do it automatically?

    3. I agree with the people who think the UI could be made prettier, and more end-user friendly, particularly in terms of correlating the drive on the control panel to the actual physical drive you just installed.  I love how one of the drives I have shows up in Device Manager as "WDC WD1002FAEX-00Z3A0" instead of something a normal human would understand, like "Western Digital 120 GB HDD – Model 1002" (or whatever).  Heck, it'd be even cooler if OEMs could include pictures of their HD's like we get in the "Devices and Printers" window, so I could actually know the drive by sight when I'm looking in my machine.

    4. Ditto with the suggestions that MS work with OEMs to turn hot-swapping into a task that doesn't involve anything beyond opening a door and just pulling the drive out, like a game console cartridge.

    And now the biggie, as far as I'm concerned.

    "In Windows 8, you cannot boot from a space".

    I'm getting cold shivers from this, like maybe some of the others here.  What does this mean?

    All of my machines have Windows and all my apps on two RAID 0 SSDs, and all my documents, music, videos and pictures on large slow HDDs.

    How do I get all the juicy benefits of Storage Spaces on all the SSD stuff?  If Windows won't boot from a space, does that mean that Windows itself — and be extension, everything in Program Files — can't get the automatic resiliency benefits of Windows Spaces?

    I have to tell you, it is FAR more punishing to me when a drive with Windows on it goes down then when my data drives go down, b/c I back my data up pretty regularly.  But having to rebuild my machine from scratch is a total, weekend-destroying, pain in the rear.  Yeah, I know I should back an image of my main drive up, but the overhead of that is so much higher in some ways than just periodically dragging all my data files from one drive to another.  Heck, my kids' machines each have less than 1 MB each to back up.  (Low-tech I know, but super simple, almost zero upfront time investment, etc.)

    So basically, my concern is that if Windows 8 cannot boot from a space, it'll kill one of the most potent benefits of this technology for me:  not having to worry about Windows and my apps being trashed in a time-consuming-to-restore way if a drive goes down.

    Help me Rajeev, tell me this isn't so!

    (Tried posting this yesterday, but it seems like this blogging system ate it.)

  123. Stephen Cleary says:

    I'm looking forward to this feature!

    Out of curiosity, is this technology something that was developed for Azure server, and then brought into the public product?

  124. @xpclient – a single space can use only one mechanism for resiliency i.e. either mirroring or parity. However, as mentioned in the blog, different spaces, each with their own resiliency attribute will use the same backing physical disks comprising the pool. The control panel does expose all common tasks. Pool and space metadata is stored on the physical disks comprising the pool.

    @Carsten – the standard Windows I/O stack executes on Storage Spaces which can certainly accommodate third party filter drivers. Deduplication is available on Windows 8 server and leverages capabilities exposed by Storage Spaces. Mirroring and parity mechanisms are applied to allocated data or metadata.

    @Win5000 – to clarify, Storage Spaces are a fundamental platform technology and, therefore, also used on Windows 8 server. Remote administration, scripting, and usage of more advanced capabilities are all enabled through PowerShell. We have made a deliberate effort to enable all common tasks through the control panel and would appreciate feedback on anything we have missed.

    @folken – you can certainly access more than 26 spaces through NTFS mount points.

    @See it in action! – great suggestion.

    @ David Gešvindr – in Windows 8, we do not enable changing the layout post creation. As described earlier, the size of the parity space is not constrained by the total size of 8 drives – in fact, the size of a parity space is only constrained by the maximum available physical capacity of the total number of physical drives within the pool.

    More comments later today. Keep up the dialog – I highly appreciate the effort and candor.

  125. @Offtopic

    The IME remembers the state just fine.  It sounds like you are using an application that changes the state and then does not change it back.  For example, certain edit controls in an application might be coded to default to Hiragana and not switch back when you leave the control.  Unfortunately that's in the app code not in the IME code that would not know the intent of the app developer.

  126. Tom Servo says:

    It'd still be worthwhile to know whether Storage Spaces can reclaim slabs that have been cleared in a logical volume.

  127. GregH says:

    Would be good if you could add an Azure storage drive that securely mirrors the data in the background off site and bills you simply in your Windows account.

  128. Cameron_Eldridge says:

    Very nice. It's good to see some features available in other OS's coming to native windows instead of relying on 3rd party "solutions" or hardware.

    Aching question for me though, do you think we'll see deduplication in the workstation version(s) of win 8 at all? Just thinking of the immense amounts of data I personally store locally, which is of very little need to be backed up centrally, but managing the space usage is a nightmare with all the duplicate data lying around.

    Perhaps another question that stems from this is, will 3rd parties be able to extend the functionality of storage spaces? So even if dedup is only available on the server OS, you can buy 3rd party products to provide the same functionality at a workstation level?

  129. So this is what Protogon is/does.  😀

  130. Will you make drivers available for Mac OS and Linux? if not, it's a no go…

  131. But are you finally using UTF and has the name limit of NTFS been lifted/fixed?

  132. This could be pretty cool with Thunderbolt, you will support Thunderbolt, right?

  133. @sainik biswas – yes, certainly.

    @Tom Servo – controlled drive replacement is possible but only so for resilient spaces.

    @azyr – yes, you can construct a single three-way mirrored space from your 6 disks and your space will tolerate failure of any two physical drives.

    @AEonCIpher – you will need to create a pool either with new drives or back up data from your current physical drives someplace else before creating a pool from them.

    @Tom Servo – NTFS does issue trims to Storage Spaces as does the storage optimizer. Storage Spaces will return capacity back to the pool upon receipt of trim requests indicating that a slab is no longer in use.

    @BranM and @David Gešvindr – Windows 8 server includes data deduplication which also utilizes Storage Spaces based redundancy.

    @Sergey – a scrubber periodically runs (in the background) to detect and correct inconsistent copies of data.

    @sean.e – stay tuned for information on Windows 8 support to protect data from loss. You can use spaces as the destination for your backup copies. Large media files are typically sequentially written and read – mirrored spaces perform very well on such workloads. Drive within a pool can spin down independently.

    @Julien Couvreur – specifying a maximum logical size for a space allows you to limit maximum physical capacity allocated to the space. Note that multiple spaces are backed by the physical drives comprising the pool. The space size is also exposed to the file system and applications.

    @Vladimir-Corneliu Nicolici – you are correct in your analysis that quorum is a necessary yet not sufficient condition. Quorum enables safe modification of pool and space metadata since a majority is present. Despite quorum, non-resilient (simple) spaces cannot tolerate loss of any disk backing the space. Two-way mirrored and parity spaces can tolerate the loss of a single physical disk backing the space. Three-way mirrored spaces can tolerate the loss of up to two physical disks backing the space.

    @Chris – not so. You can explicitly constrain a parity space to be backed by a subset of physical disks comprising the pool.

    More responses shortly. Thank you.

  134. abdo says:

    i think this is a good step..and im looking forward for it

    full-windows8.blogspot.com/…/Windows%208%20Tutorials

  135. Carsten says:

    @Rajeev Nagar: Thank you for the answers. That really sounds great, except the point that deduplication is only available in the server version.

    But I've another question: If you're applying parity/mirroring only on allocated or meta data, how do you determine what is allocated, and what not? How does the information get from the file system to the Storage Space? And would this work if you don't know the file system (for example if I use Truecrypt between the file system and the Storage Space)?

  136. jm says:

    Yet another time adding comment failed and my comment dissapered – can you repair this?

    @Rajeev Nagar:

    Thanks for answers. I have few questions:

    I have 5 HDDs – system disc and 4 data discs which fall asleep – If I create storage pool from those 4 discs – will sleeping be possible?

    You wrote "When Storage Spaces detects a sufficient number of disks for quorum, it activates the pool and contained spaces. (…) If the data on any disks becomes out of sync, Storage Spaces will automatically sync them".

    Questions:

    – do I have to wait to resume all 4 HDDs if I want to use storage space?

    – does windows informs in any way that storage space is out of sync because some discs are not present?

  137. thb says:

    there is somethin similar for Linux using SAMBA

    http://www.greyhole.net/

  138. FZB says:

    this sounds promising

  139. Dan says:

    So I take it this could work with iSCSI disks, then? Just thinking of the possibility of having a storage pool consisting of disks physically separated across the LAN. Should make highly-redundant storage solutions far more affordable.

    Now all we need is some kind of wide-area synchronisation between separate Windows 8 machines and we have solid, highly-fault-tolerant storage solution right out of the box!

    As someone who's spent years trying to design cost-effective storage solutions for SMBs, this feature looks to be a godsend. It will sell Windows 8 into businesses all on its own (on the server side, a least).

  140. jm says:

    @Rajeev Nagar

    "As long as you have created mirrored or parity spaces, you can always simply remove a physical disk within the pool, and add a different (perhaps larger) one. Within a short period of time, the impacted spaces will automatically be resynchronized"

    Is it possible to delete phisical disc from a pool (it's not visible on screens)? Because when I replace the HDD I don't want the pool to remember that disc. When I remove the HDD from a pool and format it in another computer (for "normal" use) and then attach again to a computer with the pool – what will happen then?

    If my space is not mirrored or parity – how can I move to bigger HDD? I can attach another HDD to a pool but have no control how data are distributed – how to remove small HDD?

  141. Mike says:

    For those with large media collections on secondary drives this all sounds fantastic.  But if it can't be used on the OS drive then I fear it will fail to address the problems of many users.  Capacity of the OS drive, especially in the era of smaller SSD drives, can easily become a problem.  Windows stores all photos, videos etc. on this drive by default, and it's not trivial for users to move these and maintain the security offered by the Users folder.  Similarly applications such as iTunes can easily store mountains of data in the Users folder.  

    The whole point of these storage enhancements seems to be to make it so that users don't have to think about moving files around because of space issues, but most users are going to have their files on a drive that the feature doesn't support.  It does seem that you're missing a trick here.

  142. NH says:

    Do I understand it correctly that if I create a pool of two 2TB drives, and create a parity space in that, that pool will fit 2 TB of data. If I create a pool of three 2TB drives, it'll fit 4 TB (since one drive is used up for enough parity data to handle the loss of any one drive)?

    If I create a pool of 2 drives, and create a parity space in it, and later add another drive, will Storage Spaces reorganize the data to the more efficient 3 drive organization?

  143. AlgorithmsAreCool says:

    @Rajeev Nagar

    I understand the thought process for the server version, but have you considered simply removing the logical size from the UI for the consumer version. I agree with some of the above sentiments that it is unnecessary since you can always just raise it later. Consider this:

    The Pool displays the aggregate size of the physical pool as the logical size. Whenever  the capacity becomes low it prompts the user for more disks to add. When the user adds a new disk to the pool, the OS automatically increases the logical size of the storage space. This is a pretty simple way to automate the process and gives the user less numbers to worry about.

    I can see a bit of complication with having multiple spaces on a single pool as in the example you provided. This obviously makes it confusing to judge how much space is remaining. But I feel that since the size can be increased indefinitely behind the scenes, it makes little sense to give it a upper bound at all. Storage is storage, even as a power user, i don't see much utility in having this limit for the consumer OS.  Besides why make the user adjust the total space unless it actually needs to be adjusted physically. Also it raises a configuration question in my mind. How big to make the space? 1TB? 50TB? 500TB? You yourself state that it has no real bearing on the performance. In effect for most people it's a meaningless number. For a consumer, it really only hurts them if they pick too low then have to adjust later, and provides no benefit of they pick too high. Why make them pick at all if they can only worsen their situation? As a personal thought, i would bother me a bit seeing the 50TB or whatever size being displayed knowing i didn't care about it. I just want a place to put my media. All of it.

    In the server world i could see some use in limiting the size of a pool. For example you have 3 spaces sharing a pool, each with different permissions, configurations and users. You don't want one space eating up more than it's fair share. I am a bit disappointed by the lack of configurability, i hope the server version will have more knobs to adjust. I could see some benefit to altering the slab size based on the type of backing store. I would also like to know if there will be any choice in checksum/hash algorithms for performance/integrity trading. But i assume this will be answered later in some more formal documentation.

    Beyond that this is still a killer feature and i am pleased to see it. Thank you for your hard work and i look forward to the beta.

  144. Spoonman says:

    I understand that a simple space does not have data resiliency — should a drive fail in the pool the space's data is likely gone.

    However, what about the case where a drive doesn't fail, but it is desired to replace a disk in the pool? Is it possible to tell the system to migrate all of the simple space's data off a particular disk so that the disk can be removed and replaced? Then, after replacement, allow the space to use the new disk again?

    Sometimes I just don't need data resiliency but it'd be nice to use extra pool storage for data without loosing that data when upgrading the pool's capacity.

  145. @theyarecomingforyou – over time, we expect that our partners (hardware vendors) will continue to deliver innovative solutions/packaging including perhaps like you suggest. The logical size associated with a space is the maximum size the space will grow to assuming availability of sufficient physical capacity.

    @ Computermensch – thank you for the comments. Stay tuned for more information on the work done in Windows 8 to protect your data from loss (i.e. “backup”). Pools can only be comprised of physical disks connected through SATA, SAS, or USB. Anti-virus applications operate at the file system level and are, therefore, abstracted from the underlying storage.

    @Alistair, @daviesow, and many others – thank you for the thoughtful comments on thin provisioning and logical size of a space. Some context:

    Thin provisioning enables better utilization of available capacity across multiple "units of provisioning" i.e. spaces. It is optional i.e. users/applications can choose “fixed-provisioned” spaces. Thin provisioning typically delivers value because:

    (a) most applications either do not use all allocated capacity

    (b) during periods of intense activity, applications occasionally use more storage than is "normal". However, this capacity is not needed all the time and can subsequently be relinquished thereby making it available to other spaces/users/applications

    (c) applications/users grow to use maximum projected capacity only over a long period of time. In such cases, storage purchases can be deferred until only when needed. History teaches us that commodity storage pricing decreases in real terms over time so deferring purchases helps lower costs

    By associating a logical size with a space, the user can both specify the maximum physical capacity that will be used by the space and also enable legacy (i.e. existing/old) and new applications to execute without worrying about deployment limitations. Admittedly, this is a rather technical capability and does impose a slightly higher cognitive load on the average user. It is quite possible that many users will simply continue to use single disks as they do today and do not need the capabilities offered through spaces … and, therefore, do not need to worry about thin provisioning. We are happy to take feedback on how this can be further simplified.

    @SA – you can certainly back-up content from a system/boot disk to a space. However, as mentioned, you will need to restore from the space to a designated system/boot disk since Windows 8 does not support booting from a space. Storage Spaces do issue trims to underlying physical drives (including SSDs).

    @kman – yes, resynchronization of data will resume automatically post reboot.

    @Tom – Storage Spaces deliver resiliency to enable applications to “keep running” despite hardware failure. “Backup”, however, provides additional protection such as the ability to recover data that was “accidentally deleted”. Your comments on the need for better support from Windows for protecting user data from loss are spot on. Please stay tuned for a subsequent write-up that describes enhancements in this regard within Windows 8.

    Thank you for the continued support and very thoughtful feedback. More responses later today. Please keep this dialog going.

  146. @AlgorithmsAreCool. At least for me I think folks are thinking a lot about the ability to be unbounded as the key scenario.  But at least for me I think the limits will always be deliberately picked as a consumer and that unbounded growth or planning isn't the consumer norm.  for me, I use an online backup service and have a fixed size so I want to know when I am at that limit too.  I also like to do robocopy backups to a 3TB removeable drive (or largest single drive available) so will likely pick a size that can accommodate that even if I start with some 1tb drives.  

    Unbounded us cool, but not necessarily the only way to think of the feature.  

  147. Matthias Wolf - MVP GPO says:

    Great Stuff!

    But by the way, there is one open issue:

    "For example, you will need four of the six disks comprising the My Home Storage pool to be healthy and physically connected to the PC in order to access either the Documents or the Multimedia space."

    Seems to me, that this is true for the mirroed space.

    The parity space can only absorb one hard drive failure?

    (n-1)

  148. Alvaro says:

    @Rajeev Nagar [MSFT] : Thanks for all  the answers …

    This is one of best post in the blog… with a feature satisfying a great need, easy to understand explanation (and images), a FAQ and positive feedback.

    A  follow up would be great, maybe including some performance measurements/comparisons and a video-clip for those too lazy to read the whole thing …

    kudos

    Alvaro

  149. With all this enormous and various secure storage capabilities provided for in Windows 8 I can't keep from wondering when will Microsoft get rid of the $RECYCLE BIN entity that is automatically created on each volume and under each user account in the amount of 10 % of the disc space. It is not very convenient and sometimes even not possible to edit this feature just to preserve system's memory integrity. I honestly believe that considering all available tools for file and system backups the $recycle bin feature spanning all volumes is utterly deprecated and Microsoft should make it optional and subject to some administrative or group policy control in the least. It should at least be configured to be by default disabled subject to further user optional configuring.

  150. Mike S says:

    @Rajeev, one thing that I still don't follow surrounds the concept of pools.  Why have more than one pool for existence?  Since the parity of mirroring is done at the space level, why not put any and all disks in the same pool?  Is this to separate primarily removable storage from fixed storage, etc….?

    Another question of maximum volume size – if I had a 100 disks of 2 TB each, and created a parity space of 20 TB, how does the issue of having a maximum of 8 columns cause spaces to allocate storage over the pool?  Does this mean the parity has to be distributed over only 8 disks at a time of the 100 for any one  file in the space?

  151. AlgorithmsAreCool says:

    I think it ate my last comment so i'll repost it.

    Thank you Steven for your response I have great faith in you and your development team in this matter. Your use case is quite legitimate and i am inclined to agree with keeping them in the client version of windows at this point. But i still think that my original point stands. That is, for a average folk user the number might not have enough significance to them to keep it there.

    Perhaps a compromise. Offer an option to have the size automatically manged by the machine OR specify a fixed size. Again if you encourage the user, even passively, to just make a space 50TB or greater then why not? I would lean towards having unbounded/manged as being the default but that would probably up to your telemetry. Thin provisioning is very nifty, but if the average joe can be spared the extra management decision why not? (Besides development costs and such of course…)

    In any case thank you, I am not seeking a reply for this comment. Just take it under consideration.

    P.S.

    @chipvlad

    The recycle bin should stay pretty much like it is in my opinion. It only occupies space if it hasn't been emptied. It's a feature and even as a power user, i still use it to some degree. Besides, for the less savvy user it has been there since forever. Why ditch it now? Keep in mind that the "Previous Versions" or shadow copy feature is much less reliable in low diskspace situations since windows will remove older copies behind automatically in when it's under space pressure. By my thinking, it's harmless and still needed (or at least expected) by a large group of less savvy users.

    Lastly, If you don't want to use it right click on it, select Properties. And set it to be "Do not move files to the recycle bin". Not that tricky i think. Also there is already a "Do not move deleted files to the recycle bin" group policy on windows 7 unless i am misunderstanding something.

  152. New Windows 8 demo at CES: same disappointing metro multitasking experience. This may be good for small screens, but this is so frustrating for 17-inch+ screens. Looks like the beta won't get improved on this regard… I just hope the metro windows management will get better with Windows 9. Until then, I'll limit myself to the Desktop…

  153. RainerD says:

    What happens, when I remove a disk, which already contains pool data, from the pool?

    1. The data will be moved to the left disks of the pool, so the disk finally contains an empty volume just like after removing disk from pool supported by WHS/Drive Extender

    2. The data will be copied to the left disks of the pool, so all data stay on the disk and can be accessed by a NTFS 3.1 compatible OS

    The images of your blog suggest, that the process "adding disk to pool" is destructive like the simular process in Drive Extender. More useful would be, that existing data on added disks don't disappear but the process embeds this data into the pool like Drive Extender replacements for WHS 2011 of third party manufacturers (Drive Bender etc.) can do.

    I'm missing an answer of Lionel's question, if hardlinks/symlinks/alternate streams/EFS files/very long paths/other NTFS features are supported in a Storage Space. I am too impatient?

  154. @Microsoft. you guys need to make it EASY to remove packages and features with AIK in Windows 8. every package in the OS needs to be removable….

  155. Chris says:

    any thoughts on using dual parity?

    With 8 drives in a pool – 7 disks worth of data and one disks worth of parity, i would worry about the failure of a second drive during a reconstruction/rebuild

  156. JaviAl says:

    Nices features but not relevant because the ugly user interface Metro that most users dont want and continues with Windows 7. If Microsoft continues with Metro UI, all other features are not relevant because nobody wants to upgrade to Windows 8 if there is no posible to disable Metro UI.

  157. Tristan says:

    Personally, I would like to see these improvements to Storage Spaces.  

    1. For parity protection I would like a box or drop-down to specify my maximum drive failure tolerance (ex: 2 disk failures similar to RAID6).  Drive Spaces would then automatically create the parity necessary to make that level of tolerance possible.

    2.  Include checksums and auto repair for both mirroring and parity as built in features.

    3.  Provide an option to turn off data Striping.  Although performance would be slower, this would be useful for home or small business users who may not have the best backup solutions.  When using parity protection and more than 1 drive fails, the data on the remaining drives would still be intact and recoverable.

    Overall a great advancement for Windows. Keep up the great work.

  158. @Rajeev Nagar [MSFT]

    Thank you for your answers so far 🙂

    I understand, that resiliency is not meant to replace Backup.

    Getting back to your example use case of a large media library with parity protection:

    Depending on the size of the library, backup may be exceedingly expensive.

    Also, it being a media library and not a store for vital documents, backup would not be necessary.

    Instead, the resiliency features coupled with periodical scans for inconsistencies

    AS WELL AS automatic correction of these inconsistencies would be protection enough.

    If I understand correctly, this behaviour cannot be achieved out of the box, though.

    The periodical scrubs you mentioned can not be used to repair damaged files, only to detect the damage.

    In the example of a parity space the system would not be able to tell, which device contains the damaged information, only that there is such damage.

    Please correct me, if I should be under a misapprehension here.

    If I got this right, though, I would really like having the OPTION of taking checksums in order to enable automatic corrections.

  159. Tom Servo says:

    @Rajeev Can spaces be made resilient after the fact? I.e. turning an unresilient one into a mirror?

    I still think it'd be better to be able to do drive replacements on the pool level. The disk would theoretically only need to have the same amount of blocks. The pool could then just move the slabs over as is, allowing for replacements independent of any layouts on top of it, and when done release the disk to be removed from the pool.

  160. Tom Servo says:

    To be clear, my current way to replace disks is to put the new one in, copy data over, move driver letters, then take the old one out. There's a timespan where both the replacement and the "replacee" are in system. With something like Storage Spaces, I actually kind of expect similar functionality. I put the new disk in, I tell Storage Spaces to migrate the data from the old to the new disk, and when it's done, it'll drop the old one from the pool, at which point I can remove it.

    If I'm getting enticement to use SS, I'd like to use it for absolutely everything (minus the boot drive). The idea that an unresilient space (scratch disks for video apps for instance) pretty much trashes any flexibility in device upgrades kind of sours all of it.

  161. Ryan says:

    @Steven Sinofsky, et al: Don't worry about what the consumer norm is today–you have the technology to change the consumer norm. Don't treat that lightly! 🙂 You have a huge potential hit on your hands. What consumer wouldn't want an "Infinite Drive"? That is something just about every consumer can easily, instantly appreciate. All they really need to worry about is the current space available before they need to add another drive. Unfortunately, as I see it currenlty, if my mom bought a Windows 8 machine, she probably wouldn't even know this feature existed, but even if she did, I'd still have to set it up for her.

    Thin provisioning, spaces, pools, parity vs. mirroring–these are storage industry terms and concepts that are relevant and make a lot of sense for a storage administrator, but average consumers? Even as simplified as you've made some of the setup (compared with typical storage solutions), it would be a shame not to get as high an uptake as you could because many average consumers would not grasp the concepts.

    As someone else suggested, I'd also suggest marketing it as "Infinite Drive" or something similar, and then simplify the UX and terminology for the home user. Better yet, perhaps partner for launch with manufacturers like Western Digital, Seagate, etc. to make enclosures with open bays that can be filled as needed. Make it just about a one-step process–plug in the enclosure, then ask to click OK if they want to incorporate the new drives into their Infinite Drive (or click Advanced). I'd suggest simply to make it all parity for average users–the reason most people want "infinite storage" is for pictures, music, and video, and even if they put Word, etc. files there, the perf "hit" (vs. mirroring) should be barely noticeable on a file of that size.

    You have a big, big, big potential winner on your hands, and I think it really all depends on marketing and UX at this point–not the technology, which is already awesome. Put on a UX that masks the underlying power for the average user, and you will have a big hit.

  162. I am glad that Drive Extender has been brought back to Windows.  It WAS the killer feature of WHS, IMO.

  163. Sort of off-topic but since the answer remains elusive……There is only one question I would like answered(assuming this post like others doesn't disappear)  and its the one Microsoft are doing their best to avoid and that is if the metro UI/Start screen and be turned off. All these advancements under the hood are great but its no good building a great motor hen trying to run with square wheels. I will reserve my final judgement until I see the beta but based on current indicators I can see me giving Win 8 on a PC a miss.

  164. GregH says:

    Am interested to hear what backup options there will be in Windows 8 – we need something better!

  165. @grollinger – a periodic background scrubber task will help identify and correct latent corruption. We do try to carefully control on-disk format/metadata changes – that said, please note that we will continue to evolve on-disk metadata as needed until product release.

    @Carnitron – as noted earlier, Storage Spaces are available in Windows 8 client and server. For Windows 8, you cannot boot from a space. However, you can continue to use a dynamic volume for boot, in Windows 8.

    @Tom Servo – yes, NTFS and the storage optimizer will issue trim requests and thinly provisioned spaces will support capacity reclamation.

    @cameron_eldridge – Storage Spaces are a platform technology and 3rd party products can certainly build-on/extend the core functionality. As you are aware, the Windows I/O stack enables layering and easy extensibility.

    @Carsten – write requests from the file system result in Storage Spaces allocating capacity (if not previously allocated). Subsequent trim commands from the file system, the storage optimizer, or from an application will result in reclamation of capacity.

    @jm – you can certainly remove a disk from a pool. Once you disconnect a disk, format it on some other machine and reconnect it, Storage Spaces will not use the disk as part of any pool. Obviously any data on that disk will be lost although Storage Spaces will still provide access to data contained in resilient spaces.

    @NH – for parity spaces, you need a minimum of 3 physical disks within the pool. For your 3 drive example, the space can continue to grow if you add more disks later. Whenever you add physical disks to the pool, Storage Spaces uses the additional capacity efficiently for subsequent allocations.

    @Spoonman – for simple spaces, we do not currently expose the functionality you suggest.

    @Matthias Wolf – you are correct that a parity space can tolerate failure of a single disk backing the space. It is not necessary that a parity space use all physical disks comprising the pool although Storage Spaces will try to maximize all available drives to deliver optimal performance and capacity allocation.

    @Mike S – pools are units of administration enabling isolation and delegation of administrative privileges. Some sophisticated server deployments (e.g. hosting companies or datacenter deployments supporting multiple departments) could decide to use multiple pools. In your example, Storage Spaces will allocate slabs in multiple of “n” (up to 8) columns across all available disks although you could constrain the parity space to a subset of the disks within the pool.

    Thank you.

  166. jm says:

    @Rajeev Nagar Thank you for answer

    I really would like to see some technical explanation how pools are created:)

    Does pool is identified by some kind of GUUID so when I attach a HDD to PC, windows checks the HDD, read GUUID from partition and says "I have another HDD with same pool's GUUID – they should work together – check if we have enough discs to activate space"?

    Are HDDs remembered in pool (by discs' serial numbers for example?) or are identified only by partition info (pools' identifiers)? That's why I've asked what happened when disc is detached from pool and formated.

    And storage space user question – let's say I have pool on 4 HDDs and three spaces within pool – two with parity and one without parity and not mirrored (that's mine possible configuration) – how to exchange one of HDDs for a bigger one?

    If all spaces are mirrored than that's not a problem – attach a new drive, take out an old one, wait for resync. Am I correct?

    But if one space is not mirrored (nor parity), I see that I have to create temporary new space (mirrored or parity), copy not mirrored space to new one, delete not mirrored space, remove old HDD, wait for resync, create not mirrored space, copy previously created mirrored space to a new one not mirrored and finally delete the temporary mirrored space. Well, looks horrible – is there an another way?

  167. jm says:

    Oh, excuse me posting second time but sth came to my mind:

    If all storage spaces live within the same storage pool – what happen if I move huge amount of data between such spaces? Are data moved physically on discs or only file system entries are moved between spaces (like moving files on same hdd currently)?

  168. Sid says:

    I used to have a Windows 2003 server software RAID-5 configuration set up on 3 dynamic hard disks. Once, following a power cord unplug, the system rebooted and re-synced the drives, unfortunately overwriting my data with trash. I was unable to recover my data but for one hard disk (i.e. RAID was mounted, no logical disk was visible by Windows, just some stripes contained data in a good state: photos were readeable just partially, appearing corrupted). I did not find any information about this behaviour on the internet, but maybe this can be the case of a "RAID 5 write hole". Can this be the case? Is there any chance that this can happen with the new Windows 8 storage spaces feature? Are other kind of failure possible? (I know, this is OT, but just in case: If the write hole is the possible reason for the problem I experienced, is there a way to recover my data?) – Thanks

  169. Feels like a really good feature, looking forward to testing it irl:)

  170. Matt says:

    If Windows 7 will not recognize a storage space disk, I think it would advice to format the drive.

    Wouldn't it be better to update Win7 that a storage space disk would not get this message? Otherwise non expert users could accidentally format storage space disks.

  171. Regarding USV: User Space Virtualization

    One more thing that I think is important to remember. As I wrote previously – I moved all my files into the user space … like a pseudo-contruction (static capacity – simply just oversized) of this dynamic Windows Storage Spaces that promises to always supply or provision an actual file system with enough capacity.

    However, when I did that – as I also run a test server Windows Server 2008 R2 with Roaming User Profiles and Folder Redirection of the User Space … one important thing happened – that you will have to take care off.

    The Folder Redirection Client, responsible for syncing the offlines flipped out and wants to sync the entire thing.

    So we need improovements for intellimirror.

    With Windows Storage Spaces the consequence will be, that everything goes into the User Space. Because we get virtualization for storage locally, right. No imagine so a PC being joined to a domain in a "private cloud" and starting to sync.

    In that case we don't need to sync i.e. 4 TB onto the PC, right? We can configure only a part of that to be sync be the local FR Client (on the PC)? If we put the files on the server?

    The problem with the FR Client is that it seems to default to sync the entire folder hierarchy. By design it seems to mark everything for sync. The only current way is to disable the FR Client sync everything – or you need to partition your file hierarchy again – exactly what we don't want to and what Windows Storage Spaces helps get rid off. However, if a PC suddenly have to be joined to a domain – then would we have a scenario where the FR Client wants to sync the whole thing into the CSC case?

    What's the solution? Will intellimirroer be better or do we get options to exclude folder hierarchies for intellimirror – or something else? I haven't tested Windows Server 8 yet, but know it's just as new as Windows 8.

    If we still have to use Intellimirror – and now on top of Windows Storage Spaces I think it may be important we get more flexibility to control the behaviour of the FR Client (a nightmare if the CSC cache was to become 4 TB? – unless the CSC cache is contained inside Windows Storage Spaces as welll on the local client.

    Hope you can eloborate on this? Has intellimirror and FR Client been accomodated to Windows Storage Sapces or replaced by new technology as well fitting Windows Storage Spaces?

  172. jm says:

    "In the event that Storage Spaces metadata on a physical disk becomes corrupt (which will be obvious since the disk health will indicate a problem with the physical disk), you can treat the disk just as you would any other failed disk – simply remove it from the pool. If the physical disk is healthy, you can subsequently re-add it to the pool."

    How can I check that physical disc is healthy if I remove it from computer? Can I remove disc from pool without physically disconnect disc from computer? Can I format disc that is already in pool to remove it from pool?

    Sorry for so many questions but this feature looks as nice as dangerous for data…

  173. Thor says:

    This is a very interesting solution, BUT, as long as existing files on drives you want to add to the pool is lost, that is a HUGE limitation. Also what happens if you have 1 disk for parity in a pool and 2 disks fail. Then all your files are lost on any other drives too?..

    I think I'll keep with FlexRaid which handles these things much better, as it doesn't touch existing files for instance.

  174. Sid says:

    Maybe a short comparison sheet between Win8 solution and other ones (i.e. FlexRaid, …) will help making the pros and cons more clear

  175. xpclient says:

    "The control panel does expose all common tasks." It would be even better if the Control Panel exposes all of the tasks related to Storage Spaces so those who don't wish to learn PowerShell won't be limited by what the GUI allows. Please build a GUI who isn't dumbed down. I would even take cmd prompt and VBScripts over the very elaborate and complex PowerShell syntax.

  176. Let's say I want to create a storage pool with 5 identical USB drives on my laptop. Then I create a parity space using all drives. This means that the storage space will function even if I only have 4 drives.

    What is the safe procedure to "unmount" this pool from my system and physically disconnect the disks, while the laptop is running? Because I don't want to have to shutdown the laptop to be able to disconnect the USB disks from the system.

    What is the safe procedure to add the disks back to a running laptop? If I plug the disks one by one, after I plug the 3rd disk, is the pool "mounted" automatically as it reached the minimal quorum? If so, what happens to the parity space that still needs at least one more disk to be mounted?

    What about when I add the 4th disk? Now the space has the minimum required number of disks to function. Will the storage space be mounted automatically at this time? If so, what happens when I add the 5th and final disk? Will it need to resync? Because that will be very annoying.

  177. Martin says:

    Looks interesting but I note that the management interface is not a Metro UI. Looks like Microsoft is committed to Windows having a split personality – some features will always require jumping out of Metro and into the desktop.

  178. sean.e says:

    @Martin

    Of course they are committed to split personality.  Metro apps can not replace all desktop apps (that's the advice given at Build in September).  Why is that surprising?