Cluster Shared Volumes (CSV) Cache is a feature which allows you to allocate system memory (RAM) as a write-through cache. The CSV Cache provides caching of read-only unbuffered I/O. This can improve performance for applications such as Hyper-V, which conducts unbuffered I/O when accessing a VHD or VHDX file. Unbuffered I/O’s are operations which are not cached by the Windows Cache Manager. What CSV Block Cache delivers is caching which can boost the performance of read requests, with write-through for no caching of write requests.
CSV Cache delivers caching at the block level, which enables it to perform caching of pieces of data being accessed within the VHD file. The primary difference from caching solutions in the form of a PCI card with flash which you add to the server, is that CSV Block Cache reserves its cache from system memory. The CSV Cache also tracks VM mobility and invalidates the cache when it moves from host to host, this removes the need to replicate and keep the cache coherent on all nodes in the cluster. This improves efficiency by not having to cache all VMs on all nodes, as well as reduces the performance overhead of pushing the data between nodes.
CSV Cache is completely integrated into the Failover Clustering feature and handles orchestration across the sets of nodes in the cluster.
Deployment Planning
CSV Cache will deliver the most value in scenarios where VMs are heavy read requests, and are less write intensive. Scenarios such as Pooled VDI VMs or also for reducing VM boot storms. Because the applicability of CSV Cache depends on the workload and your specific deployment considerations, it is disabled by default. The customer feedback on CSV Cache has been overwhelmingly positive and we generally recommend turning it on for all scenarios, including both Hyper-V Clusters using CSV and Scale-out File Servers using CSV.
You can allocate up to 80% of the total physical RAM for CSV write-through cache, which will be consumed from non-paged pool memory.
- Hyper-V – Our preliminary testing has found 512 MB to deliver excellent gain at minimal cost, and is the recommend starting point / minimal value if enabled. Then based on your specific deployment and the I/O characteristics of the workloads in the VMs you may wish to increase the amount of memory allocated. Since system memory is a contended resource on a Hyper-V cluster, it is recommended to keep the CSV Cache size moderate. Such as 512 MB, 1 GB, or 2 GB
- Scale-out File Server – It is recommended to allocate a significantly larger CSV Cache on a SoFS as physical memory is typically not a contended resource, you may want to allocate 4 GB, 6 GB, or even more…
There are two configuration settings that allow you to control CSV Cache.
- BlockCacheSize – This is a cluster common property that allows you to define how much memory (in megabytes) you wish to reserve for the CSV Cache on each node in the cluster. If a value of 512 is defined, then 512 MB of system memory will be reserved on each node in the Failover Cluster. Configuring a value of 0 disables CSV Block Cache.
- EnableBlockCache – This is a private property of the cluster Physical Disk resource. It allows you to enable/disable caching on an individual disk. This gives the flexibility to configure cache for read intensive VMs running on some disks, while allowing to disable and prevent random I/O on other disks from purging the cache. For example parent VHD’s with high reads you would enable caching on Disk1, and high writes for differencing disks the CSV cache could be disabled on Disk2. The default setting is 1 for enabled.
Configuring CSV Cache
The CSV Cache is disabled by default, you only need to define how much memory you want to allocate to enable it with the following process:
- Open an elevated Windows PowerShell prompt
- Define the size of the size of the cache to be reserved (example of setting to 1 GB)
(Get-Cluster). BlockCacheSize = 1024
Disabling on a per disk basis
Once the CSV Cache is enabled, all disks on all nodes will be cached by default. You have the flexibility to disable the CSV Cache on an individual disk using the following process:
Get-ClusterSharedVolume “Cluster Disk 1” | Set-ClusterParameter EnableBlockCache 0
Optimizing CSV Cache
The CSV Cache also provides a set of counters you can use to monitor the performance of the cache. You can leverage the Performance Monitor tool (PerfMon.msc) to add the following counts to monitor different aspects of the CSV Cache.
Open Performance Monitor, and under Add Counters you will find “Cluster CSV Volume Cache” with the following counters.
I/O satisfied from cache:
- Cache IO Read-Bytes
- Cache IO Read-Bytes/Sec
- Cache Read
- Cache Read/Sec
I/O satisfied from disk:
- Disk IO Read-Bytes
- Disk IO Read-Bytes/Sec
- Disk Read
- Disk Read/Sec
Total I/O:
- IO Read-Bytes
- IO Read-Bytes/Sec
- IO Read
- IO Read/Sec
Considerations:
The CSV Cache was introduced in Windows Server 2012 and the above applies to all releases following. The CSV Cache has evolved over the releases, and below are a list of considerations with previous releases.
Windows Server 2012 R2:
- Enabling CSV Cache on an individual disk requires that the Physical Disk resource be recycled (taken Offline / Online) for it to take effect. This can be done with no downtime by simply moving ownership of the Physical Disk resource from one node to another.
- Recommended not to exceed allocating 64 GB on Windows Server 2012 R2
- CSV Cache will be disabled on:
- Tiered Storage Space with heat map tracking enabled
- Deduplicated files using in-box Windows Server Data Deduplication feature (Note: Data will instead be cached by the dedup cache)
- ReFS volume when integrity streams is enabled
Windows Server 2012
- Maximum of 20% of the total physical RAM can be allocated for the CSV write-through cache with Windows Server 2012
- The cache size can be modified with no downtime, however for the Hyper-V root memory reserve in the parent partition to be modified to accommodate the memory allocated to the CSV cache it does require a server reboot with Windows Server 2012. To ensure resource contention is avoided, it is recommended to reboot each node in the cluster after modifying the memory allocated to the CSV cache.
- Enabling CSV Cache on an individual disk requires that the Physical Disk resource be recycled (taken Offline / Online) for it to take effect. This can be done with no downtime by simply moving ownership of the Physical Disk resource from one node to another.
- The EnableBlockCache private property is named CsvEnableBlockCache in Windows Server 2012
- The BlockCacheSize common property is named SharedVolumeBlockCacheSizeInMB in Windows Server 2012
- The way it is enabled is also slightly different, here is the process:
- Open an elevated Windows PowerShell prompt
- Define the size of the size of the cache to be reserved (example of setting to 1 GB)
(Get-Cluster). SharedVolumeBlockCacheSizeInMB = 1024
- Enable CSV Cache on an individual disk (must be executed for every disk you wish to enable caching)
Get-ClusterSharedVolume “Cluster Disk 1” | Set-ClusterParameter CsvEnableBlockCache 1
Thanks!
Elden Christensen
Principal PM Manager
Clustering & High-Availability
Microsoft
Many thanks Elden, great post, good explanation
when will you provide caching to flash (ssd or pci)? we are waiting for this in sql, file server,etc..
I've turned this on in my lab environment, and seen great improvement of read performance in our Server 2012 RC Test Cluster.
Now– can you provide a little more details on the resource allocation? I.E. If I were to allocate 20% of system RAM to this, would that cache get trimmed back as additional VMs are started (or ramp up their dynamic memory)?
I have only one file server and doesn't want to use file server cluster. Can I force file server read/write-back cache enabled regardless hyper-v's unbuffered io? otherwise it is so slow.
Is there any reason why this isn't enabled by default?
Are there any drawbacks to CSV cache except the extra memory utilization?
In general we recommend turning on the CSV Cache for any/all Hyper-V or Scale-out File Server deployments.
The only reason it is not enabled by default, is the default amount of memory may vary from deployment to deployment… depending on the amount of physical memory in the nodes, the VM density you are trying to achieve, and for a Hyper-V deployment it may be less and for a Scale-out File Server it will be more. So we just didn't have a good memory allocation number to set by default… but it is something we are definitely thinking about
Great write up and post for Windows Server 2012. This worked great on our Cisco UCS Blade Chassis deployments.
Does anyone have any more info on tuning the size of the cache? I see the perfmon counters, but not really sure how to interpret them as far as tuning. Can't seem to find much in any documentation.
I suggest you do a utility, which analyzes perflog or something, and offers to enable a certain amount of csv cache to each disk.
SAN vendors recommend for tired storage following relation: 5% – 25% – 70% (gold data – silver data – bronze data). This ratio may vary depending on read/writes IO. So, if you have 1 TB data your best size of cache – 50GB. You can use RAM or SSD accelerator card (HP, EMC, NetApp and others vendors have solutions with SSD).
Is there such a thing as having too large a cache? I'm running 4 blades with 384gb of memory each, and even on a busy day all our VM's combined don't use 300gb of ram. So I currently have 200gb of csv cache per host but I'm unsure if that could have possible negative side effects.
QUESTION: Is this feature available to (Windows 7 Ultimate 64-bit) users running a native VHD environment?
My OS is installed to VHDwin7u.vhd. I don't even have a 'recovery' system partition. Not multiboot yet (working on it), designing "differencing disk layers" (conceptually), ie
SOLID (BASE) READ-ONLY (pre-requisite of {grand}parent 🙁 )
window
drivers
————————-
Parent (the 'thin, winfix' profile)
Stuff you shouldn't be without, ie autoruns et al, AHK, MAYBE coolMidi, Everything (winblows search replacement) Sandboxie? …
————————-
DAW Child 1
Barebones standalone profile with any connectivity tools (rtpMidi to iphone…)
POWERuser Child 2
Default profile, connected to web so avasted, defender/wse? along with office et al…
TESTbed Child 3
New apps.
This is a draft off the top of my head. The main challenge I see, given the fact that, once you deploy a child (my understand) parent VHDs MUST be read-only, cannot be modified? … makes planning (super) important.
Is "CSV" another thing for me to consider?!
Thanks
I have multiple CSVs served by SOFS cluster – some are on Tiered storage, some are not. Is the CSV cache turned off only for Tiered storage then?
I plan to have some Tiered storage connected to SOFS cluster over iSCSI from a WS 2012 R2 target. I doubt SOFS recognize it is Tiered storage. What could be the side effects here?
Thanks
CSV Cache has the flexibility to be enabled or disabled on a per disk basis. CSV Cache can satisfy read requests from memory, so it can augment a storage array which supports tiering… by keeping reads from ever hitting the array.
Thanks!
Elden
Could you expand on this?
"Enabling CSV Cache on an individual disk requires that the Physical Disk resource be recycled (taken Offline / Online) for it to take effect. "
What is meant be "individual disk," do you mean CSV volume? What is the clean why in Hyper-V to take an entire CSV offline? That sounds like a horrendous thing to have to do if you have 65 machines in the volume.
This only applies to Win2012, and is why we have improved this experience in Win2012 R2.
But this means toggling on the cache, then moving the CSV volume to another node. The process of failover will take the CSV volume offline and online again, and is transparent to the VMs… they will continue running.
Thanks!
Elden
OK thank you, no big deal there then.
enabled on a hyper-v 2012 r2 cluster, but perf counters show all zero. is there any delay after which the settings are applied, or am i missing something?
Very good article, but how do we coordinate the use of CSV Cache with smart shared storage array caching? For example if we have read/write caching on our intelligent arrays, should we set 100% of the physical controller's memory to write caching thereby benefiting from that, while depending on the CSV block caching from the OS? Also does only the owner of the CSV volume allocate cache? Or does every cluster node regardless of volume ownership?
I see the same as Tym Diekhans, set the csv cache size to 2048 on server 2012 r2, but the performance counters al stay at zero on the nodes.
The performance counters require a reboot after enabling the CSV Cache to become active.
Thanks!
Elden
When I checked the counters, taking the resource offline and online, or moving it, enabled the counters as well…
Hi. I would like to pipe out all volumes where the value is 0 to apply a set-clusterparameter command
I am failing to pipe out using this:
Get-ClusterSharedVolume | Get-ClusterParameter | where {$_.csvenableblockcache -eq “0” } |
Could anyone advise what I need to do to get the where clause to work.
This command successfully out puts all the values without any issues.
Get-ClusterSharedVolume “*(volume*” | Get-ClusterParameter csvenableblockcache
Many Thanks