The behavior of Windows Server AppFabric 1.1 memory consumption is puzzling to many people. You may observe that the DistributedCacheService.exe consumes a large amount of memory even when the total size of objects stored in the cache is really small, the memory consumption is creeping up over time and can reach twice as much as the configured host cache size.
The behavior is expected if it is AppFabric 1.1. The memory management has changed from 1.0 to 1.1. AppFabric 1.1 manages the memory as its own pools. It adds the released memory to its own free pools but would not give back to GC. That is the reason why the memory usage on the OS level can be high and keeps climbing.
You can validate no memory leak by pushing additional data into the cache when the memory usage of DistributedCacheService.exe is already high or close to the configured host cache size. If AppFabric is able to accommodate, then you know AppFabric is managing it well.
By default, the amount of memory reserved for the cache on a given server is 50% of the total RAM. The host size in MB is available caching memory on the cache host. This only accounts for items in the cache and not the memory used for the caching service itself. You can configure it lower, but configuring the amount higher is not supported
Additional info on AppFabric 1.1 memory management:
- When you configure a host size, this is the amount of memory used to store user objects. This memory is paged into physical memory gradually (i.e. it doesn’t do it all at bootstrap for performance reasons). So irrespective of write load on the cache and the amount of data in the cache, the entire host size worth of memory is brought into physical memory eventually. So for host size = 16 GB eventually all of it would be in working set even if you are storing only 100 MB worth of objects in the cache.
- For AppFabric Cache 1.1 the overheads are 1:1. Which means with host size = 16 GB the working set of the process may go close to 32 GB. This is the reason we don't recommend configuring the host size greater than half of the RAM.
Thanks for the information. I'm trying to get my head around some of the issues and differences we have measured btwn AppFabric 1.0 and 1.1. Reading above (#1) it appears as if you are saying ALL cache hosts will eventually consume their entire allotment of memory (aka host size). However, I am not seeing that behavior in our test environment. It sort of looks like it is doing that in production, but not quite.
Really what would be great is if someone had a way for us to anticipate what resources AppFabric will require. So, if my cache size is x, my anticipated resource is y.
Reading point #2 above, seems like you are saying (when combining it with point 1) that eventually all memory will be consumed b/c "irrespective of write load on the cache and the amount of data in the cache, the entire host size worth of memory is brought into physical memory". Marrying that with point 2) overhead is 1:1 and further setting your cache size to 50% of physical memory leads me to believe that all memory 50% cache size, 50% overhead, will be consumed, if/when you use all your cache.
In practice, we are seeing limited cache size <100M with a DistributedCacheSerivce.exe >5G. It just perpetually grew to this size, then plateaued. I can't find any explanation for why it stopped at that spot.
What measures can we make or take to anticipate growth and resources necessary? How do we size out an environment to handle a certain load?
Thanks for any thoughts or advice.
Regards.
Hi,
I used Appfabric Cache 1.1 in practical scenario. I need to know what are the resources are used when client call the object stored in the memory of Appfabric server.
When I enable the local cache it takes 44 seconds for ten thousands reads… and it takes 15 minutes when local cache is disable…
Hi Rameshvar,
You should install the latest cumulative update (CU5, KB2932678) on the cache hosts which requires .NET 4.5 and follow the instructions in CU3 article (KB2787717) to enable background GC in the DistributedCacheService.exe.config file. See my another blog article: blogs.msdn.com/…/how-to-update-microsoft-appfabric-1-1-for-windows-server-with-cumulative-update-packages.aspx
Since you use local cache, I want to let you know there is a local cache bug that was first fixed in CU4 on the client side (doesn't really need .NET 4.5).