Cache retry fails .. what next ??

When using In-Role Cache or Cache Service applications may get retry’ble error such as below

ErrorCode<ERRCA0017>:SubStatus<ES0002>:There is a temporary failure. Please retry later. (The request did not find the primary.). Additional Information : The client was trying to communicate with the server: net.tcp://<IP>:20004/. ---> Microsoft.ApplicationServer.Caching.DataCacheException:  ……………

.

.

ErrorCode<ERRCA0016>:SubStatus<ES0001>:The connection was terminated_ possibly due to server or network problems or serialized Object size is greater than MaxBufferSize on server. Result of the request is unknown.. Additional Information : The client was trying to communicate with the server: net.tcp://<IP>:20003 ………….

Reasons in general can be in case of High Availability the underlying cache service is load balancing the partitions and the secondary node is transitioning to primary and the client still is sending request to old primary node OR for some reason the cache service got moved to a different VM as part of service healing process but cache client still is having the old IP address of cache service VM.

Though its good to have a retry policy in place but in extreme cases where retry is not helping then you could use below approach in your application to mitigate the errors by refreshing the cache client wen an exception is thrown.

Note : Microsoft Azure Cache Product Group have incorporated the refresh logic in Azure SDK 2.7. Please leverage below code for the applications using Azure SDK 2.6 or below versions. Please find the modified code for Azure SDK 2.7 at the bottom of this article.

Sample code (Azure SDK 2.6 or below version)

Application Code

 try {
 
 DataCacheHelper.DataCache.Get("key");
 
 }
 
 catch (DataCacheException) {
 
 DataCacheHelper.Refresh();
 
 }
 

DataCacheHelper.cs

 using Microsoft.ApplicationServer.Caching;
 
 using System;
 
 using System.Reflection;
 
 namespace DataCacheHelpers {
 
 public static class DataCacheHelper {
 
 private static DataCacheFactory _factory;
 
 private static DataCache _cache;
 
 public static DataCacheFactory DataCacheFactory {
 
 get {
 
 if (_factory == null) {
 
 _factory = new DataCacheFactory();
 
 }
 
 return _factory;
 
 }
 
 }
 
 public static DataCache DataCache {
 
 get {
 
 if (_cache == null) {
 
 _cache = DataCacheFactory.GetDefaultCache();
 
 }
 
 return _cache;
 
 }
 
 }
 
 public static void Refresh() {
 
 var factory = _factory;
 
 if (factory != null) {
 
 factory.Dispose();
 
 _factory = null;
 
 }
 
 _cache = null;
 
 // Clear DataCacheFactory._connectionPool
 
 var coreAssembly = typeof(DataCacheItem).Assembly;
 
 var simpleSendReceiveModulePoolType = coreAssembly.
 
 GetType("Microsoft.ApplicationServer.Caching.SimpleSendReceiveModulePool", throwOnError: true);
 
 var connectionPoolField = typeof(DataCacheFactory).GetField("_connectionPool", BindingFlags.Static | BindingFlags.NonPublic);
 
 connectionPoolField.SetValue(null, Activator.CreateInstance(simpleSendReceiveModulePoolType));
 
 // Clear DistributedCacheSessionStateStoreProvider._staticInternalProvider
 
 var providerType = typeof(Microsoft.Web.DistributedCache.DistributedCacheSessionStateStoreProvider);
 
 var providerField = providerType.GetField("_staticInternalProvider", BindingFlags.Static | BindingFlags.NonPublic);
 
 providerField.SetValue(null, null);
 
 }
 
 }
 
 }
 

Sample code (Azure SDK 2.7 or above version)

Starting Azure SDK 2.7 and above, the core refresh logic remains same, only the re-initialization part changes to use new API instead of reflection. The refresh method now changes as below, rest of the code remains same.

DataCacheHelper.cs

 <snip>public static void Refresh()
 {
    var factory = _factory;     if (factory != null)
      {
         factory.Dispose();
        _factory = null;
        }
    _cache = null;    DataCacheFactory.Reinitialize();
    DistributedCacheSessionStateStoreProvider.Reinitialize();    
   }<snip>