Accidentally deleted all my production data in Azure Table Storage!!! What now?

  • Article

So you moved to the Cloud and you fully understand Azure Storage and all the greatness that comes with the various replication options i.e. Locally redundant storage (LRS), Zone-redundant storage (ZRS), Geo-redundant storage (GRS).   All these replication features are designed for transient hardware failures, network or power outages, massive natural disasters, and so on.   But, none of these are intended to help with mass or accidental delete operations.

I hope that last comment makes you think for a second.  I have seen many very large Enterprise customers be impacted by accidental delete operations.  Unfortunately, none of the replication options are going to help you with this.  Also, Microsoft has no magic tools to recover such data, as data deleted from a storage account is purged immediately, whether it’s Blobs, Files or Tables.

Where does this leave us?

All of us are familiar with traditional replication and backup features for VMs, SQL, files etc.  For Cloud solutions I think there are many variables that are super easy to overlook as we have many ways to approach the design.  Do we use Azure Tables, CosmosDB, MonogoDB, Service Bus, the list goes on and on.

When it comes to Enterprise applications I am a huge fan of defining solutions, services and data using a tiered model.  Using a tiered model to define the criticality of each component helps determine which dependencies are business critical and which parts are not on the critical path.  I have seen folks use a 2 tiered model and some use a 4 tiered model.  At a minimum you should use 2 tiers.  Here are some short examples of how you might categorize components of your solution as Tier 1 if it does down.

Tier 1 Data or Services

  • Widespread business stoppage with major financial impact
  • Company-wide productivity compromised
  • Risk of human health/environment
  • List can go on and on…

The point is that anything in Tier 1 is deemed as “business critical” so reliability, business continuity, RTO, RPO are very important.

In my case, the data that was deleted in the Azure Table Storage was considered “business critical” but the data was never defined as Tier 1 data!  If it were defined as Tier 1 it would have forced the team to evaluate their options for various failure scenarios and they would have put a plan in-place for backup and restoring of such important Azure Table data.

What options do we have for Azure Table Backup and Restore?

There are many ways this can be approached so don’t think that the approaches I mention here are going to always be perfect for you.  You need to evaluate the situation and determine how you want to approach it.

AZCopy

AZCopy now has support for various copy operations to and from Azure Storage, include Azure Tables.  The following article explains all the various options and features that are exposed, I highly recommend reading it.  You can also download the latest version of AZCopy from the important references section of this post.

Azure PowerShell

Yes, it’s possible to do this via PowerShell if you want to explore this option would recommend you read the following article which was written before AZCopy had support for Azure Tables.

3rd Party Tools

There are several 3rd party tools that cover this space as well.  I will list a few of them here for reference

Cerebrata – Azure Management Studio

Eastfivecom – Automated backup of Azure Storage Tables and Blobs

 

Recommendations

Leverage AZCopy in conjunction with Azure Automation and/or Azure Scheduler.   If your business needs are more complex you may need to consider a more comprehensive approach that requires use of various APIs, tools or services.

Keep in mind that there are many ways to approach the triggering of the backup or restore.  You could leverage a logic app that triggers the backup or a service that polls the table storage then triggers the backup.  Or simply schedule the backup at a specific interval.

Important References:

Great blog post on Application Criticality levels

Download the latest version of AZCopy

Example of Automated backup of Table and Blob Storage