A closer look at the DPM 2007 Storage Calculator...

  • Article

*Please note that the storage calculator has been updated since this blog was released and so some of the calculations in the blog may need to be altered. I'll post the updates as soon as I can...

The DPM 2007 Storage Calculator for Exchange Server was released last year and the blog that accompanied the release of the calculator provided a load of information about what the results actually mean but I thought I would attempt to deconstruct some of the calculations to understand a bit better for myself where the results have come from.  For the purposes of this blog I am going to focus on the two main factors that are going to concern most large scale DPM designs; the number of DPM servers that should be deployed and the storage capacity required for each DPM server.  There are numerous other calculations that the calculator produces including the number of storage groups per protection group and RAM and processor core sizing guidelines but I will not go into these here.  I am not going to make any recommendations about how to size DPM either but I hope this article will be of use when you are involved in discussions over designing for DPM...

I'm going to start by running through some of the data that the calculator uses with the name values from the calculator itself:

Total No. of Exchange Servers NumExSvrs This is the number Exchange Servers that DPM will be protecting.
Total Storage per Exchange Server TotalDPMStoragePerExSvr The total storage requirement for DPM calculated by the storage calculator per Exchange Server. After determining the size of the recovery point volume per storage group and the restore volume per storage group this is combined and multiplied by the number of storage groups per Exchange Server.
Maximum Storage Capacity per DPM Server VarMaxStorageOnDPM This is the maximum storage capacity for a DPM server - the current guideline and value used in the storage calculator is 40TB.
Storage Group Size ExSGSize This is the total database disk space required per storage group.  The value is taken from the Exchange 2007 Mailbox Storage Requirements Calculator and is the "Total Database Disk Space Required / Replica" value.  This is actually based on the actual size of a single mailbox database on disk (once content indexing & deleted item retention etc.. have been taken into account) plus a data overhead factor of say 20% which it is recommended to add when calculating storage requirements for the mailbox database LUN's.
Transaction Logs Generated per Storage Group CalcNumExSGTLogs This is the average number of transaction logs per storage group generated per day (Note the average is used here!) and this is used as the basis for understanding what the rate of change of your mailbox database is.
Transaction Log Size VarExLogSize The size of each transaction log. (I am focusing on Exchange 2007 only here so this will be 1MB.)
DPM Storage Buffer DPMStorageSafetyNet This is a built in buffer to the storage estimates to account for sudden surges in backup churn. This is manually added to the calc but the default is 5%.
DPM Retention Range DPMRetentionRange The short term retention range that is the length of time for which data is kept on disk.

Storage Capacity Requirement for each DPM Server

This is the difficult bit but the calculations for storage capacity and server numbers are dependant on each other so might as well start here. The capacity requirement is based on a series of calculations as follows:

    1. Capacity required per DPM Server = Total capacity requirement across all Exchange Servers/No. of DPM Servers
    2. Total capacity requirement across all Exchange Servers = Total DPM capacity required per Exchange Server (TotalDPMStoragePerExSvr) * No. of Exchange Servers (NumExSvrs)
    3. Total DPM capacity required per Exchange Server = DPM capacity required per storage group * No. of storage groups per Exchange Server
    4. DPM Capacity required per storage group = Replica Size capacity requirement per storage group + Recovery Point Volume capacity requirement per storage group

Or to put in a single formula:

Capacity required per DPM Server = (((Replica Size per SG + Recovery Point Volume per SG) * No. of storage groups per Exchange Server) * No. of Exchange Servers)/No. of DPM Servers

So the basis of the capacity calculation is an understanding of the 'Replica Size' requirement and the 'Recovery Point Volume Size' requirement at a single storage group level.  The replica size is the amount of space required for a single mailbox database plus changes. The recovery point volume size is based on the number of transaction logs generated on an Exchange Server and the time for which the logs are kept; the retention range.

...so for replica size; think database, and for recovery point; think transaction logs.

The formula used by the DPM storage calculator to understand the 'Replica Size' is as follows:

Replica Size = ExSGSize*(1+((2 * CalcNumExSGTLogs*varExLogSize)/ExSGSize)/1024)/0.85 * (1 + DPMStorageSafetyNet)

To put it a little more simply:

Replica Size = Single database disk space requirement*(1+((2 * Avg no. of TL's generated per storage group per day in MB)/Single database disk space requirement)/1024)/0.85 + safety buffer

So if we break this out into the different parts as follows:

1+ Testing has shown that VSS requires at least 1GB free space per storage group volume.
2 * Avg no. of TL's generated per storage group per day in MB This is to take into account the fact that the data this is based on is an average so it is important that we cater for days where large numbers of transaction logs are generated.
/1024 To turn the MB value representing transaction log values into a GB value.
/0.85 This value has been calculated by Microsoft following their own testing of the product and is used here to factor in the difference between the changes as represented by number of transaction logs and the differences that will be protected which will actually be changes to database pages.

So in summary we determine the rate of change to the database based upon the number of transaction logs generated per storage group each day.  To account for the difference between database changes in terms of database pages and corresponding disk blocks versus the numbers of transaction logs generated we add approximately 18% (or more accurately divide by 0.85). The replica size is the result of this calculation added to the original database disk space required per storage group value.

(Note that the retention range of data on disk in not included in the calculations.  In other words whether the data is retained for 7 days or 14 days makes no difference to the capacity requirements of the replica volume.)

So to express the formula even more simply:

Replica Size = 1GB + (database changes per day/0.85) + % safety buffer

The formula used by the storage calculator to understand the 'Recovery Point Volume Size' is more simple and is as follows:

Recovery Point Volume Size = 3 * DPMRetentionRange * ((CalcNumExSGTLogs*varExLogSize)/1024) * (1 + DPMStorageSafetyNet) + 1

To put it a little more simply:

Recovery Point Volume Size = 3 * Retention Range * (Avg no. of TL's generated per server per day in MB/1024) + safety buffer + 1

So if we break this out into the different parts as follows:

3 * Retention Range This is not only to take into account the fact that the data this is based on is an average so it is important that we cater for days where large numbers of transaction logs are generated, but also to cater for DPM outages. If we are not truncating transaction logs for multiple days then then will be more logs to synchronise and therefore a larger requirement for capacity on the Recovery Point Volume.
/1024 To turn the MB value representing transaction log values into a GB value.
+1 Testing has shown that VSS requires at least 1GB free space per storage group volume.

So in summary to determine the size of the recovery point volume you take the retention range and multiply this number of days by 3 to take into account a tolerance to DPM failure and to factor in days where the number of transaction logs generated is relatively high; multiply this value by the average number of transaction logs generated per storage group and add a safety buffer followed by 1GB.

So to express the formula even more simply:

Recovery Point Volume Size = 3 * Retention Range * Avg no. of TL's generated per server per day in GB + safety buffer + 1GB

Required Number of DPM Servers

The formula used by the storage calculator to understand the number of servers that you require is very simple.  It is as follows:

No. of DPM Servers = MAX(ROUNDUP((NumExSvrs*SG)/250, 0), ROUNDUP(TotalDPMStoragePerExSvr * NumExSvrs/varMaxStorageOnDPM, 0))

In plain text the result is the greater of 2 values:

  1. The total number of storage groups across all mailbox role servers that DPM will be protecting divided by the maximum recommended number of storage groups that a single DPM server should protect (250).
    ((No. of Storage Groups per Exchange Mailbox Role Server * No. of Exchange Servers)/250)
  2. The total storage capacity required to protect all the Exchange data divided by the recommended maximum storage capacity for a single DPM server (40TB).
    (Total Storage Capacity to protect all Exchange Servers/40TB)

The two values that form the basis of this calculation are recommendations from the DPM product group following their own testing of the product.  I believe DPM will scale beyond these guidelines but any design that does exceed these recommended maximums must be validated by appropriate testing for both performance and manageability.

An Example Calculation:

So just to illustrate these calculations I am going to make up some figures and work out the number of DPM Servers that would be required based upon these calculations and the storage capacity per DPM Server.

Example data:

Capacity required per DPM Server = (((Replica Size per SG + Recovery Point Volume per SG) * No. of storage groups per Exchange Server) * No. of Exchange Servers)/No. of DPM Servers
=> (((53GB + 183GB) * 50) * 5)/No. of DPM Servers
=> 59,000GB/No. of DPM Servers

No. of DPM Servers = the greater of...

(No. of Storage Groups per Exchange Mailbox Role Server * No. of Exchange Servers)/250
=> (50 * 5)/250 = 1

AND,

(Total Storage Capacity to protect all Exchange Servers/40TB)
=> (((53GB + 183GB) * 50) * 5)/40TB = 2

No. of DPM Servers = 2

Capacity required per DPM Server = 59,000GB/2 = 29,500GB or 29.5TB

So in this example the requirement is for 2 DPM Servers each with 29.5TB of storage capacity.

Some Conclusions

Consultants and architects designing DPM solutions will use the DPM Storage Calculator to provide a quick and accurate way of calculating their DPM requirements, which is aligned with the recommendations and guidelines from Microsoft and the DPM product group.  I hope this blog makes it a bit easier to understand where these calculations come from.  Of course whichever method you use to size your DPM deployments it is vital that your design is verified with appropriate testing....