The cheapest (and most likely fastest) way to transfer 300GB of data from San Francisco to New York is to FedEx the hard drive…

In his fantastic paper, Jim Gray describes the Distributed Computing Economics (it was published in 2003, if you have not read it, it is a MUST READ). The basic premise is simple (from the paper):


A computation task has four characteristic demands:
• Networking – delivering questions and answers,
• Computation – transforming information to produce new information,
• Database Access – access to reference information needed by the computation,
• Database Storage – long term storage of information (needed for later access).

The ratios among these quantities and their relative costs are pivotal. It is fine to send a GB over the network if it saves years of computation – but it is not economic to send a kilobyte question if the answer could be computed locally in a second.

To make the economics tangible, take the following baseline hardware parameters


2 GHz cpu with 2GB ram (cabinet and networking)                            $2,000
200 GB disk with 100 accesses/ second and 50MB/s transfer             $200
1 Gbps Ethernet port-pair                                                                  $200
1 Mbps WAN link                                                                                 $100/month

From this we conclude that one dollar equates to
= 1 $
≈ 1 GB sent over the WAN
≈ 10 Tops (tera cpu operations)
≈ 8 hours of cpu time
≈ 1 GB disk space
≈ 10 M database accesses
≈ 10 TB of disk bandwidth
≈ 10 TB of LAN bandwidth


The ideal mobile task is stateless (no database or database access), has a tiny network input and output, and has huge computational demand. Seti@Home is a perfect example.

The break-even point is 10,000 instructions per byte of network traffic or about a minute of computation per MB of network traffic.

Conclusion: Put the computation near the data. The recurrent theme of this analysis is that “On Demand” computing is only economical for very cpu-intensive (100,000 instructions per byte or a cpu-day-per gigabyte of network traffic) applications.

[Read the paper for all the details]


Why am I telling you all this and how does it relate to SaaS?

Firstly, as I mentioned earlier it is a great paper and reading it has value per se. Secondly, based on many of my discussions with ISVs, the relative cost of the various resources (CPU, Bandwidth and Storage) hardly ever come up, as if they were all the same, this paper shows the much higher (relative) cost of bandwidth vs CPU and Storage. In addition to that, if you plot Moore’s law, the cost of GB of storage and the cost of GB of bandwidth over time, you will see that the slowest curve to go down is the cost of GB of bandwidth. In other terms, the current trend is making bandwidth even more (relatively to the others) expensive.

Take aways:

(a)  If you are designing a SaaS application, make sure that you are optimizing the bandwidth transfer (over WAN) as it is likely to be the most costly aspect.

(b) Now link that to the increasing debate around SaaS UI (Browser only vs. Smart Client). It seems to me that in addition to give you the possibility to work offline and provide a richer user experience, a smart client could also diminish the bandwidth consumption (highest relative cost) by using the local CPU and storage for some of the tasks… sounds like a good plan to me.

I can already hear many of you say: “what about deployment, what about management, what about updates…”. Yes, I agree these are still aspects where browser-only has an edge. But when the browser vs. smart client choice is discussed among various stakeholders, bandwidth consumption is rarely included, I thought it might be worth throwing it in there. Specially if we are talking massive scale (i.e. millions of users pounding on the SaaS application) transferring GBs of data over WAN lines.

One could argue that AJAX and other newer browser technologies are already optimizing the bandwidth consumption by downloading only part of the page and allow local computation (via client side JavaScript). This is true and a good thing. As it validates the model of local computation utilization, which means that the real issue with Smart Client is not the model per se, it is purely deployment. If the deployment issues were solved Rich Client would be much more pervasive.

Does this make sense?

Comments (4)

  1. MSDNArchive says:

    Your title is catchy because of the migration effort we went through with one of our customers. We had to migrate over 100GB of data to our hosted platform and found that it was indeed faster and cheaper to FedEx a drive.

    Network bandwidth and latency are certainly one of the most difficult aspects of providing hosted SharePoint to enterprises. Unfortunately, document management apps such as SharePoint will always be subject to these problems for as long as files remain relatively large compared to bandwidth or until more efficient protocols can be developed. Fortunately, app acceleration devices such as Certeon help minimize network performance problems, but installing devices at customer locations is a burden.

  2. ciruli says:

    Jim’s paper is definitely a classic, although the landscape has changed some even in the last 3 years.  One thing to bear in mind is that networks keep getting faster.

    See this post for some details:

    The point is not, however, that it’s ok to write software that uses tons and tons of remote data.  However, as networks improve, the scope of problems that can be addressed remotely will continue to grow.

  3. gianpaolo says:

    Yes, network bandwidth is improving and getting cheaper. But the rate at which it is improving is slower than the rate the storage capacity is increasing or even CPU’s computational power is imporving.

    The point I was trying to make is that out of CPU, Storage and Bandwidth (the 3 main components of a network based app) bandwidth is the most costly and the one with the slowest decreasing cost making it (relatively to the 2 others) even more expensive.

  4. johnwest says:

    Dan – I definitely agree that as networks get faster the universe of problems we can feasibly address remotely grows. What I took away from Jim’s article though is that this isn’t economically advantageous until the _costs_ of moving a byte over the network decrease faster than Moore’s law.

    My take on this is that your conclusions are valid if network costs aren’t considered. While in some situations that’s probably an ok assumption to make, if you are analyzing the macro picture costs have to be taken into account.