Over the past few weeks, I had an opportunity to work with colleague Denny Lee and David P Mariani (VP of Engineering, Klout) on a very interesting Big Data story about how Klout.com takes advantage of Analysis Services. Klout analyzes massive amounts of social network user data to measure the impact of opinions, links, and recommendations and identify influential individuals on the social web. Their data warehouse holds more than 800 terabytes of data. The signal collectors score hundreds of millions of profiles and process over 12 billion data points every day—and with Analysis Services Klout achieves query response times of less than 10 seconds on 1 trillion rows of data on a single server with 24 cores and 128 GB of memory.
It’s very clever how Klout connected a multidimensional Analysis Services database to Hadoop/Hive. The clear idea is to use a linked server and distributed queries via the relational SQL Server engine. It helps Klout to reduce data latencies, eliminate maintenance overhead and costs, move aggregation processing to Hadoop, and shorten development cycles. If you are searching for an efficient way to get similar benefits in your Big Data Analytics solution, look no further, read the white pape SQL Server Analysis Services to Hive – A Klout Case Study at http://msdn.microsoft.com/en-us/library/jj710329.aspx. I hope you’ll enjoy it.