Azure and High Scale Compute

Azure High Scale Compute (HSC) is a Windows Azure implementation of the Map-Reduce parallel processing architecture. 

In this architecture a controller task (which may be located on a customers premises, in the Azure cloud or distributed between them) is responsible for accepting a work stream, breaking it up into parallel processing units and deploying it to multiple worker nodes in the cloud. It is also responsible for collecting the results of each parallel work stream and putting it all back together again to achieve overall reduced processing time.

The implementation of this process in Windows Azure uses either an on-premises Windows based controller application or an in-cloud Azure Compute node (Web Role) that schedules many Worker nodes by uploading the work streams to Windows Azure Blob Storage and then using Windows Azure Queues to get the work to the multiple worker nodes that poll the queues looking for work to perform.

Recently I had the privilege of helping to architect an Azure HSC application for a major pharmaceutical company. (See here.)

Traditional High Performance Compute (HPC) solutions can be very expensive and time consuming to provision and maintain. Azure HSC eliminated the need to purchase hardware and software for a computational process that has peak demands but does not occur on a continuous basis. 

By allowing them to parallelize long running processes and to dynamically expand and contract the number of application instances required on demand they were able to reduce a process that normally takes hours or days to a matter of minutes.

This makes it possible to perform molecular analyses that previously were either impossible or too time-consuming to perform. The result of these analyses is the development of new compounds that can be used to develop new medications.

This is an outstanding use of Windows Azure, a highly scalable fault tolerant pay for what you use service that is changing the way companies think of solving their business problems.

Bill Zack

Technorati Tags: Windows Azure High Scale Compute,High Performance Computing.