HPC Application Deployment Scenarios and Redistributable Dependencies

  • Article

I was recently asked about dependency considerations relative to applications that leverage the HPC Server 2008 Scheduler API; i.e. "What binaries must be installed on client workstations in order to run a Scheduler-aware application?"...   This scenario is common with HPC solutions which offer a custom "front-end" application to the end-user and interact programmatically with the "back-end" cluster Scheduler to submit jobs.   This scenario does not apply to the tasks actually running within a back-end cluster job.   Front-end applications typically run on Windows XP or Vista.  Examples range from custom Excel extensions to sophisticated graphics applications.

Anyway, with Windows HPC Server 2008, the question is now rather "tricky"...   Unfortunately, I'm also rather easily tricked.  So, I slept overnight at one of those Holiday Express hotels and awoke with a more enlightened vision.

With Compute Cluster Server (CCS version 1), only 1 API was exposed for Scheduler integration, although you could extend that interface yourself via Web Services, for instance.  With HPC Server 2008 (aka CCS version 2), four Scheduler API's are exposed, namely:

  1. CCP API:  The same v1 API referred to as "CCP API".  This interface is embodied within a .NET assembly (i.e. ccpapi.dll).  The "CCP SDK" provides type information that client applications bind with at development time.
  2. Store API:  This v2-only API accomplishes much the same functionality as the v1 "CCP API" but is indicative of the underlying v2 scheduler architectural enhancements.   Currently, it may be considered the "advanced" API relative to CCP API and may or may not become the defacto API for applications linking directly with API type information.  This interface is embodied within a .NET assembly named storeapi.dll.
  3. SOA WCF Interface:  This "Service" API is new and based upon the Windows Communication Foundation (WCF) platform technology.  Please see a more detailed description further down in this blog.
  4. OGF Service Interface:  This API is a new Web Service interface to the Scheduler and specifically intended for heterogeneous platform interoperability.  Imagine a cluster client (perhaps even another cluster scheduler) submitting a job without regard, or knowledge of, the receiving scheduler's make, model, or even vendor.

[ Just to be concise (or rather pedant), one might also consider the new PowerShell interface along with the legacy VBScript interface as Scheduler API's consumable via client applications.]

With options 1, 2, and 3 a client application deployment will need to account for the installation of binaries that enable connection and interaction with the cluster Scheduler.  This implies installing the Compute Cluster Pack (v1), or the HPC Pack (v2), on each client workstation.  The installation mode would be "Client Management Utilities".  

Note that option 3 is dependent upon the "Microsoft.ComputeCluster.Scheduler.Session" classes published with the new "Store API" along with the "System.ServiceModel" WCF classes published with the .NET Framework.  With WCF, the transport options enable loosely connected scenarios and thus minimal client-side dependencies.  However, the "Session" API published via the "Store API" introduces a client-side dependency thus requiring the "Client Management Utilities".

With option 4, no client application deployment dependencies exist.  That is, the Scheduler API dependency is based upon Web Services type information discovery which occurred either at client application development time or even at client application run-time using Web Service Description Language (WSDL) type discovery mechanisms.

In summary, no client "redistributable" package (e.g. "HPC_redist.exe") is currently available...  although it would be an ideal addition to Microsoft's HPC product.  Tightly coupled Scheduler client application deployment scenarios will require the additional installation of the "Client Management Utilities".  Loosely coupled Scheduler client applications will benefit from no additional deployment dependencies.