June 3, 2013

Utility HPC: What Every Enterprise Can Learn from Utility Supercomputing

Jason Stowe

By Jason Stowe, CEO

I’m struck by the number of IT professionals I speak with that find unexpected uses for Utility Supercomputing. In fact, whether your compute demands warrant 50 cores of HPC, or 50,000 cores, the management challenges are quite similar. Cycle Computing is probably best known for the dramatic results our software has made possible in such compute-intensive applications as Computational Chemistry, Genome Sequencing, and Risk Analytics. For example, a Top 10 Pharma customer of ours built a 10,600 instance cluster – the equivalent of $44M in infrastructure – in just 2 hours, then used it to run 40 years of science in 11 hours for $4,372.

But many of our customers use our Utility HPC software to orchestrate workloads on clusters in the range of 10s – 100s of cores – most of which are hybrid clusters (private and public) at that. It turns out that the fact that we know how to prepare a cluster of 10,000 cores means we can actually create a small cluster that much more reliably.

So what lessons can any Enterprise IT organization learn from the experiences of their larger compute-consuming colleagues? 

1. Know thy workload to optimize costs.

Regardless of how large (or small) scale your compute requirements are, you must understand how parallelizable the applications you’ll be running on Cloud HPC resources are before you can  optimize the size and duration of the instances you procure to support them. For example, the more modular the job requests, the easier it is to parse them across low-cost spot instances of compute when they are available.

2. Play nice with others (Openness matters).

Proprietary approaches to orchestrating workloads across private and public HPC environments should be avoided in any enterprise. When looking for ways to automate and administer your HPC resources, why restrict your options in the areas of configuration or storage management, infrastructure and development tools? Even if you don’t anticipate your environment changing, it’s better to select an orchestration solution that is open and works with a wide range of supporting tools, such as Open Grid Scheduler, HTCondor and Chef, and works with applications developed in a variety of languages. Support for multiple cloud providers is also an important consideration.

3. Wherefore art thou, Data?

You don’t have to have “Big Data” requirements, to make data awareness an important part of your HPC orchestration strategy. Applications don’t live by compute alone – they inevitably require input and generate output, regardless of how large or small. Many organizations face a cost / latency tradeoff when it comes to data access: save money with lower cost, “cold storage” options such as Amazon’s Glacier which require users to wait days to restore archived data if and when they need it. Or store data in more expensive, readily available EC2 storage. It doesn’t have to be an all or nothing scenario however, if you have the ability to centrally orchestrate data movement and access as applications demand it. Another Top 10 Pharma customer is deploying Cycle’s DataManager to automate archival and retrieval of 75TB data sets to and from Amazon Glacier, without the need to re-code or disrupt their application’s workflow.

4. Manage from a central vantage point.

Not unlike an air traffic controller, Enterprise IT needs a 360⁰ view of the HPC resources at their disposal. The lack of visibility into what’s actually running where – whether on internal or external HPC clusters – forces many organizations to overprovision additional cores to handle unexpected peak loads. When organizations can manage all of their workloads from a central management console, they can readily harvest unused resources when needed. Additional resources can be cost-effectively provisioned on-demand, as required to support larger runs such as quarterly regulatory reporting risk analysis.

5. Security, Security, Security!

Security concerns are one of the most cited reasons organizations use to restrict which workloads they run in the cloud. The reality is that in fact, centrally orchestrated HPC environments afford greater control over how encryption is handled. User authentication can likewise be centrally governed and leverage existing LDAP or other established protocols to enforce consistent secure access to applications and data on-premise in the cloud. License keys are also centrally monitored so organizations can easily distinguish cloud-based access to licensed applications. This is a key benefit for IT organizations and ISVs looking to cloud-enable custom developed software without re-architecting their license key management approach. 

Applying these lessons proven successful in our real customer workloads, organizations can accelerate business analytic, product development, and scientific processes. Truly breaking through with utility HPC. For more information, please visit www.cyclecomputing.com or email info@cyclecomputing.com