The vast majority of enterprises these days are leveraging the agility, cost advantages, and convenience offered by cloud computing.^[1] High performance computing (HPC) facilities are also exploring the benefits of cloud, but perhaps not for the same reasons.

Most organizations, HPC or otherwise, are not forklifting out their entire on-premises IT infrastructures and moving entirely to the cloud. Instead, each company, or university, or research facility is finding ways to supplement or enhance their own data center capabilities with those offered by off-site computing services providers. These new arrangements are called hybrid cloud architectures, and as the number of cloud providers supporting a particular environment proliferates, hybrid cloud becomes multi-cloud. 

Enterprises are looking to the cloud for many reasons. Backup and recovery solutions have proven to be early drivers of cloud adoption. You contract with a cloud services provider for compute resources somewhere else that are a reasonable facsimile of your own data center. Then, when disaster strikes, you fire up these backup resources and keep on computing.

Data storage in the cloud has also been popular. The majority of data that enterprises keep these days is not accessed very frequently, if at all. Once data gets “cold,” the business priorities associated with it shift from performance and accessibility to cost. Cloud storage can be very economical, though not very fast, an ideal solution for this very common IT requirement.

But backup and recovery requirements for most HPC installations are inherently different from more traditional IT environments. And HPC data sets are often so enormous that moving them across networks to take advantage of cloud storage economics becomes less attractive and even prohibitive.

So, do hybrid cloud architectures fail to offer many benefits to HPC users? In fact, there is a rather common HPC challenge that’s ripe for hybrid cloud solutions – the peaks and valleys in demand for HPC resources, based on business/workflow requirements. 

Most HPC facilities serve groups of users potentially working on multiple projects with differing business priorities. Situations occur where multiple users, projects, and applications all compete for resources – more resources than are available. These are instances where the magic of workload schedulers can really be appreciated. But an alternative to debating priorities and ultimately making certain users wait their turns, is to arrange for compute resources from the cloud.

Because of the batch processing nature of most HPC workloads, as opposed to the real-time analytics voraciously consumed by industries such as retail and financial services, leveraging compute resources from off-premises providers can actually become a very viable and attractive option – if you use the right tools.

This is where IBM Spectrum Computing enters your picture. Hybrid cloud may be powerful, but it’s not necessarily simple by nature. Cloud compute resources, unlike cloud storage, can be more expensive than on-premises resources. Security is always an issue. And you need to have apps that are cloud ready.

IBM Spectrum LSF Suites, a member of the IBM Spectrum Computing family of solutions, can help meet the range of needs an HPC environment faces, including hybrid cloud. For users, IBM Spectrum LSF Suites is all about simplification. For the infrastructure, it can help get the most from additional compute capacity available in the cloud during peaks in workloads.

IBM Spectrum LSF includes a capability known as resource connector that enables LSF clusters to borrow resources from supported cloud providers to satisfy pending workloads. The borrowed resources join the LSF cluster as hosts. When the resources become idle, LSF resource connector returns them to the resource provider.

Also, IBM Spectrum LSF automatically pre-stages data to the cloud before hosts are provisioned, significantly accelerating time to results. And LSF actually caches data in the cloud to avoid repeatedly moving the same files.

Governed by limit and growth policies that you set, your cluster dynamically grows and shrinks in response to demand, so you can minimize the use of the expensive cloud resources. This is HPC hybrid cloud made simple and cost effective. 

Hybrid cloud solutions can offer real benefits to HPC facilities, even if not the same ones valued by other types of computing environments. What is the same is how hybrid cloud is quickly dissolving the boundaries between local and remote, limited and limitless, the unimaginable and the very real.

 

[1] Growing up hybrid: Accelerating digital transformation, IBM Center for Applied Insights, 2016 https://www-01.ibm.com/common/ssi/cgi-bin/ssialias?subtype=WH&infotype=SA&htmlfid=GMW14087USEN&attachment=GMW14087USEN.PDF

High performance computing (HPC) facilities are also exploring the benefits of cloud, but perhaps not for the same reasons. Read more…

" share_counter=""]