Preparing for an HPC Multi-Cloud Future

By Andy Morris, IBM Cognitive Infrastructure

September 30, 2019

[Connect with Spectrum users and learn new skills in the IBM Spectrum LSF User Community.]

It’s all but inevitable, but there are concrete steps that will help reduce cost

The use of cloud in HPC is on the rise. While less than 10% of all HPC workloads run in the cloud today, 74% of HPC users already run at least some workloads in the cloud according to research from Hyperion(1). HPC cloud usage is expected to grow from 10% to 15% in just three years, outpacing the HPC market overall. There are both pull and push factors at play, as HPC offerings from major cloud providers including IBM, Amazon, and Microsoft are growing in capability and becoming easier to deploy and manage. Also, customers often struggle with the cost and complexity of HPC infrastructure and often prefer fast access to infrastructure via flexible pay-per-use models.

The rise of multi-cloud deployments

Despite valiant efforts by HPC managers to standardize on a single cloud provider, multi-cloud deployments are on the rise. They arise organically through third-party collaborations, M&A activities, LOBs making independent purchasing decisions, and SaaS, PaaS and managed service provider offerings that have affinities to particular cloud providers.

A recent survey on cloud adoption by Gartner Research revealed that 80% of cloud users already deal with more than one cloud provider(2). In collaborative research and joint engineering environments, data is frequently spread across multiple clouds. For example, firms engaged in life sciences research may source reference genomes from public and private object stores and find themselves shifting workloads because of data gravity and cost considerations. By 2022, 50% of cloud spending decisions are expected to be based on the location of cloud data assets as opposed to cloud product capabilities(2).

[Read also: BlueBee – Speeding clinical results with a cloud-based platform for genomics data analytics]

Navigating complex deployment models

The variety of cloud deployment models is growing. Cloud users face choices about where workloads run, the types of cloud services to consume (IaaS, PaaS, or SaaS), and multiple deployment options including on-premises, dedicated cloud, hybrid cloud, multi-cloud, and even inter-cloud where individual applications or workflows span cloud providers. Multi-cloud deployments have all the complexities of regular HPC but also bring new challenges related to latency, security, managing cross-cloud data transfers, multiple management interfaces, and separate cost accounting.

Cost management is a major challenge

In multi-cloud deployments, cost management is a particular challenge. Gartner estimates that 80% of IaaS users will overshoot their budgets through 2020(3) because they lack necessary cloud-spending process controls. Idle or orphaned instances, improper instance sizing, and unnecessary storage costs all contribute to cost overruns, and all of these issues are harder to manage in multi-cloud environments.

Preparing for a multi-cloud future

Fortunately, there are concrete steps that organizations can take to help manage multi-cloud environments more effectively.

  • Beware of multiple workload managers – HPC workload managers prioritize and schedule workloads, and provide centralized monitoring and reporting on resource consumption by user, group, project, and application. Siloed, cloud-specific workload managers make it much harder to have a consolidated view of workloads and how they consume resources. Ideally organizations should standardize on a single workload manager that can span multiple clouds to centralize policy controls, resource monitoring, and reporting.


  • Cloud agnostic provisioning – There are many ways to deploy cloud-based clusters ranging from cloud-specific tools and CLIs to open-source toolkits for managed service providers that provide their own multi-cloud management tools. Ideally, a single provisioning engine should be used that works across multiple clouds to ensure flexibility and portability. Cloud resources should be scale up and down automatically based on workload management policies


  • Embrace containers – A prerequisite for running across multiple clouds is ensuring that workloads are portable. Container solutions such as Docker and Singularity help applications move easily across clouds by encapsulating application dependencies and avoiding the need to install complex software environments. The keys are to containerize application workloads, using secure registries to serve containers across clouds and to look for workload managers with native support for Docker, Singularity and other container runtimes.


  • Invest in automation and data-aware scheduling – In multi-cloud environments, process automation plays a critical role. Application workflows are often comprised of multiple steps that run locally or in multiple clouds. Administrators need to be able to automate decisions and actions at runtime, such as whether to burst to a particular cloud provider or whether to migrate data between less-expensive storage tiers based on application requirements. Automating these actions requires flexible process management that can orchestrate workloads across clouds and automate decisions at runtime.

For users preparing for a multiple cloud future, IBM Spectrum LSF can help simplify multi-cloud application environments. IBM Spectrum LSF supports dedicated, hybrid cloud and multi-cloud deployments with workload-aware cloud bursting to Amazon EC2, Microsoft Azure, IBM Cloud, Google Compute Cloud, and OpenStack using flexible resource connectors. Spectrum LSF also provides centralized monitoring and reporting and native support for containerized workloads. Process management and data-aware scheduling capabilities can optimize data movement and workload placement across clouds to maximize efficiency and minimize cost.



  1. Hyperion Research Opinion – March 2019 –
  1. Gartner Research May 24, 2019 – Are you Ready for Multicloud and Intercloud Data Management?
  2. How to Identify Solutions for Managing Costs in Public Cloud IaaS –



It’s all but inevitable, but there are concrete steps that will help reduce cost

The use of cloud in HPC is on the rise. Read more…

" share_counter=""]
Return to Solution Channel Homepage

Follow @IBMSystems

IBM Systems on Facebook