Expanding Resources with Rapid HPC Orchestration in the Cloud

By Brock Taylor, Director of HPC Solutions at Intel

June 23, 2020

The industry has reached an interesting point for high performance computing in the cloud. You could almost measure this solely by the growing presence of cloud providers at the two major HPC conferences over the past few years. All the major cloud vendors had large, prime real estate on the exposition floor, and many other service providers were spread through kiosks and booths. HPC in the cloud is now one of the hottest topics at these shows, and there is plenty of interest with companies and research institutions eager to figure out how to expand their resource capabilities in fiscally responsible ways.

HPC in the cloud is not new. It’s been building and gaining momentum since the early days that cloud computing emerged, but the past few years have seen dramatic changes and improvements in infrastructure that puts HPC in reach of a much broader audience. You find more specialized instances that provide bigger processing power and memory bandwidth for computationally intensive applications. There’s also more focus on providing higher performing fabric for scaling out workloads in the cloud. Emergence of more HPC-friendly system orchestration and software services also have made leaps and bounds to help “assemble” cluster instances in the cloud. It’s been great progress and really interesting to be part of helping this growth.

That said, migrating HPC applications to the cloud can still be an easier said than done activity even with the advancements in cloud infrastructure. There’s still a learning curve involved to figure out the interface and mechanisms to spin up an instance for a particular cloud provider. If you want or need to have multiple cloud providers, then you have multiple paths to learn. There are many service providers and companies that help address the learning curve and simplify landing workloads in the cloud, but even they face the same issue. At some point in the workflow there is a need to have a common interface to launch a workload with a back-end interface to the cloud provider of choice.

Rapid HPC Orchestration (RHOC)

Recently, Intel and Google started collaborating on creating some tooling that would provide a common interface. That resulted in Intel launching an open source project called Rapid HPC Orchestration in the Cloud or RHOC. Why spin up this effort? As alluded to above, we look at RHOC as an enabler to help users tap into HPC in the cloud. It’s the enzyme that provides an accelerated path for users to realize the collective value of Intel and our cloud partners. Talking a little bit about how RHOC works will help explain the value.

Not trying to reinvent the wheel, RHOC utilizes two common utilities from Hashicorp as the underlying mechanisms to provide multi-cloud support for spinning up instances. Using Terraform’s infrastructure as code approach and Packer’s mechanisms to create cloud images, RHOC uses templates to define connections to specific instances within a cloud vendor and uses templates that define the image to spin up in that vendor.

Users launch jobs direct from the command line and specify which templates to use as well as provide their user account credentials for the cloud provider. RHOC then handles setting up a cluster in the cloud provider. RHOC builds the compute image or reuses a prebuilt image, spins up the desired instances in the cloud, configures them as an HPC cluster, and launches the job. It defaults to running a job as an ephemeral cluster and will take care of grabbing your output data and spinning down the instances when a job has completed. It does provide an option to keep the cluster running, though, when it’s needed to keep the instances active. Just remember to shut it down later!

Optimized HPC

This means that through RHOC, Intel and our cloud partners can collaborate on providing templates that define optimized instances for HPC and optimized compute node images for a given cloud provider. For example, Intel collaborates with many of the individual cloud providers to optimize and tune fabric drivers utilized by the Intel MPI library. RHOC gives us the mechanism to provide this collaboration in the template for that given provider. Through RHOC, these optimizations become part of the images that users get when launching their jobs on a supported cloud. This is collective goodness that helps streamline access to expanded HPC resources.

Ilias Kastardis, HPC Solution Lead at Google, who contributes to the collaborative efforts, commented, “Google Cloud is looking forward to providing RHOC to our HPC customers, as it will give them a simplified way to set up and manage performance optimized HPC clusters. We believe it will be a valuable toolkit for our customers in running their HPC workloads.”

Google is the first provider included in RHOC, but more are coming. We are also expanding our partnerships to help enrich the feature set of the tool and continue to push out new optimizations within templates.

For a look at RHOC, you can check out this quick demonstration at the Intel® HPC +AI Pavilion, and if you are figuring out how to simplify your experience of running HPC in the cloud, watch this space!

Return to Solution Channel Homepage

Intel’s Vision for Exascale Computing

Follow @IntelHPC

Interactive Demos

Podcasts

Intel Resources

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Q&A with Altair CEO James Scapa, an HPCwire Person to Watch in 2021

May 14, 2021

Chairman, CEO and co-founder of Altair James R. Scapa closed several acquisitions for the company in 2020, including the purchase and integration of Univa and Ellexus. Scapa founded Altair more than 35 years ago with two Read more…

HLRS HPC Helps to Model Muscle Movements

May 13, 2021

The growing scale of HPC is allowing simulation of more and more complex systems at greater detail than ever before, particularly in the biological research spheres. Now, researchers at the University of Stuttgart are le Read more…

Behind the Met Office’s Procurement of a Billion-Dollar Microsoft System

May 13, 2021

The UK’s national weather service, the Met Office, caused shockwaves of curiosity a few weeks ago when it formally announced that its forthcoming billion-dollar supercomputer – expected to be the most powerful weather and climate-focused supercomputer in the world when it launches in 2022... Read more…

AMD, GlobalFoundries Commit to $1.6 Billion Wafer Supply Deal

May 13, 2021

AMD plans to purchase $1.6 billion worth of wafers from GlobalFoundries in the 2022 to 2024 timeframe, the chipmaker revealed today (May 13) in an SEC filing. In the face of global semiconductor shortages and record-high demand, AMD is renegotiating its Wafer Supply Agreement and bumping up capacity. Read more…

Hyperion Offers Snapshot of Quantum Computing Market

May 13, 2021

The nascent quantum computer (QC) market will grow 27 percent annually (CAGR) reaching $830 million in 2024 according to an update provided today by analyst firm Hyperion Research at the HPC User Forum being held this we Read more…

AWS Solution Channel

Numerical weather prediction on AWS Graviton2

The Weather Research and Forecasting (WRF) model is a numerical weather prediction (NWP) system designed to serve both atmospheric research and operational forecasting needs. Read more…

Hyperion: HPC Server Market Ekes 1 Percent Gain in 2020, Storage Poised for ‘Tipping Point’

May 12, 2021

The HPC User Forum meeting taking place virtually this week (May 11-13) kicked off with Hyperion Research’s market update, covering the 2020 period. Although the HPC server market had been facing a 6.7 percent COVID-re Read more…

Behind the Met Office’s Procurement of a Billion-Dollar Microsoft System

May 13, 2021

The UK’s national weather service, the Met Office, caused shockwaves of curiosity a few weeks ago when it formally announced that its forthcoming billion-dollar supercomputer – expected to be the most powerful weather and climate-focused supercomputer in the world when it launches in 2022... Read more…

AMD, GlobalFoundries Commit to $1.6 Billion Wafer Supply Deal

May 13, 2021

AMD plans to purchase $1.6 billion worth of wafers from GlobalFoundries in the 2022 to 2024 timeframe, the chipmaker revealed today (May 13) in an SEC filing. In the face of global semiconductor shortages and record-high demand, AMD is renegotiating its Wafer Supply Agreement and bumping up capacity. Read more…

Hyperion Offers Snapshot of Quantum Computing Market

May 13, 2021

The nascent quantum computer (QC) market will grow 27 percent annually (CAGR) reaching $830 million in 2024 according to an update provided today by analyst fir Read more…

Hyperion: HPC Server Market Ekes 1 Percent Gain in 2020, Storage Poised for ‘Tipping Point’

May 12, 2021

The HPC User Forum meeting taking place virtually this week (May 11-13) kicked off with Hyperion Research’s market update, covering the 2020 period. Although Read more…

IBM Debuts Qiskit Runtime for Quantum Computing; Reports Dramatic Speed-up

May 11, 2021

In conjunction with its virtual Think event, IBM today introduced an enhanced Qiskit Runtime Software for quantum computing, which it says demonstrated 120x spe Read more…

AMD Chipmaker TSMC to Use AMD Chips for Chipmaking

May 8, 2021

TSMC has tapped AMD to support its major manufacturing and R&D workloads. AMD will provide its Epyc Rome 7702P CPUs – with 64 cores operating at a base cl Read more…

Fast Pass Through (Some of) the Quantum Landscape with ORNL’s Raphael Pooser

May 7, 2021

In a rather remarkable way, and despite the frequent hype, the behind-the-scenes work of developing quantum computing has dramatically accelerated in the past f Read more…

IBM Research Debuts 2nm Test Chip with 50 Billion Transistors

May 6, 2021

IBM Research today announced the successful prototyping of the world's first 2 nanometer chip, fabricated with silicon nanosheet technology on a standard 300mm Read more…

AMD Chipmaker TSMC to Use AMD Chips for Chipmaking

May 8, 2021

TSMC has tapped AMD to support its major manufacturing and R&D workloads. AMD will provide its Epyc Rome 7702P CPUs – with 64 cores operating at a base cl Read more…

Intel Launches 10nm ‘Ice Lake’ Datacenter CPU with Up to 40 Cores

April 6, 2021

The wait is over. Today Intel officially launched its 10nm datacenter CPU, the third-generation Intel Xeon Scalable processor, codenamed Ice Lake. With up to 40 Read more…

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from Julialang.org. I Read more…

CERN Is Betting Big on Exascale

April 1, 2021

The European Organization for Nuclear Research (CERN) involves 23 countries, 15,000 researchers, billions of dollars a year, and the biggest machine in the worl Read more…

HPE Launches Storage Line Loaded with IBM’s Spectrum Scale File System

April 6, 2021

HPE today launched a new family of storage solutions bundled with IBM’s Spectrum Scale Erasure Code Edition parallel file system (description below) and featu Read more…

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

Saudi Aramco Unveils Dammam 7, Its New Top Ten Supercomputer

January 21, 2021

By revenue, oil and gas giant Saudi Aramco is one of the largest companies in the world, and it has historically employed commensurate amounts of supercomputing Read more…

Quantum Computer Start-up IonQ Plans IPO via SPAC

March 8, 2021

IonQ, a Maryland-based quantum computing start-up working with ion trap technology, plans to go public via a Special Purpose Acquisition Company (SPAC) merger a Read more…

Leading Solution Providers

Contributors

AMD Launches Epyc ‘Milan’ with 19 SKUs for HPC, Enterprise and Hyperscale

March 15, 2021

At a virtual launch event held today (Monday), AMD revealed its third-generation Epyc “Milan” CPU lineup: a set of 19 SKUs -- including the flagship 64-core, 280-watt 7763 part --  aimed at HPC, enterprise and cloud workloads. Notably, the third-gen Epyc Milan chips achieve 19 percent... Read more…

Can Deep Learning Replace Numerical Weather Prediction?

March 3, 2021

Numerical weather prediction (NWP) is a mainstay of supercomputing. Some of the first applications of the first supercomputers dealt with climate modeling, and Read more…

Livermore’s El Capitan Supercomputer to Debut HPE ‘Rabbit’ Near Node Local Storage

February 18, 2021

A near node local storage innovation called Rabbit factored heavily into Lawrence Livermore National Laboratory’s decision to select Cray’s proposal for its CORAL-2 machine, the lab’s first exascale-class supercomputer, El Capitan. Details of this new storage technology were revealed... Read more…

African Supercomputing Center Inaugurates ‘Toubkal,’ Most Powerful Supercomputer on the Continent

February 25, 2021

Historically, Africa hasn’t exactly been synonymous with supercomputing. There are only a handful of supercomputers on the continent, with few ranking on the Read more…

GTC21: Nvidia Launches cuQuantum; Dips a Toe in Quantum Computing

April 13, 2021

Yesterday Nvidia officially dipped a toe into quantum computing with the launch of cuQuantum SDK, a development platform for simulating quantum circuits on GPU-accelerated systems. As Nvidia CEO Jensen Huang emphasized in his keynote, Nvidia doesn’t plan to build... Read more…

New Deep Learning Algorithm Solves Rubik’s Cube

July 25, 2018

Solving (and attempting to solve) Rubik’s Cube has delighted millions of puzzle lovers since 1974 when the cube was invented by Hungarian sculptor and archite Read more…

The History of Supercomputing vs. COVID-19

March 9, 2021

The COVID-19 pandemic poses a greater challenge to the high-performance computing community than any before. HPCwire's coverage of the supercomputing response t Read more…

Microsoft to Provide World’s Most Powerful Weather & Climate Supercomputer for UK’s Met Office

April 22, 2021

More than 14 months ago, the UK government announced plans to invest £1.2 billion ($1.56 billion) into weather and climate supercomputing, including procuremen Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire