Kubernetes and HPC Applications in Hybrid Cloud Environments – Part II

By Daniel Gruber,Burak Yenier and Wolfgang Gentzsch, UberCloud

March 19, 2020

With the rise of cloud services, CIOs are recognizing that applications, middleware, and infrastructure running in various compute environments need a common management and operating model. Maintaining different application and middleware stacks on-premises and in cloud environments, by possibly using different specialized infrastructure and application management solutions for each cloud provider, adds lots of friction in dynamically allocating, using, and managing those resources.

Lack of a common management and operating model in hybrid cloud environments can cause:

  • Inhomogeneous, fragmented environments create additional complexity for managers, operators, and security.
  • Speed of innovation slows down due to hybrid environments without common management.
  • Cloud resources are hard to change or shutdown when dependent on a cloud provider’s specific services.
  • Workloads can’t be easily migrated back to on-premises environments when bound to specific cloud environment setups, and vice versa.

Kubernetes has become the de-facto standard container orchestrator as pointed out in a previous article. All major companies provide and build solutions on top of a standardized API which is available everywhere. CIOs are now looking into the applicability of Kubernetes for HPC in hybrid-cloud as it offers a common management and operating model for every environment.

Kubernetes: A Common Management and Operating Model for Hybrid Cloud

Kubernetes facilitates the use and administration of countless containers running on fleets of servers. It is the new standard platform for hybrid environments supported by many IT vendors and cloud providers. CIOs can now allocate a fully configured and supported container orchestrator as base for all of their application workloads.

Kubernetes, unlike proprietary infrastructure solutions, provides portability, ease of administration, high availability, integrability, and monitoring capabilities. When managing resources on Kubernetes CIOs are no longer bound to a specific infrastructure. They can offer their users the same set of functionalities, be it on-premises or in any cloud, using the same application stack. Users are not even aware that their applications are running on Kubernetes, nor on which infrastructure they are running: in their own data centers or at a specific cloud provider, like Google, Microsoft, or Amazon.

Reducing complexity in hybrid cloud environments by using a standardized software stack like Kubernetes comes with many advantages: improvements made for one platform can be made automatically available on other platforms; deployment and operational aspects can be simplified; and security audits are easier and rigorously to execute.

Kubernetes and HPC

Kubernetes is the de facto platform for AI and ML already. However, when it comes to traditional HPC, some challenges remain. There is still a set of features built into HPC workload managers not yet available in Kubernetes. We discussed the major differences already previously in our HPCwire Part I article. Major gaps of Kubernetes for HPC currently are: native support for distributed memory jobs, namely MPI applications, and a missing job queueing system compatible with existing HPC applications.

Kubernetes has built-in high availability on many layers. However, for HPC jobs, it is not enough to restart a single container that failed because the whole distributed job itself might have failed already. In this case, automatic rescheduling of the entire distributed memory job is required. This is something Kubernetes doesn’t handle.

Beside these challenges, Kubernetes comes with many benefits for HPC: for example, the environment for the engineer and for the containerized HPC application is always the same, be it on-premises or running in a cloud-based environment; and the capability to quickly change from one infrastructure to another allows the HPC team to align with their company’s cloud roadmap. The freedom to move workloads between infrastructures based on a common API – the Kubernetes API – is what becomes valuable.

Containerized HPC Applications on Kubernetes

Over the past five years, dozens of HPC applications have been containerized, be it commercial, like ANSYS, COMSOL, STAR-CCM+, or open source packages like OpenFOAM and GROMACS, along with HPC cluster schedulers like Univa Grid Engine and Slurm. Thanks to container technology, a constant stream of updates and improvements is provided which can be promptly and seamlessly updated by customers. Additionally, the container images allow users to go back at any time to a previous application version so that they always can reproduce their previous results.

Example HPC Application Cluster Architecture running on Managed Kubernetes.

In the meantime, many container environments have been implemented by using infrastructure and configuration management tools like Terraform and Puppet or by building cloud specific HPC integrations into existing portals. But with the advent of Kubernetes, container environments became easier to maintain and are much more dynamic. Rolling out a cluster, rescaling the worker nodes, using a constant set of preemptible instances, and high availability are driven by controllers which continuously drive the cluster to the desired state. Thus, major HPC gaps of Kubernetes have been closed. This way, today, distributed memory/MPI jobs can be supported in any Kubernetes environment, which provides a built-in HPC workload manager integration running inside HPC containers. That allows traditional HPC applications to run without any changes. Also, GPU and non-GPU enabled applications based on Ansys and COMSOL have been launched successfully, through a high-performance, GPU enabled desktop running inside a pod. Once logged in to the desktop the engineer can start submitting batch jobs or single MPI applications which are distributed across a set of pods allocated on multiple nodes.

Conclusions

Kubernetes not only supports microservice based enterprise applications, but also self-service engineering HPC applications. In summary, as this research has shown, the key advantages of using Kubernetes as a foundation for running containerized engineering applications are:

  • Unified application stack available on virtually any infrastructure
  • True hybrid cloud usage scenarios for engineering workload. For the engineers it is transparent where the application runs, be it on-premises or in the cloud
  • which leads to providing the best performance for running engineering applications by allocating always the newest and fastest machines available in the cloud
  • Building and resizing a self-contained HPC application and compute cluster as self-service for the engineer which is only limited by cloud quotas and budget per time period
  • Robust management stack, supported by many Cloud providers
  • Optimizing costs by only paying for what is used. No idle resources which need to be allocated before they are going to be used.
  • High security through self-contained dedicated compute clusters
  • Minimal operational overhead by self-provisioning and disposable components for which updates are simple destroy and re-create commands
  • Kubernetes based workload is easier to integrate in widely adopted continuous integration and deployment solutions (like Tekton, Concourse, or future versions of Jenkins)

In this research, container-based HPC application environments have been implemented on top of Kubernetes (e.g. on Google GCP and Amazon AWS) and also used as self-service test environments which can be deployed from scratch by HPC application specialists, not operators. It has also been used in CI/CD pipelines to automatically build test environments which run tests against existing container solutions and shut down the infrastructure afterwards. In customer environments, the IT group benefits from an easier to maintain system using a supported, managed Kubernetes which can ramp up, resized and deleted computing resources within minutes.

About the Authors

Daniel Gruber, Burak Yenier, and Wolfgang Gentzsch are with UberCloud, a company that started in 2013 with developing HPC container technology and containerized engineering applications, to facilitate access and use of engineering HPC workload in a shared on-premise or on-demand cloud environment. In this article and the part-one article published on HPCwire last September, they describe their experiences during the last 12 months using UberCloud HPC containers on Kubernetes.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Supercomputer Analysis Shows the Atmospheric Reach of the Tonga Eruption

January 21, 2022

On Saturday, an enormous eruption on the volcanic islands of Hunga Tonga and Hunga Haʻapai shook the Pacific Ocean. The explosion, which could be heard six thousand miles away in Alaska, caused tsunamis across the entir Read more…

NSB Issues US State of Science and Engineering 2022 Report

January 20, 2022

This week the National Science Board released its biannual U.S. State of Science and Engineering 2022 report, as required by the NSF Act. Broadly, the report presents a near-term view of S&E based mostly on 2019 data. To a large extent, this year’s edition echoes trends from the last few reports. The U.S. is still a world leader in R&D spending and S&E education... Read more…

Researchers Achieve 99 Percent Quantum Accuracy with Silicon-Embedded Qubits 

January 20, 2022

Researchers in Australia and the U.S. have made exciting headway in the quantum computing arms race. A multi-institutional team including the University of New South Wales and Sandia National Laboratory announced that th Read more…

Trio of Supercomputers Powers Estimate of Carbon in Earth’s Outer Core

January 20, 2022

Carbon is one of the essential building blocks of life on Earth, and it—along with hydrogen, nitrogen and oxygen—is one of the key elements researchers look for when they search for habitable planets and work to unde Read more…

Multiverse Targets ‘Quantum Computing for the Masses’

January 19, 2022

The race to deliver quantum computing solutions that shield users from the underlying complexity of quantum computing is heating up quickly. One example is Multiverse Computing, a European company, which today launched the second financial services product in its Singularity product group. The new offering, Fair Price, “delivers a higher accuracy in fair price calculations for financial... Read more…

AWS Solution Channel

shutterstock 718231072

Accelerating drug discovery with Amazon EC2 Spot Instances

This post was contributed by Cristian Măgherușan-Stanciu, Sr. Specialist Solution Architect, EC2 Spot, with contributions from Cristian Kniep, Sr. Developer Advocate for HPC and AWS Batch at AWS, Carlos Manzanedo Rueda, Principal Solutions Architect, EC2 Spot at AWS, Ludvig Nordstrom, Principal Solutions Architect at AWS, Vytautas Gapsys, project group leader at the Max Planck Institute for Biophysical Chemistry, and Carsten Kutzner, staff scientist at the Max Planck Institute for Biophysical Chemistry. Read more…

Students at SC21: Out in Front, Alongside and Behind the Scenes

January 19, 2022

The Supercomputing Conference (SC) is one of the biggest international conferences dedicated to high-performance computing, networking, storage and analysis. SC21 was a true ‘hybrid’ conference, with a total of 380 o Read more…

Supercomputer Analysis Shows the Atmospheric Reach of the Tonga Eruption

January 21, 2022

On Saturday, an enormous eruption on the volcanic islands of Hunga Tonga and Hunga Haʻapai shook the Pacific Ocean. The explosion, which could be heard six tho Read more…

NSB Issues US State of Science and Engineering 2022 Report

January 20, 2022

This week the National Science Board released its biannual U.S. State of Science and Engineering 2022 report, as required by the NSF Act. Broadly, the report presents a near-term view of S&E based mostly on 2019 data. To a large extent, this year’s edition echoes trends from the last few reports. The U.S. is still a world leader in R&D spending and S&E education... Read more…

Multiverse Targets ‘Quantum Computing for the Masses’

January 19, 2022

The race to deliver quantum computing solutions that shield users from the underlying complexity of quantum computing is heating up quickly. One example is Multiverse Computing, a European company, which today launched the second financial services product in its Singularity product group. The new offering, Fair Price, “delivers a higher accuracy in fair price calculations for financial... Read more…

Students at SC21: Out in Front, Alongside and Behind the Scenes

January 19, 2022

The Supercomputing Conference (SC) is one of the biggest international conferences dedicated to high-performance computing, networking, storage and analysis. SC Read more…

Q-Ctrl – Tackling Quantum Hardware’s Noise Problems with Software

January 13, 2022

Implementing effective error mitigation and correction is a critical next step in advancing quantum computing. While a lot of attention has been given to effort Read more…

Nvidia Defends Arm Acquisition Deal: a ‘Once-in-a-Generation Opportunity’

January 13, 2022

GPU-maker Nvidia is continuing to try to keep its proposed acquisition of British chip IP vendor Arm Ltd. alive, despite continuing concerns from several governments around the world. In its latest action, Nvidia filed a 29-page response to the U.K. government to point out a list of potential benefits of the proposed $40 billion deal. Read more…

Nvidia Buys HPC Cluster Management Company Bright Computing

January 10, 2022

Graphics chip powerhouse Nvidia today announced that it has acquired HPC cluster management company Bright Computing for an undisclosed sum. Unlike Nvidia’s bid to purchase semiconductor IP company Arm, which has been stymied by regulatory challenges, the Bright deal is a straightforward acquisition that aims to expand... Read more…

SC21 Panel on Programming Models – Tackling Data Movement, DSLs, More

January 6, 2022

How will programming future systems differ from current practice? This is an ever-present question in computing. Yet it has, perhaps, never been more pressing g Read more…

IonQ Is First Quantum Startup to Go Public; Will It be First to Deliver Profits?

November 3, 2021

On October 1 of this year, IonQ became the first pure-play quantum computing start-up to go public. At this writing, the stock (NYSE: IONQ) was around $15 and its market capitalization was roughly $2.89 billion. Co-founder and chief scientist Chris Monroe says it was fun to have a few of the company’s roughly 100 employees travel to New York to ring the opening bell of the New York Stock... Read more…

US Closes in on Exascale: Frontier Installation Is Underway

September 29, 2021

At the Advanced Scientific Computing Advisory Committee (ASCAC) meeting, held by Zoom this week (Sept. 29-30), it was revealed that the Frontier supercomputer is currently being installed at Oak Ridge National Laboratory in Oak Ridge, Tenn. The staff at the Oak Ridge Leadership... Read more…

AMD Launches Milan-X CPU with 3D V-Cache and Multichip Instinct MI200 GPU

November 8, 2021

At a virtual event this morning, AMD CEO Lisa Su unveiled the company’s latest and much-anticipated server products: the new Milan-X CPU, which leverages AMD’s new 3D V-Cache technology; and its new Instinct MI200 GPU, which provides up to 220 compute units across two Infinity Fabric-connected dies, delivering an astounding 47.9 peak double-precision teraflops. “We're in a high-performance computing megacycle, driven by the growing need to deploy additional compute performance... Read more…

Intel Reorgs HPC Group, Creates Two ‘Super Compute’ Groups

October 15, 2021

Following on changes made in June that moved Intel’s HPC unit out of the Data Platform Group and into the newly created Accelerated Computing Systems and Graphics (AXG) business unit, led by Raja Koduri, Intel is making further updates to the HPC group and announcing... Read more…

Nvidia Buys HPC Cluster Management Company Bright Computing

January 10, 2022

Graphics chip powerhouse Nvidia today announced that it has acquired HPC cluster management company Bright Computing for an undisclosed sum. Unlike Nvidia’s bid to purchase semiconductor IP company Arm, which has been stymied by regulatory challenges, the Bright deal is a straightforward acquisition that aims to expand... Read more…

D-Wave Embraces Gate-Based Quantum Computing; Charts Path Forward

October 21, 2021

Earlier this month D-Wave Systems, the quantum computing pioneer that has long championed quantum annealing-based quantum computing (and sometimes taken heat fo Read more…

Killer Instinct: AMD’s Multi-Chip MI200 GPU Readies for a Major Global Debut

October 21, 2021

AMD’s next-generation supercomputer GPU is on its way – and by all appearances, it’s about to make a name for itself. The AMD Radeon Instinct MI200 GPU (a successor to the MI100) will, over the next year, begin to power three massive systems on three continents: the United States’ exascale Frontier system; the European Union’s pre-exascale LUMI system; and Australia’s petascale Setonix system. Read more…

Three Chinese Exascale Systems Detailed at SC21: Two Operational and One Delayed

November 24, 2021

Details about two previously rumored Chinese exascale systems came to light during last week’s SC21 proceedings. Asked about these systems during the Top500 media briefing on Monday, Nov. 15, list author and co-founder Jack Dongarra indicated he was aware of some very impressive results, but withheld comment when asked directly if he had... Read more…

Leading Solution Providers

Contributors

Lessons from LLVM: An SC21 Fireside Chat with Chris Lattner

December 27, 2021

Today, the LLVM compiler infrastructure world is essentially inescapable in HPC. But back in the 2000 timeframe, LLVM (low level virtual machine) was just getting its start as a new way of thinking about how to overcome shortcomings in the Java Virtual Machine. At the time, Chris Lattner was a graduate student of... Read more…

2021 Gordon Bell Prize Goes to Exascale-Powered Quantum Supremacy Challenge

November 18, 2021

Today at the hybrid virtual/in-person SC21 conference, the organizers announced the winners of the 2021 ACM Gordon Bell Prize: a team of Chinese researchers leveraging the new exascale Sunway system to simulate quantum circuits. The Gordon Bell Prize, which comes with an award of $10,000 courtesy of HPC pioneer Gordon Bell, is awarded annually... Read more…

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from Julialang.org. I Read more…

Nvidia Defends Arm Acquisition Deal: a ‘Once-in-a-Generation Opportunity’

January 13, 2022

GPU-maker Nvidia is continuing to try to keep its proposed acquisition of British chip IP vendor Arm Ltd. alive, despite continuing concerns from several governments around the world. In its latest action, Nvidia filed a 29-page response to the U.K. government to point out a list of potential benefits of the proposed $40 billion deal. Read more…

Top500: No Exascale, Fugaku Still Reigns, Polaris Debuts at #12

November 15, 2021

No exascale for you* -- at least, not within the High-Performance Linpack (HPL) territory of the latest Top500 list, issued today from the 33rd annual Supercomputing Conference (SC21), held in-person in St. Louis, Mo., and virtually, from Nov. 14–19. "We were hoping to have the first exascale system on this list but that didn’t happen," said Top500 co-author... Read more…

TACC Unveils Lonestar6 Supercomputer

November 1, 2021

The Texas Advanced Computing Center (TACC) is unveiling its latest supercomputer: Lonestar6, a three peak petaflops Dell system aimed at supporting researchers Read more…

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

Intel Launches 10nm ‘Ice Lake’ Datacenter CPU with Up to 40 Cores

April 6, 2021

The wait is over. Today Intel officially launched its 10nm datacenter CPU, the third-generation Intel Xeon Scalable processor, codenamed Ice Lake. With up to 40 Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire