Kubernetes and HPC Applications in Hybrid Cloud Environments – Part II

By Daniel Gruber,Burak Yenier and Wolfgang Gentzsch, UberCloud

March 19, 2020

With the rise of cloud services, CIOs are recognizing that applications, middleware, and infrastructure running in various compute environments need a common management and operating model. Maintaining different application and middleware stacks on-premises and in cloud environments, by possibly using different specialized infrastructure and application management solutions for each cloud provider, adds lots of friction in dynamically allocating, using, and managing those resources.

Lack of a common management and operating model in hybrid cloud environments can cause:

  • Inhomogeneous, fragmented environments create additional complexity for managers, operators, and security.
  • Speed of innovation slows down due to hybrid environments without common management.
  • Cloud resources are hard to change or shutdown when dependent on a cloud provider’s specific services.
  • Workloads can’t be easily migrated back to on-premises environments when bound to specific cloud environment setups, and vice versa.

Kubernetes has become the de-facto standard container orchestrator as pointed out in a previous article. All major companies provide and build solutions on top of a standardized API which is available everywhere. CIOs are now looking into the applicability of Kubernetes for HPC in hybrid-cloud as it offers a common management and operating model for every environment.

Kubernetes: A Common Management and Operating Model for Hybrid Cloud

Kubernetes facilitates the use and administration of countless containers running on fleets of servers. It is the new standard platform for hybrid environments supported by many IT vendors and cloud providers. CIOs can now allocate a fully configured and supported container orchestrator as base for all of their application workloads.

Kubernetes, unlike proprietary infrastructure solutions, provides portability, ease of administration, high availability, integrability, and monitoring capabilities. When managing resources on Kubernetes CIOs are no longer bound to a specific infrastructure. They can offer their users the same set of functionalities, be it on-premises or in any cloud, using the same application stack. Users are not even aware that their applications are running on Kubernetes, nor on which infrastructure they are running: in their own data centers or at a specific cloud provider, like Google, Microsoft, or Amazon.

Reducing complexity in hybrid cloud environments by using a standardized software stack like Kubernetes comes with many advantages: improvements made for one platform can be made automatically available on other platforms; deployment and operational aspects can be simplified; and security audits are easier and rigorously to execute.

Kubernetes and HPC

Kubernetes is the de facto platform for AI and ML already. However, when it comes to traditional HPC, some challenges remain. There is still a set of features built into HPC workload managers not yet available in Kubernetes. We discussed the major differences already previously in our HPCwire Part I article. Major gaps of Kubernetes for HPC currently are: native support for distributed memory jobs, namely MPI applications, and a missing job queueing system compatible with existing HPC applications.

Kubernetes has built-in high availability on many layers. However, for HPC jobs, it is not enough to restart a single container that failed because the whole distributed job itself might have failed already. In this case, automatic rescheduling of the entire distributed memory job is required. This is something Kubernetes doesn’t handle.

Beside these challenges, Kubernetes comes with many benefits for HPC: for example, the environment for the engineer and for the containerized HPC application is always the same, be it on-premises or running in a cloud-based environment; and the capability to quickly change from one infrastructure to another allows the HPC team to align with their company’s cloud roadmap. The freedom to move workloads between infrastructures based on a common API – the Kubernetes API – is what becomes valuable.

Containerized HPC Applications on Kubernetes

Over the past five years, dozens of HPC applications have been containerized, be it commercial, like ANSYS, COMSOL, STAR-CCM+, or open source packages like OpenFOAM and GROMACS, along with HPC cluster schedulers like Univa Grid Engine and Slurm. Thanks to container technology, a constant stream of updates and improvements is provided which can be promptly and seamlessly updated by customers. Additionally, the container images allow users to go back at any time to a previous application version so that they always can reproduce their previous results.

Example HPC Application Cluster Architecture running on Managed Kubernetes.

In the meantime, many container environments have been implemented by using infrastructure and configuration management tools like Terraform and Puppet or by building cloud specific HPC integrations into existing portals. But with the advent of Kubernetes, container environments became easier to maintain and are much more dynamic. Rolling out a cluster, rescaling the worker nodes, using a constant set of preemptible instances, and high availability are driven by controllers which continuously drive the cluster to the desired state. Thus, major HPC gaps of Kubernetes have been closed. This way, today, distributed memory/MPI jobs can be supported in any Kubernetes environment, which provides a built-in HPC workload manager integration running inside HPC containers. That allows traditional HPC applications to run without any changes. Also, GPU and non-GPU enabled applications based on Ansys and COMSOL have been launched successfully, through a high-performance, GPU enabled desktop running inside a pod. Once logged in to the desktop the engineer can start submitting batch jobs or single MPI applications which are distributed across a set of pods allocated on multiple nodes.

Conclusions

Kubernetes not only supports microservice based enterprise applications, but also self-service engineering HPC applications. In summary, as this research has shown, the key advantages of using Kubernetes as a foundation for running containerized engineering applications are:

  • Unified application stack available on virtually any infrastructure
  • True hybrid cloud usage scenarios for engineering workload. For the engineers it is transparent where the application runs, be it on-premises or in the cloud
  • which leads to providing the best performance for running engineering applications by allocating always the newest and fastest machines available in the cloud
  • Building and resizing a self-contained HPC application and compute cluster as self-service for the engineer which is only limited by cloud quotas and budget per time period
  • Robust management stack, supported by many Cloud providers
  • Optimizing costs by only paying for what is used. No idle resources which need to be allocated before they are going to be used.
  • High security through self-contained dedicated compute clusters
  • Minimal operational overhead by self-provisioning and disposable components for which updates are simple destroy and re-create commands
  • Kubernetes based workload is easier to integrate in widely adopted continuous integration and deployment solutions (like Tekton, Concourse, or future versions of Jenkins)

In this research, container-based HPC application environments have been implemented on top of Kubernetes (e.g. on Google GCP and Amazon AWS) and also used as self-service test environments which can be deployed from scratch by HPC application specialists, not operators. It has also been used in CI/CD pipelines to automatically build test environments which run tests against existing container solutions and shut down the infrastructure afterwards. In customer environments, the IT group benefits from an easier to maintain system using a supported, managed Kubernetes which can ramp up, resized and deleted computing resources within minutes.

About the Authors

Daniel Gruber, Burak Yenier, and Wolfgang Gentzsch are with UberCloud, a company that started in 2013 with developing HPC container technology and containerized engineering applications, to facilitate access and use of engineering HPC workload in a shared on-premise or on-demand cloud environment. In this article and the part-one article published on HPCwire last September, they describe their experiences during the last 12 months using UberCloud HPC containers on Kubernetes.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Watch Nvidia’s GTC21 Keynote with Jensen Huang Livestreamed Here, Monday at 8:30am PT

April 9, 2021

Join HPCwire right here on Monday, April 12, at 8:30 am PT to see the Nvidia GTC21 keynote from Nvidia’s CEO, Jensen Huang, livestreamed in its entirety. Hosted by HPCwire, you can click to join the Huang keynote on our livestream to hear Nvidia’s expected news and... Read more…

The US Places Seven Additional Chinese Supercomputing Entities on Blacklist

April 8, 2021

As tensions between the U.S. and China continue to simmer, the U.S. government today added seven Chinese supercomputing entities to an economic blacklist. The U.S. Entity List bars U.S. firms from supplying key technolog Read more…

Argonne Supercomputing Supports Caterpillar Engine Design

April 8, 2021

Diesel fuels still account for nearly ten percent of all energy-related U.S. carbon emissions – most of them from heavy-duty vehicles like trucks and construction equipment. Energy efficiency is key to these machines, Read more…

Habana’s AI Silicon Comes to San Diego Supercomputer Center

April 8, 2021

Habana Labs, an Intel-owned AI company, has partnered with server maker Supermicro to provide high-performance, high-efficiency AI computing in the form of new training and inference servers that will power the upcoming Read more…

Intel Partners Debut Latest Servers Based on the New Intel Gen 3 ‘Ice Lake’ Xeons

April 7, 2021

Fresh from Intel’s launch of the company’s latest third-generation Xeon Scalable “Ice Lake” processors on April 6 (Tuesday), Intel server partners Cisco, Dell EMC, HPE and Lenovo simultaneously unveiled their first server models built around the latest chips. And though arch-rival AMD may... Read more…

AWS Solution Channel

Volkswagen Passenger Cars Uses NICE DCV for High-Performance 3D Remote Visualization

 

Volkswagen Passenger Cars has been one of the world’s largest car manufacturers for over 70 years. The company delivers more than 6 million automobiles to global customers every year, from 50 production locations on five continents. Read more…

What’s New in HPC Research: Tundra, Fugaku, µHPC & More

April 6, 2021

In this regular feature, HPCwire highlights newly published research in the high-performance computing community and related domains. From parallel programming to exascale to quantum computing, the details are here. Read more…

The US Places Seven Additional Chinese Supercomputing Entities on Blacklist

April 8, 2021

As tensions between the U.S. and China continue to simmer, the U.S. government today added seven Chinese supercomputing entities to an economic blacklist. The U Read more…

Habana’s AI Silicon Comes to San Diego Supercomputer Center

April 8, 2021

Habana Labs, an Intel-owned AI company, has partnered with server maker Supermicro to provide high-performance, high-efficiency AI computing in the form of new Read more…

Intel Partners Debut Latest Servers Based on the New Intel Gen 3 ‘Ice Lake’ Xeons

April 7, 2021

Fresh from Intel’s launch of the company’s latest third-generation Xeon Scalable “Ice Lake” processors on April 6 (Tuesday), Intel server partners Cisco, Dell EMC, HPE and Lenovo simultaneously unveiled their first server models built around the latest chips. And though arch-rival AMD may... Read more…

Intel Launches 10nm ‘Ice Lake’ Datacenter CPU with Up to 40 Cores

April 6, 2021

The wait is over. Today Intel officially launched its 10nm datacenter CPU, the third-generation Intel Xeon Scalable processor, codenamed Ice Lake. With up to 40 Read more…

HPE Launches Storage Line Loaded with IBM’s Spectrum Scale File System

April 6, 2021

HPE today launched a new family of storage solutions bundled with IBM’s Spectrum Scale Erasure Code Edition parallel file system (description below) and featu Read more…

RIKEN’s Ongoing COVID Research Includes New Vaccines, New Tests & More

April 6, 2021

RIKEN took the supercomputing world by storm last summer when it launched Fugaku – which became (and remains) the world’s most powerful supercomputer – ne Read more…

CERN Is Betting Big on Exascale

April 1, 2021

The European Organization for Nuclear Research (CERN) involves 23 countries, 15,000 researchers, billions of dollars a year, and the biggest machine in the worl Read more…

AI Systems Summit Keynote: Brace for System Level Heterogeneity Says de Supinski

April 1, 2021

Heterogeneous computing has quickly come to mean packing a couple of CPUs and one-or-many accelerators, mostly GPUs, onto the same node. Today, a one-such-node system has become the standard AI server offered by dozens of vendors. This is not to diminish the many advances... Read more…

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from Julialang.org. I Read more…

Intel Launches 10nm ‘Ice Lake’ Datacenter CPU with Up to 40 Cores

April 6, 2021

The wait is over. Today Intel officially launched its 10nm datacenter CPU, the third-generation Intel Xeon Scalable processor, codenamed Ice Lake. With up to 40 Read more…

CERN Is Betting Big on Exascale

April 1, 2021

The European Organization for Nuclear Research (CERN) involves 23 countries, 15,000 researchers, billions of dollars a year, and the biggest machine in the worl Read more…

Programming the Soon-to-Be World’s Fastest Supercomputer, Frontier

January 5, 2021

What’s it like designing an app for the world’s fastest supercomputer, set to come online in the United States in 2021? The University of Delaware’s Sunita Chandrasekaran is leading an elite international team in just that task. Chandrasekaran, assistant professor of computer and information sciences, recently was named... Read more…

HPE Launches Storage Line Loaded with IBM’s Spectrum Scale File System

April 6, 2021

HPE today launched a new family of storage solutions bundled with IBM’s Spectrum Scale Erasure Code Edition parallel file system (description below) and featu Read more…

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

Saudi Aramco Unveils Dammam 7, Its New Top Ten Supercomputer

January 21, 2021

By revenue, oil and gas giant Saudi Aramco is one of the largest companies in the world, and it has historically employed commensurate amounts of supercomputing Read more…

Quantum Computer Start-up IonQ Plans IPO via SPAC

March 8, 2021

IonQ, a Maryland-based quantum computing start-up working with ion trap technology, plans to go public via a Special Purpose Acquisition Company (SPAC) merger a Read more…

Leading Solution Providers

Contributors

Can Deep Learning Replace Numerical Weather Prediction?

March 3, 2021

Numerical weather prediction (NWP) is a mainstay of supercomputing. Some of the first applications of the first supercomputers dealt with climate modeling, and Read more…

Livermore’s El Capitan Supercomputer to Debut HPE ‘Rabbit’ Near Node Local Storage

February 18, 2021

A near node local storage innovation called Rabbit factored heavily into Lawrence Livermore National Laboratory’s decision to select Cray’s proposal for its CORAL-2 machine, the lab’s first exascale-class supercomputer, El Capitan. Details of this new storage technology were revealed... Read more…

New Deep Learning Algorithm Solves Rubik’s Cube

July 25, 2018

Solving (and attempting to solve) Rubik’s Cube has delighted millions of puzzle lovers since 1974 when the cube was invented by Hungarian sculptor and archite Read more…

African Supercomputing Center Inaugurates ‘Toubkal,’ Most Powerful Supercomputer on the Continent

February 25, 2021

Historically, Africa hasn’t exactly been synonymous with supercomputing. There are only a handful of supercomputers on the continent, with few ranking on the Read more…

The History of Supercomputing vs. COVID-19

March 9, 2021

The COVID-19 pandemic poses a greater challenge to the high-performance computing community than any before. HPCwire's coverage of the supercomputing response t Read more…

HPE Names Justin Hotard New HPC Chief as Pete Ungaro Departs

March 2, 2021

HPE CEO Antonio Neri announced today (March 2, 2021) the appointment of Justin Hotard as general manager of HPC, mission critical solutions and labs, effective Read more…

AMD Launches Epyc ‘Milan’ with 19 SKUs for HPC, Enterprise and Hyperscale

March 15, 2021

At a virtual launch event held today (Monday), AMD revealed its third-generation Epyc “Milan” CPU lineup: a set of 19 SKUs -- including the flagship 64-core, 280-watt 7763 part --  aimed at HPC, enterprise and cloud workloads. Notably, the third-gen Epyc Milan chips achieve 19 percent... Read more…

Microsoft, HPE Bringing AI, Edge, Cloud to Earth Orbit in Preparation for Mars Missions

February 12, 2021

The International Space Station will soon get a delivery of powerful AI, edge and cloud computing tools from HPE and Microsoft Azure to expand technology experi Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire