HPC and AI Resources in the Great GPU Squeeze Panel

By Doug Eadline

October 19, 2023

The recent HPC + AI on Wall Street show presented a panel that brought together key vendors to discuss the current GPU Squeeze (shortage).   The “Squeeze” is due to the rapid and continuing rise of Generative AI/LLMs and has now pushed the demand for GPUs to unprecedented levels and created a “squeeze” on GPU availability.

A full 40-minute panel video is available for viewing. (In addition to all the other talks and panels of the event.) The panel was designed to provide views from several vantage points, including hardware, cloud, and on-prem perspectives. The was moderated by myself and consisted of the following members:

  • Wyatt Gorman – Solutions Manager, HPC and AI Infrastructure, Google Cloud
  • Prabhu Ramamoorthy – Global Partner Success Manager, Nvidia
  • Thomas Jorgensen – Senior Director, Technology Enablement, Supermicro
  • Kiran Agrahara – Cloud Solutions Architect, Intel Corporation

The panel covered four topics: HPC and GenAI, The GPU Shortage, Alternatives, and HPC/GenAI hardware convergence. A synopsis of the panel’s discussion is provided for each question. The full discussion is available in the video.

2023 HPC + AI on Wall Street GPU Squeeze Panel

Question 1: HPC and GenAI
Some users can find the line between traditional HPC and GenAI a little blurry. The need for fast GPUs indicates both types of applications are doing lots of number crunching, but is a traditional HPC Monte Carlo risk analysis similar to a GenAI trained to do risk analysis? In other words, how do you see GenAI and HPC? Are they in the same broad category or represent two distinct markets? Do they complement each other?

NVidia’s Prabhu Ramamoorthy mentioned right away that he sees this convergence happening now, and they see clients doing a mix of both HPC plus AI solutions as they are working toward their end use cases.

Google Cloud’s Wyatt Gorman replied that he sees people bringing machine learning techniques into HPC processes at different levels. So consider it a subset, a domain within HPC. He suggests it might spin out and become more siloed like Big Data may have out of HPC. But for now, he thinks it’s converging.

Question 2: The Shortage
Based on the news we have reported, I assume there are some challenges for HPC users accessing GPUs (purchased or in the cloud). I want to check my assumption and ask from your company’s perspective how big an issue is the “GPU Squeeze.”

Thomas Jorgensen of Supermicro shared an interesting data point. “I can tell you that we have tens of thousands of systems on backorder because of the lack of H100 GPUs. But, I will also say that there are a number of alternatives on the market, and we have a group of newer CPUs, for instance, for HPC workloads, which traditionally has been serviced by GPUs.”

Thomas mentioned the new Intel Max Xeon CPUs with 64 GB of embedded HBM2 memory, greatly enhancing some of the HPC workloads. He suggests that on-prem HPC is one of the ways to alleviate the situation if you cannot get Nvidia GPUs. As a hardware-neutral vendor, he also mentioned other GPUs from both Intel and AMD are easily available.

Thomas also mentions an important point, “A lot of people are saying we are dependent on CUDA, but if you have to wait a year for NVidia GPUS, you can convert a lot of code in that period. So there is absolutely life in some of the other GPUs available on the market, and some of them can be hacked with very little wait time.”

He concluded by also mentioning the Nvidia L40 GPUs, which he sees as a great replacement right now or at least something you can purchase. In addition, Thomas mentioned, “So, I will say for our on-prem customers, there’s a lot of alternatives, and customers are exploring that right now. And to great success, I would add.”

Kiran Agrahara from Intel added, “It depends on the use case. When you look at the shortages, it depends on what use case you’re looking into.” He continued, “There are alternatives available, and most of the learning and inferencing can be done with Xeon Gen 4 processors today.” He mentioned Intel has benchmarks available for different use cases. He also pointed out that everybody thinks an NVidia GPU is needed for every use case.

Kiran also pointed out that Intel has Gaudi 2, which is more of an inferencing processor than a learning processor and is available on-prem and in the cloud. Continuing, he mentioned Datacenter GPU Max Series that could be a substitute or an alternative to A100s. He concluded by calling attention to Intel’s OpenVINO (Open Visual Inference and Neural Network Optimization) — a software layer on the top of all the Intel HPC and AI products.

Prabhu Ramamoorthy pointed out that people think that this GPU shortage happened recently, but we have seen it for several years. For instance, the GPUs ran in the cloud three years ago doing NLP (Natural Language Processing). A lot of hedge funds were running all these workloads. Generative AI is like the new, beautiful, sexier term for NLP, and now there is much more demand for these GPUs. We want customers to choose the right use case –as pointed out by Thomas. He also reminded everyone that NVidia employs more software engineers than hardware engineers. As a result of these efforts, he mentioned Nvidia software like TensorRT-LLM that optimizes applications and ensures that GPUs are utilized more efficiently.

Wyatt Gorman concluded that he thinks it’s important to plan your GPU needs for the long term. Now, look at alternatives and not get too far ahead of your applications. Consider what possibilities are out there now, and there are ways to optimize GPU use, and that are good alternatives. are good alternatives.

Question 3: GPU Alternatives
Reports of the shortage may last another 18 months. Are there alternative ways to suggest HPC users run applications (cloud, alternative hardware, software optimizations)? For instance, NAMD, an HPC molecular dynamics code, will run using CPUs and MPI and/or with GPUs. Will users revert to MPI?

Kiran Agrahara recalled an older example where users needed 20 GPUs, and for the same work to be done, they needed about 180 CPUs. But he suggested looking at what’s changed in the last three years. He continues, “The CPUs have come a long way. Back when we did not have hardware accelerators. If you look at the next-generation Xeon processors, we have an advanced matrix architecture called RMX. So today, the CPUs have come a long way. You’re looking at 128 cores, and then you have RMX architecture, which I can say is like AVX512 on steroids. So the answer is yes, you can accomplish this with CPUs.”

Wyatt Gorman added that it is really important to make people aware of smaller GPUs. Everyone is thinking about A100s and H100s right now. The slower Nvidia T4, L4, and L40 GPUs have the capacity and are available on Google Cloud because, right now, they aren’t in such high demand. If you do some tuning and optimization for these GPUs, you can see the acceleration of your code.

Question 4: HPC and Data Center Convergence
Typically HPC is in its own kind of silo, and then you have rapidly growing GenAI with similar hardware in another developing silo. Do you see that this may benefit HPC when people start to recognize we could bring both of these resources together? What do you think about the convergence of HPC applications running in the data center versus in the R&D lab with separate hardware?

Wyatt Gorman shared that Google sees more AI capability in HPC RFPs. He thinks as these problem types, as HPC and AI converge, we see AI techniques being brought into traditional HPC problem-solving applications, and we’ll see more and more of that and more and more resources being necessary. As I mentioned, Google TPUs are now supported in Slurm, and you don’t necessarily need Kubernetes to run HPC now or in the future. And you know, it won’t be necessary even when it’s an option. 

Thomas Jorgensen of Supermicro mentioned that the Intel fifth-generation roadmap showed up to 288 cores on some future CPUs. There are absolutely things also happening on the CPU platform that will result in more performance than a traditional CPU platform. Another point is the Max GPU that Intel has created is an HPC plus AI GPU. The first test we’ve done at Supermicro shows real performance for HPC with that GPU.

From a hardware standpoint, Supermicro tries to be Switzerland. We work with Nvidia, AMD, and Intel.

Thomas continued, “Like Intel AMD platforms, we have very high core count Epyc processors and a science experiment with Nvidia. In this test, we can place ten GPUs in a single system. The HPC benchmarks, and the performance we are getting out of one system on HPC is astounding. But it’s a way forward, at least for, I mean, it doesn’t help with the squeeze, right, because using H100, but there’s some performance there that shows real promise for HPC workloads. So I’m very hopeful for the hardware and some optimizations we’re seeing that can wring more from existing hardware.”

As the moderator, I should note that after the panel concluded, the panelists continued this discussion in the speaker’s green room for about 20 minutes. The market is not done with this topic.

These above responses are a summary of the full GPU Squeeze panel. There is a lot more insightful discussion to be found on all the videos.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industry updates delivered to you every week!

A Big Memory Nvidia GH200 Next to Your Desk: Closer Than You Think

February 22, 2024

Students of the microprocessor may recall that the original 8086/8088 processors did not have floating point units. The motherboard often had an extra socket for an optional 8087 math coprocessor. The math coprocessor ma Read more…

IonQ Reports Advance on Path to Networked Quantum Computing

February 22, 2024

IonQ reported reaching a milestone in its efforts to use entangled photon-ion connectivity to scale its quantum computers. IonQ’s quantum computers are based on trapped ions which feature long coherence times and qubit Read more…

Apple Rolls out Post Quantum Security for iOS

February 21, 2024

Think implementing so-called Post Quantum Cryptography (PQC) isn't important because quantum computers able to decrypt current RSA codes don’t yet exist? Not Apple. Today the consumer electronics giant started rolling Read more…

GenAI Having Major Impact on Data Culture, Survey Says

February 21, 2024

While 2023 was the year of GenAI, the adoption rates for GenAI did not match expectations. Most organizations are continuing to invest in GenAI but are yet to derive any substantial value from it. However, the GenAI hyp Read more…

QED-C Issues New Quantum Benchmarking Paper

February 20, 2024

The Quantum Economic Development Consortium last week released a new paper on benchmarking – Quantum Algorithm Exploration using Application-Oriented Performance Benchmarks – that builds on earlier work and is an eff Read more…

AWS Solution Channel

Shutterstock 2283618597

Deep-dive into Ansys Fluent performance on Ansys Gateway powered by AWS

Today, we’re going to deep-dive into the performance and associated cost of running computational fluid dynamics (CFD) simulations on AWS using Ansys Fluent through the Ansys Gateway powered by AWS (or just “Ansys Gateway” for the rest of this post). Read more…

Atom Computing Reports Advance in Scaling Up Neutral Atom Qubit Arrays

February 15, 2024

The scale-up challenge facing quantum computing (QC) is daunting and varied. It’s commonly held that 1 million qubits (or more) will be needed to deliver practical fault tolerant QC. It’s also a varied challenge beca Read more…

A Big Memory Nvidia GH200 Next to Your Desk: Closer Than You Think

February 22, 2024

Students of the microprocessor may recall that the original 8086/8088 processors did not have floating point units. The motherboard often had an extra socket fo Read more…

Apple Rolls out Post Quantum Security for iOS

February 21, 2024

Think implementing so-called Post Quantum Cryptography (PQC) isn't important because quantum computers able to decrypt current RSA codes don’t yet exist? Not Read more…

QED-C Issues New Quantum Benchmarking Paper

February 20, 2024

The Quantum Economic Development Consortium last week released a new paper on benchmarking – Quantum Algorithm Exploration using Application-Oriented Performa Read more…

The Pulse of HPC: Tracking 4.5 Million Heartbeats of 3D Coronary Flow

February 15, 2024

Working in Duke University's Randles Lab, Cyrus Tanade, a National Science Foundation graduate student fellow and Ph.D. candidate in biomedical engineering, is Read more…

It Doesn’t Get Much SWEETER: The Winter HPC Computing Festival in Corpus Christi

February 14, 2024

(Main Photo by Visit Corpus Christi CrowdRiff) Texas A&M University's High-Performance Research Computing (HPRC) team hosted the "SWEETER Winter Comput Read more…

Q-Roundup: Diraq’s War Chest, DARPA’s Bet on Topological Qubits, Citi/Classiq Explore Optimization, WEF’s Quantum Blueprint

February 13, 2024

Yesterday, Australian start-up Diraq added $15 million to its war chest (now $120 million) to build a fault tolerant computer based on quantum dots. Last week D Read more…

2024 Winter Classic: Razor Thin Margins in HPL/HPCG

February 12, 2024

The first task for the 11 teams in the 2024 Winter Classic student cluster competition was to run and optimize the LINPACK and HPCG benchmarks. As usual, the Read more…

2024 Winter Classic: We’re Back!

February 9, 2024

The fourth edition of the Winter Classic Invitational Student Cluster Competition is up and running. This year, we have 11 teams of eager students representin Read more…

CORNELL I-WAY DEMONSTRATION PITS PARASITE AGAINST VICTIM

October 6, 1995

Ithaca, NY --Visitors to this year's Supercomputing '95 (SC'95) conference will witness a life-and-death struggle between parasite and victim, using virtual Read more…

SGI POWERS VIRTUAL OPERATING ROOM USED IN SURGEON TRAINING

October 6, 1995

Surgery simulations to date have largely been created through the development of dedicated applications requiring considerable programming and computer graphi Read more…

U.S. Will Relax Export Restrictions on Supercomputers

October 6, 1995

New York, NY -- U.S. President Bill Clinton has announced that he will definitely relax restrictions on exports of high-performance computers, giving a boost Read more…

Dutch HPC Center Will Have 20 GFlop, 76-Node SP2 Online by 1996

October 6, 1995

Amsterdam, the Netherlands -- SARA, (Stichting Academisch Rekencentrum Amsterdam), Academic Computing Services of Amsterdam recently announced that it has pur Read more…

Cray Delivers J916 Compact Supercomputer to Solvay Chemical

October 6, 1995

Eagan, Minn. -- Cray Research Inc. has delivered a Cray J916 low-cost compact supercomputer and Cray's UniChem client/server computational chemistry software Read more…

NEC Laboratory Reviews First Year of Cooperative Projects

October 6, 1995

Sankt Augustin, Germany -- NEC C&C (Computers and Communication) Research Laboratory at the GMD Technopark has wrapped up its first year of operation. Read more…

Sun and Sybase Say SQL Server 11 Benchmarks at 4544.60 tpmC

October 6, 1995

Mountain View, Calif. -- Sun Microsystems, Inc. and Sybase, Inc. recently announced the first benchmark results for SQL Server 11. The result represents a n Read more…

New Study Says Parallel Processing Market Will Reach $14B in 1999

October 6, 1995

Mountain View, Calif. -- A study by the Palo Alto Management Group (PAMG) indicates the market for parallel processing systems will increase at more than 4 Read more…

Leading Solution Providers

Contributors

CORNELL I-WAY DEMONSTRATION PITS PARASITE AGAINST VICTIM

October 6, 1995

Ithaca, NY --Visitors to this year's Supercomputing '95 (SC'95) conference will witness a life-and-death struggle between parasite and victim, using virtual Read more…

SGI POWERS VIRTUAL OPERATING ROOM USED IN SURGEON TRAINING

October 6, 1995

Surgery simulations to date have largely been created through the development of dedicated applications requiring considerable programming and computer graphi Read more…

U.S. Will Relax Export Restrictions on Supercomputers

October 6, 1995

New York, NY -- U.S. President Bill Clinton has announced that he will definitely relax restrictions on exports of high-performance computers, giving a boost Read more…

Dutch HPC Center Will Have 20 GFlop, 76-Node SP2 Online by 1996

October 6, 1995

Amsterdam, the Netherlands -- SARA, (Stichting Academisch Rekencentrum Amsterdam), Academic Computing Services of Amsterdam recently announced that it has pur Read more…

Cray Delivers J916 Compact Supercomputer to Solvay Chemical

October 6, 1995

Eagan, Minn. -- Cray Research Inc. has delivered a Cray J916 low-cost compact supercomputer and Cray's UniChem client/server computational chemistry software Read more…

NEC Laboratory Reviews First Year of Cooperative Projects

October 6, 1995

Sankt Augustin, Germany -- NEC C&C (Computers and Communication) Research Laboratory at the GMD Technopark has wrapped up its first year of operation. Read more…

Sun and Sybase Say SQL Server 11 Benchmarks at 4544.60 tpmC

October 6, 1995

Mountain View, Calif. -- Sun Microsystems, Inc. and Sybase, Inc. recently announced the first benchmark results for SQL Server 11. The result represents a n Read more…

New Study Says Parallel Processing Market Will Reach $14B in 1999

October 6, 1995

Mountain View, Calif. -- A study by the Palo Alto Management Group (PAMG) indicates the market for parallel processing systems will increase at more than 4 Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire