AMD’s MI300 APUs to Power Exascale El Capitan Supercomputer

By Tiffany Trader

June 21, 2022

Additional details of the architecture of the exascale El Capitan supercomputer were disclosed today by Lawrence Livermore National Laboratory’s (LLNL) Terri Quinn in a presentation delivered to the 79th HPC User Forum at Oak Ridge National Laboratory (ORNL). Quinn revealed that AMD’s forthcoming MI300 APU would be the computational bedrock of El Capitan, which is slated for installation at LLNL in late 2023.

A render of El Capitan.

El Capitan, an HPE-built, AMD-powered exascale system, is on track to deliver at least 2 exaflops of peak performance courtesy of tightly packaged AMD GPUs and CPUs bundled into HPE Cray XE racks and tied together with Slingshot-11 networking.

Now, we know a bit more about that AMD hardware: El Capitan will be powered by AMD’s forthcoming MI300 APUs.

“It’s the first time we’ve publicly stated this,” said Quinn, associate director for HPC at LLNL. “I cut these words out of [AMD’s] investors document and that’s what it says: it’s a 3D chiplet design with AMD CDNA3 GPUs, Zen 4 CPUs, cache memory and HBM chiplets.”  

“I can’t give you all the specs, but [El Capitan] is at least 10× greater than Sierra performance on average figures of merit,” said Quinn. “Theoretical peak is two double-precision exaflops, [and we’ll] keep it under 40 megawatts—same reason as Oak Ridge, the operating cost.”

AMD MI300 rendering

The MI300 APU is the follow-on to the MI200 GPU, the powerhouse of the exascale Frontier supercomputer that is in the final stages of acceptance testing at Oak Ridge. The culmination of AMD’s expertise in processors, packaging and fabric technology, the MI300 will leverage AMD’s fourth generation Infinity architecture, which enables 2.5D and 3D chiplet integration with unified system level coherency. It will be implemented with AMD’s next-generation Zen 4 “Genoa” CPU, the successor to Milan.

At AMD’s 2022 Financial Analyst Day earlier this month, AMD’s Forrest Norrod said that the MI300 APU would deliver 8× more training performance using its new FP8 numerical format. Further, AMD reported that Genoa (the CPU engine inside El Capitan) will deliver 75 percent higher Java Enterprise performance.

Quinn also said that while Livermore has traditionally used the vendor’s system software and management software, they are switching over to use the custom NNSA Tri-Lab Operating System Stack (TOSS) that they’ve used for traditional Linux clusters.

It makes sense, said Quinn, if you think of El Capitan as, essentially, a big Linux cluster. “It’s not thousands of nodes, it’s [on the] order of nodes that we get on our other clusters. So we put it on here, and we can have every one of our systems in our center running exactly the same operating system, so it’s a big attraction for us.”

Livermore is already operating a number of test and development systems in preparation for receiving El Capitan. Three such early access systems — rzVernal, Tioga and Tenaya — ranked in the top 200 of the recent Top500 list. All are built by HPE using “Frontier-style” blades that consist of AMD MI250X GPUs, Milan CPUs and Slingshot-11 networking inside Cray EX cabinets.

LLNL operates under the auspices of the National Nuclear Security Administration (NNSA), which strives to bolster national security through the military applications of nuclear science. To that end, Quinn explained that the U.S.’ “global competitors” have been modernizing their nuclear weapons stockpiles in recent years. “The U.S. has relied on extending the life of our nuclear weapons,” she continued. “But we’ve kind of reached the point where the government wants to look at modernizing our stockpile[.] El Capitan was planned to deliver on that mission.”

Last month, NNSA completed the Exascale Computing Facility Modernization (ECFM) Project at Livermore after initiating the project in March 2020. Quinn explained that the existing facility had sufficient size (48,000 square feet) and structural integrity (625 pounds per square foot), but needed an overhaul to support the sites’ mission of enabling the operation of two exascale-class systems simultaneously. The electrical supply was upgraded from 45MW to 85MW and cooling was scaled to accommodate 28,000 tons of water, including a new 18,000-ton cooling tower.

Installation on El Capitan is slated for 2023, with the aim of putting NNSA workloads into production in the second quarter of 2024. The system is expected to operate from 2024 to 2030.

The exascale field

NNSA’s El Capitan system is currently third in the U.S. exascale queue. Front-runner Frontier — also an HPE-AMD-DOE collab — just became the first supercomputer to cross the Linpack exaflops threshold, clocking 1.102 exaflops of performance on the spring 2022 Top500 list. On the Green500, Frontier rides shotgun to its companion test and development system (Frontier TDS, aka Borg), which took the number one spot on the Green500 list. (Borg is #29 on the Top500 with 19.2 petaflops.) 

Argonne National Laboratory is awaiting completion of the Aurora supercomputer, a 2-exaflops HPE-Intel machine that has undergone several reconceptualizations. Aurora’s execution is also potentially beset by additional delays pertaining to its Sapphire Rapids CPU, so the exact timeline is fuzzy — but, reportedly, installation is underway.

Oliver Peckham contributed to this report.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

More Details on ‘Half-Exaflop’ Horizon System, LCCF Emerge

September 26, 2022

Since 2017, plans for the Leadership-Class Computing Facility (LCCF) have been underway. Slated for full operation somewhere around 2026, the LCCF’s scope extends far beyond that of the large supercomputer — Horizon Read more…

Nvidia Shuts Out RISC-V Software Support for GPUs 

September 23, 2022

Nvidia is not interested in bringing software support to its GPUs for the RISC-V architecture despite being an early adopter of the open-source technology in its GPU controllers. Nvidia has no plans to add RISC-V support for CUDA, which is the proprietary GPU software platform, a company representative... Read more…

Microsoft Closes Confidential Computing Loop with AMD’s Milan Chip

September 22, 2022

Microsoft shared details on how it uses an AMD technology to secure artificial intelligence as it builds out a secure AI infrastructure in its Azure cloud service. Microsoft has a strong relationship with Nvidia, but is also working with AMD's Epyc chips (including the new 3D VCache series), MI Instinct accelerators, and also... Read more…

Nvidia Introduces New Ada Lovelace GPU Architecture, OVX Systems, Omniverse Cloud

September 20, 2022

In his GTC keynote today, Nvidia CEO Jensen Huang launched another new Nvidia GPU architecture: Ada Lovelace, named for the legendary mathematician regarded as the first computer programmer. The company also announced tw Read more…

Nvidia’s Hopper GPUs Enter ‘Full Production,’ DGXs Delayed Until Q1

September 20, 2022

Just about six months ago, Nvidia’s spring GTC event saw the announcement of its hotly anticipated Hopper GPU architecture. Now, the GPU giant is announcing that Hopper-generation GPUs (which promise greater energy eff Read more…

AWS Solution Channel

Shutterstock 1194728515

Simulating 44-Qubit quantum circuits using AWS ParallelCluster

Dr. Fabio Baruffa, Sr. HPC & QC Solutions Architect
Dr. Pavel Lougovski, Pr. QC Research Scientist
Tyson Jones, Doctoral researcher, University of Oxford

Introduction

Currently, an enormous effort is underway to develop quantum computing hardware capable of scaling to hundreds, thousands, and even millions of physical (non-error-corrected) qubits. Read more…

Microsoft/NVIDIA Solution Channel

Shutterstock 1166887495

Improving Insurance Fraud Detection using AI Running on Cloud-based GPU-Accelerated Systems

Insurance is a highly regulated industry that is evolving as the industry faces changing customer expectations, massive amounts of data, and increased regulations. A major issue facing the industry is tracking insurance fraud. Read more…

NeMo LLM Service: Nvidia’s First Cloud Service Makes AI Less Vague

September 20, 2022

Nvidia is trying to uncomplicate AI with a cloud service that makes AI and its many forms of computing less vague and more conversational. The NeMo LLM service, which Nvidia called its first cloud service, adds a layer of intelligence and interactivity... Read more…

More Details on ‘Half-Exaflop’ Horizon System, LCCF Emerge

September 26, 2022

Since 2017, plans for the Leadership-Class Computing Facility (LCCF) have been underway. Slated for full operation somewhere around 2026, the LCCF’s scope ext Read more…

Nvidia Shuts Out RISC-V Software Support for GPUs 

September 23, 2022

Nvidia is not interested in bringing software support to its GPUs for the RISC-V architecture despite being an early adopter of the open-source technology in its GPU controllers. Nvidia has no plans to add RISC-V support for CUDA, which is the proprietary GPU software platform, a company representative... Read more…

Nvidia Introduces New Ada Lovelace GPU Architecture, OVX Systems, Omniverse Cloud

September 20, 2022

In his GTC keynote today, Nvidia CEO Jensen Huang launched another new Nvidia GPU architecture: Ada Lovelace, named for the legendary mathematician regarded as Read more…

Nvidia’s Hopper GPUs Enter ‘Full Production,’ DGXs Delayed Until Q1

September 20, 2022

Just about six months ago, Nvidia’s spring GTC event saw the announcement of its hotly anticipated Hopper GPU architecture. Now, the GPU giant is announcing t Read more…

NeMo LLM Service: Nvidia’s First Cloud Service Makes AI Less Vague

September 20, 2022

Nvidia is trying to uncomplicate AI with a cloud service that makes AI and its many forms of computing less vague and more conversational. The NeMo LLM service, which Nvidia called its first cloud service, adds a layer of intelligence and interactivity... Read more…

Nvidia Targets Computers for Robots in the Surgery Rooms

September 20, 2022

Nvidia is laying the groundwork for a future in which humans and robots will be collaborators in the surgery rooms at hospitals. The company announced a computer called IGX for Medical Devices, which will be populated in robots, image scanners and other computers and medical devices involved in patient care close to the point... Read more…

Survey Results: PsiQuantum, ORNL, and D-Wave Tackle Benchmarking, Networking, and More

September 19, 2022

The are many issues in quantum computing today – among the more pressing are benchmarking, networking and development of hybrid classical-quantum approaches. Read more…

HPC + AI Wall Street to Feature ‘Spooky’ Science for Financial Services

September 18, 2022

Albert Einstein famously described quantum mechanics as "spooky action at a distance" due to the non-intuitive nature of superposition and quantum entangled par Read more…

Nvidia Shuts Out RISC-V Software Support for GPUs 

September 23, 2022

Nvidia is not interested in bringing software support to its GPUs for the RISC-V architecture despite being an early adopter of the open-source technology in its GPU controllers. Nvidia has no plans to add RISC-V support for CUDA, which is the proprietary GPU software platform, a company representative... Read more…

AWS Takes the Short and Long View of Quantum Computing

August 30, 2022

It is perhaps not surprising that the big cloud providers – a poor term really – have jumped into quantum computing. Amazon, Microsoft Azure, Google, and th Read more…

US Senate Passes CHIPS Act Temperature Check, but Challenges Linger

July 19, 2022

The U.S. Senate on Tuesday passed a major hurdle that will open up close to $52 billion in grants for the semiconductor industry to boost manufacturing, supply chain and research and development. U.S. senators voted 64-34 in favor of advancing the CHIPS Act, which sets the stage for the final consideration... Read more…

Chinese Startup Biren Details BR100 GPU

August 22, 2022

Amid the high-performance GPU turf tussle between AMD and Nvidia (and soon, Intel), a new, China-based player is emerging: Biren Technology, founded in 2019 and headquartered in Shanghai. At Hot Chips 34, Biren co-founder and president Lingjie Xu and Biren CTO Mike Hong took the (virtual) stage to detail the company’s inaugural product: the Biren BR100 general-purpose GPU (GPGPU). “It is my honor to present... Read more…

Newly-Observed Higgs Mode Holds Promise in Quantum Computing

June 8, 2022

The first-ever appearance of a previously undetectable quantum excitation known as the axial Higgs mode – exciting in its own right – also holds promise for developing and manipulating higher temperature quantum materials... Read more…

AMD’s MI300 APUs to Power Exascale El Capitan Supercomputer

June 21, 2022

Additional details of the architecture of the exascale El Capitan supercomputer were disclosed today by Lawrence Livermore National Laboratory’s (LLNL) Terri Read more…

Tesla Bulks Up Its GPU-Powered AI Super – Is Dojo Next?

August 16, 2022

Tesla has revealed that its biggest in-house AI supercomputer – which we wrote about last year – now has a total of 7,360 A100 GPUs, a nearly 28 percent uplift from its previous total of 5,760 GPUs. That’s enough GPU oomph for a top seven spot on the Top500, although the tech company best known for its electric vehicles has not publicly benchmarked the system. If it had, it would... Read more…

The Final Frontier: US Has Its First Exascale Supercomputer

May 30, 2022

In April 2018, the U.S. Department of Energy announced plans to procure a trio of exascale supercomputers at a total cost of up to $1.8 billion dollars. Over the ensuing four years, many announcements were made, many deadlines were missed, and a pandemic threw the world into disarray. Now, at long last, HPE and Oak Ridge National Laboratory (ORNL) have announced that the first of those... Read more…

Leading Solution Providers

Contributors

Exclusive Inside Look at First US Exascale Supercomputer

July 1, 2022

HPCwire takes you inside the Frontier datacenter at DOE's Oak Ridge National Laboratory (ORNL) in Oak Ridge, Tenn., for an interview with Frontier Project Direc Read more…

AMD Opens Up Chip Design to the Outside for Custom Future

June 15, 2022

AMD is getting personal with chips as it sets sail to make products more to the liking of its customers. The chipmaker detailed a modular chip future in which customers can mix and match non-AMD processors in a custom chip package. "We are focused on making it easier to implement chips with more flexibility," said Mark Papermaster, chief technology officer at AMD during the analyst day meeting late last week. Read more…

Intel Reiterates Plans to Merge CPU, GPU High-performance Chip Roadmaps

May 31, 2022

Intel reiterated it is well on its way to merging its roadmap of high-performance CPUs and GPUs as it shifts over to newer manufacturing processes and packaging technologies in the coming years. The company is merging the CPU and GPU lineups into a chip (codenamed Falcon Shores) which Intel has dubbed an XPU. Falcon Shores... Read more…

Nvidia, Intel to Power Atos-Built MareNostrum 5 Supercomputer

June 16, 2022

The long-troubled, hotly anticipated MareNostrum 5 supercomputer finally has a vendor: Atos, which will be supplying a system that includes both Nvidia and Inte Read more…

UCIe Consortium Incorporates, Nvidia and Alibaba Round Out Board

August 2, 2022

The Universal Chiplet Interconnect Express (UCIe) consortium is moving ahead with its effort to standardize a universal interconnect at the package level. The c Read more…

Using Exascale Supercomputers to Make Clean Fusion Energy Possible

September 2, 2022

Fusion, the nuclear reaction that powers the Sun and the stars, has incredible potential as a source of safe, carbon-free and essentially limitless energy. But Read more…

Is Time Running Out for Compromise on America COMPETES/USICA Act?

June 22, 2022

You may recall that efforts proposed in 2020 to remake the National Science Foundation (Endless Frontier Act) have since expanded and morphed into two gigantic bills, the America COMPETES Act in the U.S. House of Representatives and the U.S. Innovation and Competition Act in the U.S. Senate. So far, efforts to reconcile the two pieces of legislation have snagged and recent reports... Read more…

India Launches Petascale ‘PARAM Ganga’ Supercomputer

March 8, 2022

Just a couple of weeks ago, the Indian government promised that it had five HPC systems in the final stages of installation and would launch nine new supercomputers this year. Now, it appears to be making good on that promise: the country’s National Supercomputing Mission (NSM) has announced the deployment of “PARAM Ganga” petascale supercomputer at Indian Institute of Technology (IIT)... Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire