The Final Frontier: US Has Its First Exascale Supercomputer

By Oliver Peckham

May 30, 2022

In April 2018, the U.S. Department of Energy announced plans to procure a trio of exascale supercomputers at a total cost of up to $1.8 billion dollars. Over the ensuing four years, many announcements were made, many deadlines were missed, and a pandemic threw the world into disarray. Now, at long last, HPE and Oak Ridge National Laboratory (ORNL) have announced that the first of those three systems is operational: the HPE-built, AMD-powered Frontier supercomputer, a behemoth system that delivers 1.102 Linpack exaflops of computing power.

The computer

Frontier consists of 74 cabinets, each weighing in at 8,000 pounds. 9,408 HPE Cray EX nodes are spread across these cabinets, with each node powered by an AMD “Trento” 7A53 Epyc CPU and quadruple AMD Instinct MI250X GPUs (totaling 37,632 GPUs across the system). The system has 9.2 petabytes of memory, split evenly between HBM and DDR4, and it uses HPE Slingshot-11 networking. It is supported by 37 petabytes of node-local storage on top of 716 petabytes of center-wide storage. The system is 100 percent liquid-cooled using warm (85°) water, and in terms of space, the final Frontier system occupies 372 square meters.

Image courtesy of HPE.

The 1.102 Linpack exaflops this hardware delivers easily earned it first place on the spring 2022 Top500 list (which was released today), and, in fact, the system is more powerful than the following seven Top500 systems combined. (For more on all of that, read HPCwire’s Top500 coverage.) Previously, Frontier had been characterized as a two peak exaflops system, but its first Top500 benchmark measures some 1.686 peak exaflops. (Oak Ridge said that there remains “much higher headroom on the GPUs and the CPUs” to achieve the two peak exaflops target.) Outside of Linpack and the Top500, the system benchmarks at 6.88 exaflops of mixed-precision performance on HPL-AI. The team ran out of time and was not able to submit an HPCG benchmark.

(There is, of course, an asterisk on Frontier’s debut as the first exascale system on the Top500: last year, details emerged of two operational exascale supercomputers in China: the OceanLight and Tianhe-3 systems. Those systems are not listed on the Top500.)

Unprecedented efficiency

Frontier also achieved another win out of the gate: second place on the spring 2022 Green500 list, which ranks supercomputers by their flops per watt. The Oak Ridge team accomplished this by delivering those 1.102 Linpack exaflops in a 21.1-megawatt power envelope, an efficiency of 52.23 gigaflops per watt (which works out to one exaflops at 19.15 megawatts). This puts the system well within the 20-megawatt exascale power envelope target set by DARPA in 2008—a target that had been viewed with much skepticism over the ensuing 14 years. Frontier was only outpaced in efficiency by its own 1-cabinet test and development system (Frontier TDS, aka “Borg”), which delivered 62.68 gigaflops per watt.

(This is not to say that Frontier is a low-energy system: ORNL is a direct-serve customer of the Tennessee Valley Authority, and in a press briefing, ORNL Director Thomas Zacharia said that “when you start the [Linpack] run, the machine, in less than ten seconds, begins to draw an additional 15 megawatts of power … that’s a small city in the U.S., that’s roughly about how much power the city of Oak Ridge consumes.”)

A challenging enterprise

Bringing a system like Frontier together over the course of the pandemic was no small feat. “We as a community, as a laboratory, as collaborators, we had to make a difficult decision,” Zacharia said. He explained that—before the Covid vaccines were available—much of the Oak Ridge staff involved in Frontier still came to work during the pandemic to deliver on the system while ameliorating Covid spread as much as possible through measures like contact tracing. Further, he said, HPE dedicated 18 people to “scouring the world for parts small and large in order to build a system to deliver on scope, schedule and budget” during the pandemic and the ensuing supply chain disruptions. “I don’t think that we would have been able to deliver this machine but for the fact that [Cray] is now HPE,” Zacharia said, “because it required the global region, the size of HPE, in order to tackle the supply chain issues.”

“To go from where we were seven months ago: battling an unprecedented global supply chain shortage, battling a pandemic … and delivering a first-ever exascale system with a first-ever CPU, first-ever GPU and a new fabric is truly incredible,” added Justin Hotard, EVP and GM for HPC and AI, in the press briefing. “To me, it’s the supercomputing equivalent of going to the moon.”

“Officially breaking the Linpack exascale barrier with a system comprised of new technologies and ahead of schedule is a testament to the thoughtful planning process and long-term vision of everyone associated with this machine,” commented Bob Sorensen, senior vice president of research for Hyperion Research. “I expect that just like the Roger Bannister four-minute mile, we will see this milestone number surpassed with some frequency in the next few years, but only one machine did it first. So the wait is over, we can all breathe a sigh of relief and move on to seeing the scientific and engineering advances enabled by Frontier as the system begins to address to the important real-world workloads for which it was designed.”

Moonshots

Indeed, both Hotard and Zacharia took care to emphasize the enormous impacts that Frontier stands to have across a range of scientific fields. (“This is really about the investments that we’re making in tackling science, right?” Zacharia said.) Across the press briefing, they mentioned applications ranging from materials research for energy storage in electric vehicles to regional climate simulations with resolved clouds and rain to in-depth studies of long Covid. There will, of course, be countless others—some enumerated in the graphic below—and Zacharia said that research aimed at the Gordon Bell Prize will be seen on the system by September of this year.

A sampling of the scientific codes that will be run on Frontier. Image courtesy of HPE/ORNL/Exascale Computing Project.

The Oak Ridge team is now beginning its “very rigorous acceptance testing process” for Frontier and says that it is on track for full operational availability by its previously set target date of January 1, 2023.

“It’s a proud moment, certainly, for Oak Ridge, for HPE, for AMD,” Zacharia said. “It’s a proud moment for our nation.”

The future

While we couldn’t pass up the Star Trek reference (nor could ORNL, which named Crusher, Frontier’s other – 1.5 cabinet – test system, after Beverly and Wesley Crusher), questions remain as to whether this really is the final Frontier. Asked whether the system could be expanded in the future, Zacharia said: “It is the full expanse of the current machine. We certainly have more power and space available to us. We don’t know what the future holds.”

But for now, with Frontier delivered, eyes will turn to Argonne National Laboratory’s Aurora, the second of the three U.S. exascale systems. Aurora’s installation is now underway and delivery of the Intel-based system (which is targeting “over two exaflops of peak performance”) is currently expected within 2022. After Aurora will be Lawrence Livermore National Laboratory’s El Capitan system, also from HPE and AMD and expected in 2023.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Is Time Running Out for Compromise on America COMPETES/USICA Act?

June 22, 2022

You may recall that efforts proposed in 2020 to remake the National Science Foundation (Endless Frontier Act) have since expanded and morphed into two gigantic bills, the America COMPETES Act in the U.S. House of Representatives and the U.S. Innovation and Competition Act in the U.S. Senate. So far, efforts to reconcile the two pieces of legislation have snagged and recent reports... Read more…

Cerebras Systems Thinks Forward on AI Chips as it Claims Performance Win

June 22, 2022

Cerebras Systems makes the largest chip in the world, but is already thinking about its upcoming AI chips as learning models continue to grow at breakneck speed. The company’s latest Wafer Scale Engine chip is indeed the size of a wafer, and is made using TSMC’s 7nm process. The next chip will pack in more cores to handle the fast-growing compute needs of AI, said Andrew Feldman, CEO of Cerebras Systems. Read more…

AMD’s MI300 APUs to Power Exascale El Capitan Supercomputer

June 21, 2022

Additional details of the architecture of the exascale El Capitan supercomputer were disclosed today by Lawrence Livermore National Laboratory’s (LLNL) Terri Quinn in a presentation delivered to the 79th HPC User Forum Read more…

IDC Perspective on Integration of Quantum Computing and HPC

June 20, 2022

The insatiable need to compress time to insights from massive and complex datasets is fueling the demand for quantum computing integration into high performance computing (HPC) environments. Such an integration would allow enterprises to accelerate and optimize current HPC applications and processes by simulating and emulating them on today’s noisy... Read more…

Q&A with Intel’s Jeff McVeigh, an HPCwire Person to Watch in 2022

June 17, 2022

HPCwire presents our interview with Jeff McVeigh, vice president and general manager, Super Compute Group, Intel Corporation, and an HPCwire 2022 Person to Watch. McVeigh shares Intel's plans for the year ahead, his pers Read more…

AWS Solution Channel

Shutterstock 152995403

Bayesian ML Models at Scale with AWS Batch

This post was contributed by Ampersand’s Jeffrey Enos, Senior Machine Learning Engineer, Daniel Gerlanc, Senior Director for Data Science, and Brandon Willard, Data Science Lead. Read more…

Microsoft/NVIDIA Solution Channel

Shutterstock 261863138

Using Cloud-Based, GPU-Accelerated AI for Financial Risk Management

There are strict rules governing financial institutions with a number of global regulatory groups publishing financial compliance requirements. Financial institutions face many challenges and legal responsibilities for risk management, compliance violations, and failure to catch financial fraud. Read more…

Nvidia, Intel to Power Atos-Built MareNostrum 5 Supercomputer

June 16, 2022

The long-troubled, hotly anticipated MareNostrum 5 supercomputer finally has a vendor: Atos, which will be supplying a system that includes both Nvidia and Intel CPUs and GPUs across multiple partitions. The newly reimag Read more…

Is Time Running Out for Compromise on America COMPETES/USICA Act?

June 22, 2022

You may recall that efforts proposed in 2020 to remake the National Science Foundation (Endless Frontier Act) have since expanded and morphed into two gigantic bills, the America COMPETES Act in the U.S. House of Representatives and the U.S. Innovation and Competition Act in the U.S. Senate. So far, efforts to reconcile the two pieces of legislation have snagged and recent reports... Read more…

Cerebras Systems Thinks Forward on AI Chips as it Claims Performance Win

June 22, 2022

Cerebras Systems makes the largest chip in the world, but is already thinking about its upcoming AI chips as learning models continue to grow at breakneck speed. The company’s latest Wafer Scale Engine chip is indeed the size of a wafer, and is made using TSMC’s 7nm process. The next chip will pack in more cores to handle the fast-growing compute needs of AI, said Andrew Feldman, CEO of Cerebras Systems. Read more…

AMD’s MI300 APUs to Power Exascale El Capitan Supercomputer

June 21, 2022

Additional details of the architecture of the exascale El Capitan supercomputer were disclosed today by Lawrence Livermore National Laboratory’s (LLNL) Terri Read more…

IDC Perspective on Integration of Quantum Computing and HPC

June 20, 2022

The insatiable need to compress time to insights from massive and complex datasets is fueling the demand for quantum computing integration into high performance computing (HPC) environments. Such an integration would allow enterprises to accelerate and optimize current HPC applications and processes by simulating and emulating them on today’s noisy... Read more…

Q&A with Intel’s Jeff McVeigh, an HPCwire Person to Watch in 2022

June 17, 2022

HPCwire presents our interview with Jeff McVeigh, vice president and general manager, Super Compute Group, Intel Corporation, and an HPCwire 2022 Person to Watc Read more…

Nvidia, Intel to Power Atos-Built MareNostrum 5 Supercomputer

June 16, 2022

The long-troubled, hotly anticipated MareNostrum 5 supercomputer finally has a vendor: Atos, which will be supplying a system that includes both Nvidia and Inte Read more…

D-Wave Debuts Advantage2 Prototype; Seeks User Exploration and Feedback

June 16, 2022

Starting today, D-Wave Systems is providing access to a 500-plus-qubit prototype of its forthcoming 7000-qubit Advantage2 quantum annealing computer, which is d Read more…

AMD Opens Up Chip Design to the Outside for Custom Future

June 15, 2022

AMD is getting personal with chips as it sets sail to make products more to the liking of its customers. The chipmaker detailed a modular chip future in which customers can mix and match non-AMD processors in a custom chip package. "We are focused on making it easier to implement chips with more flexibility," said Mark Papermaster, chief technology officer at AMD during the analyst day meeting late last week. Read more…

Nvidia R&D Chief on How AI is Improving Chip Design

April 18, 2022

Getting a glimpse into Nvidia’s R&D has become a regular feature of the spring GTC conference with Bill Dally, chief scientist and senior vice president of research, providing an overview of Nvidia’s R&D organization and a few details on current priorities. This year, Dally focused mostly on AI tools that Nvidia is both developing and using in-house to improve... Read more…

Royalty-free stock illustration ID: 1919750255

Intel Says UCIe to Outpace PCIe in Speed Race

May 11, 2022

Intel has shared more details on a new interconnect that is the foundation of the company’s long-term plan for x86, Arm and RISC-V architectures to co-exist in a single chip package. The semiconductor company is taking a modular approach to chip design with the option for customers to cram computing blocks such as CPUs, GPUs and AI accelerators inside a single chip package. Read more…

The Final Frontier: US Has Its First Exascale Supercomputer

May 30, 2022

In April 2018, the U.S. Department of Energy announced plans to procure a trio of exascale supercomputers at a total cost of up to $1.8 billion dollars. Over the ensuing four years, many announcements were made, many deadlines were missed, and a pandemic threw the world into disarray. Now, at long last, HPE and Oak Ridge National Laboratory (ORNL) have announced that the first of those... Read more…

AMD/Xilinx Takes Aim at Nvidia with Improved VCK5000 Inferencing Card

March 8, 2022

AMD/Xilinx has released an improved version of its VCK5000 AI inferencing card along with a series of competitive benchmarks aimed directly at Nvidia’s GPU line. AMD says the new VCK5000 has 3x better performance than earlier versions and delivers 2x TCO over Nvidia T4. AMD also showed favorable benchmarks against several Nvidia GPUs, claiming its VCK5000 achieved... Read more…

Top500: Exascale Is Officially Here with Debut of Frontier

May 30, 2022

The 59th installment of the Top500 list, issued today from ISC 2022 in Hamburg, Germany, officially marks a new era in supercomputing with the debut of the first-ever exascale system on the list. Frontier, deployed at the Department of Energy’s Oak Ridge National Laboratory, achieved 1.102 exaflops in its fastest High Performance Linpack run, which was completed... Read more…

Newly-Observed Higgs Mode Holds Promise in Quantum Computing

June 8, 2022

The first-ever appearance of a previously undetectable quantum excitation known as the axial Higgs mode – exciting in its own right – also holds promise for developing and manipulating higher temperature quantum materials... Read more…

Nvidia Launches Hopper H100 GPU, New DGXs and Grace Superchips

March 22, 2022

The battle for datacenter dominance keeps getting hotter. Today, Nvidia kicked off its spring GTC event with new silicon, new software and a new supercomputer. Speaking from a virtual environment in the Nvidia Omniverse 3D collaboration and simulation platform, CEO Jensen Huang introduced the new Hopper GPU architecture and the H100 GPU... Read more…

PsiQuantum’s Path to 1 Million Qubits

April 21, 2022

PsiQuantum, founded in 2016 by four researchers with roots at Bristol University, Stanford University, and York University, is one of a few quantum computing startups that’s kept a moderately low PR profile. (That’s if you disregard the roughly $700 million in funding it has attracted.) The main reason is PsiQuantum has eschewed the clamorous public chase for... Read more…

Leading Solution Providers

Contributors

ISC 2022 Booth Video Tours

AMD
AWS
DDN
Dell
Intel
Lenovo
Microsoft
PENGUIN SOLUTIONS

AMD Opens Up Chip Design to the Outside for Custom Future

June 15, 2022

AMD is getting personal with chips as it sets sail to make products more to the liking of its customers. The chipmaker detailed a modular chip future in which customers can mix and match non-AMD processors in a custom chip package. "We are focused on making it easier to implement chips with more flexibility," said Mark Papermaster, chief technology officer at AMD during the analyst day meeting late last week. Read more…

Intel Reiterates Plans to Merge CPU, GPU High-performance Chip Roadmaps

May 31, 2022

Intel reiterated it is well on its way to merging its roadmap of high-performance CPUs and GPUs as it shifts over to newer manufacturing processes and packaging technologies in the coming years. The company is merging the CPU and GPU lineups into a chip (codenamed Falcon Shores) which Intel has dubbed an XPU. Falcon Shores... Read more…

India Launches Petascale ‘PARAM Ganga’ Supercomputer

March 8, 2022

Just a couple of weeks ago, the Indian government promised that it had five HPC systems in the final stages of installation and would launch nine new supercomputers this year. Now, it appears to be making good on that promise: the country’s National Supercomputing Mission (NSM) has announced the deployment of “PARAM Ganga” petascale supercomputer at Indian Institute of Technology (IIT)... Read more…

Nvidia Dominates MLPerf Inference, Qualcomm also Shines, Where’s Everybody Else?

April 6, 2022

MLCommons today released its latest MLPerf inferencing results, with another strong showing by Nvidia accelerators inside a diverse array of systems. Roughly fo Read more…

AMD’s MI300 APUs to Power Exascale El Capitan Supercomputer

June 21, 2022

Additional details of the architecture of the exascale El Capitan supercomputer were disclosed today by Lawrence Livermore National Laboratory’s (LLNL) Terri Read more…

Nvidia, Intel to Power Atos-Built MareNostrum 5 Supercomputer

June 16, 2022

The long-troubled, hotly anticipated MareNostrum 5 supercomputer finally has a vendor: Atos, which will be supplying a system that includes both Nvidia and Inte Read more…

Industry Consortium Forms to Drive UCIe Chiplet Interconnect Standard

March 2, 2022

A new industry consortium aims to establish a die-to-die interconnect standard – Universal Chiplet Interconnect Express (UCIe) – in support of an open chipl Read more…

Covid Policies at HPC Conferences Should Reflect HPC Research

June 6, 2022

Supercomputing has been indispensable throughout the Covid-19 pandemic, from modeling the virus and its spread to designing vaccines and therapeutics. But, desp Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire