At ISC, the Green500 Witnesses a New Frontier in Efficient Computing

By Oliver Peckham

June 8, 2022

Back in 2008, the U.S. Defense Advanced Research Projects Agency (DARPA) set an ambitious target: an exascale supercomputer in a 20-megawatt envelope. That target, once viewed by many with skepticism—research at the time predicted that exascale systems would require hundreds of megawatts—has now officially been met by the exascale Frontier supercomputer at Oak Ridge National Laboratory (ORNL). At ISC 2022, the organizers of the Green500 list—which ranks supercomputers based on their flops-per-watt efficiency—discussed this development and more.

A new Frontier for efficiency

The “June” (late May, actually) Green500 list was led by ORNL. In first place: Frontier’s test and development system, Frontier TDS—though we prefer its less official name, “Borg.” Borg (which is effectively just a single cabinet of the same design as Frontier’s 74 main cabinets) delivered 62.68 gigaflops per watt at a total of 19.20 Linpack petaflops. “If you were to naively extrapolate this to an exaflop, it comes in at about 16 megawatts,” said Wu Feng, custodian of the Green500 list and an associate professor at Virginia Tech, during his virtual appearance at ISC 2022. This is a staggering accomplishment of computing efficiency, eclipsing the previous Green500 champion—Preferred Networks’ MN-3—by nearly 60 percent.

Representatives accept the certificate for Frontier TDS/Borg’s Green500 win at ISC 2022. On the left, Rafael Ferreira da Silva, senior research scientist for data lifecycle and scalable workflows at ORNL; on the right, Nicolas Dube, fellow, chief technologist for HPC and chief strategist for HPC at HPE.

Perhaps more impressive, however, is that Frontier itself placed second with 52.23 gigaflops per watt. “Frontier on the Green500 is the highest-placed number-one Top500 supercomputer on the Green500 list in its existence,” Feng said. According to the Green500 list, Frontier delivered 1.102 Linpack exaflops in a 21.1-megawatt envelope, which interpolates to one exaflop at 19.15 megawatts. However, Al Geist—CTO of the Oak Ridge Leadership Computing Facility (OLCF)—revealed during the session that this was a “very conservative number” and that the average power use that Oak Ridge submitted to the Green500 was actually 20.2 megawatts. That works out to 54.5 gigaflops per watt and interpolates to an exaflop in 18.33 megawatts. By this measurement, Frontier is more than 3.5× more efficient than the previous Top500 topper, Riken’s Fugaku system (15.42 gigaflops per watt).

The Frontier supercomputer. Image courtesy of HPE/ORNL.

This efficiency, Geist explained, speaks to a long legacy at ORNL. “Oak Ridge has really been working on energy efficient computing for about a decade,” he said, charting out how this ten-year effort had paid off from the use of GPUs in the Titan system back in 2012 through Frontier today. “Exascale has really been made possible by this sort of 200× improvement in energy-efficient computing.” Geist further credited AMD’s work into making its CPUs and GPUs more efficient, such as by allowing the chips to turn off unused resources at a very granular level. He also credited the list itself: “I think the Green500 has done a remarkable job of making the entire community much more aware of power efficiency and the importance of it.”

Frontier’s shadow even extends beyond the top two systems on the Green500. Frontier—and, by extension, Borg—are HPE Cray EX systems with AMD Milan “Trento” Epyc CPUs, AMD Instinct MI250X GPUs and HPE Slingshot-11 networking. That exact same architecture also appears in the third-place system, the 151.90 Linpack petaflops LUMI supercomputer in Finland (51.63 gigaflops per watt, third place on the Top500). It also appears in the fourth-place system, the 46.10 Linpack petaflops Adastra system in France (50.03 gigaflops per watt, tenth place on the Top500). “All four of these systems all use the same technology that was actually developed for Frontier,” Geist said. Both LUMI and Adastra also extrapolate to an exaflop under 20 megawatts.

The new Green500 top ten, with new entries and measurements in yellow. Image courtesy of Wu Feng.

Green500 trends

All of the top ten systems are accelerated: four with the aforementioned AMD MI250X GPUs, five with Nvidia’s A100 GPUs and one between them in fifth place using the Preferred Networks MN-Core accelerator. Further, Feng said, it was the first time that all of the top ten machines from the previous list stayed on the list—and not just on the list, but in the top 20. However, those four Frontier-type systems shot past the rest of the pack on the list: the average power efficiency of the top ten systems extrapolates to exascale at around 40 megawatts, showcasing the gap between the Frontier architecture and the competition. As shown in the box-and-whisker plot below, the remaining systems on the Green500 list showed modest improvements in efficiency compared to the November list.

Gigaflops per watt over time on the Green500 list. Image courtesy of Wu Feng.

There was another encouraging trend on the new list. The Green500 uses three tiers of efficiency reporting, with a level one measurement representing the whole system across a full run, a level three measurement representing a smaller fraction of the system across the core phase of a run, and a level two measurement somewhere in-between. “The total number of level 2 and level 3 entries continues to grow relative to level 1, so that’s really great,” said Natalie Bates, chair of the Energy Efficient HPC Working Group (EEHPCWG), during the Green500 session. This Green500 list included 102 measured submissions: 57 at level one, 31 at level two and 14 at level three.

Higher stakes, new strategies

Founded 16 years ago, the Green500 list aims to “raise awareness (and encourage reporting) of the energy efficiency of supercomputers” and to “drive energy efficiency as a first-order design constraint (on par with performance).” But when the Green500 list was being conceived, supercomputers rated in single-digit kilowatts; now, systems like Frontier are pulling down double-digit megawatts. ORNL Director Thomas Zacharia said in a press briefing that “when you start the [Linpack] run [on Frontier], the machine, in less than ten seconds, begins to draw an additional 15 megawatts of power … that’s a small city in the U.S., that’s roughly about how much power the city of Oak Ridge consumes.”

The sheer scale of systems like Frontier has put increased urgency on not only how much power the systems themselves consume, but also the efficiency of their supporting infrastructure and the sourcing of the power itself. Indeed, DARPA’s 20-megawatt target for exascale was predicated on costs, as Geist recounted during ORNL’s Advanced Technologies Section webinar last year: “The number that came back from the head of [the] Office of Science at the time was that they weren’t willing to pay over $100 million over the five years, so it’s simple math [based on an average cost of $1 million per megawatt per year]. The 20 megawatts had nothing to do with what might be possible, it was just that stake that we drove in the ground.”

In the Green500 session last week, Geist elaborated that Oak Ridge was dedicated to “not only reducing the amount of energy it takes to run the computer, but reducing the amount of energy it takes to cool the datacenter back down.” As a result, the Frontier datacenter achieves a power usage effectiveness (PUE) of just 1.03. “A lot of work has gone into trying to make this machine as well as the datacenter itself just as efficiently as possible,” Geist said.

The new LUMI datacenter. Image courtesy of CSC.

EuroHPC’s aforementioned LUMI system, meanwhile, is housed in a new datacenter designed with power efficiency and sustainability in mind (pictured above). Sited in an old paper mill in Kajaani, Finland, LUMI—which currently requires less than 10 megawatts to operate—is powered by 100 percent renewable energy (local hydropower) and is designed to sell its waste heat back to the town of Kajaani, further reducing energy costs and resulting in a net-negative carbon footprint. The location in northern Finland also, of course, reduces the need for artificial cooling. During a session on EuroHPC at ISC 2022, Anders Jensen—executive director of the EuroHPC JU—stressed the importance of these holistic energy “stories” for European supercomputers. “[The] Green500 is great,” he said, “but it doesn’t take into account where the energy came from.”

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Is Time Running Out for Compromise on America COMPETES/USICA Act?

June 22, 2022

You may recall that efforts proposed in 2020 to remake the National Science Foundation (Endless Frontier Act) have since expanded and morphed into two gigantic bills, the America COMPETES Act in the U.S. House of Representatives and the U.S. Innovation and Competition Act in the U.S. Senate. So far, efforts to reconcile the two pieces of legislation have snagged and recent reports... Read more…

Cerebras Systems Thinks Forward on AI Chips as it Claims Performance Win

June 22, 2022

Cerebras Systems makes the largest chip in the world, but is already thinking about its upcoming AI chips as learning models continue to grow at breakneck speed. The company’s latest Wafer Scale Engine chip is indeed the size of a wafer, and is made using TSMC’s 7nm process. The next chip will pack in more cores to handle the fast-growing compute needs of AI, said Andrew Feldman, CEO of Cerebras Systems. Read more…

AMD’s MI300 APUs to Power Exascale El Capitan Supercomputer

June 21, 2022

Additional details of the architecture of the exascale El Capitan supercomputer were disclosed today by Lawrence Livermore National Laboratory’s (LLNL) Terri Quinn in a presentation delivered to the 79th HPC User Forum Read more…

IDC Perspective on Integration of Quantum Computing and HPC

June 20, 2022

The insatiable need to compress time to insights from massive and complex datasets is fueling the demand for quantum computing integration into high performance computing (HPC) environments. Such an integration would allow enterprises to accelerate and optimize current HPC applications and processes by simulating and emulating them on today’s noisy... Read more…

Q&A with Intel’s Jeff McVeigh, an HPCwire Person to Watch in 2022

June 17, 2022

HPCwire presents our interview with Jeff McVeigh, vice president and general manager, Super Compute Group, Intel Corporation, and an HPCwire 2022 Person to Watch. McVeigh shares Intel's plans for the year ahead, his pers Read more…

AWS Solution Channel

Shutterstock 152995403

Bayesian ML Models at Scale with AWS Batch

This post was contributed by Ampersand’s Jeffrey Enos, Senior Machine Learning Engineer, Daniel Gerlanc, Senior Director for Data Science, and Brandon Willard, Data Science Lead. Read more…

Microsoft/NVIDIA Solution Channel

Shutterstock 261863138

Using Cloud-Based, GPU-Accelerated AI for Financial Risk Management

There are strict rules governing financial institutions with a number of global regulatory groups publishing financial compliance requirements. Financial institutions face many challenges and legal responsibilities for risk management, compliance violations, and failure to catch financial fraud. Read more…

Nvidia, Intel to Power Atos-Built MareNostrum 5 Supercomputer

June 16, 2022

The long-troubled, hotly anticipated MareNostrum 5 supercomputer finally has a vendor: Atos, which will be supplying a system that includes both Nvidia and Intel CPUs and GPUs across multiple partitions. The newly reimag Read more…

Is Time Running Out for Compromise on America COMPETES/USICA Act?

June 22, 2022

You may recall that efforts proposed in 2020 to remake the National Science Foundation (Endless Frontier Act) have since expanded and morphed into two gigantic bills, the America COMPETES Act in the U.S. House of Representatives and the U.S. Innovation and Competition Act in the U.S. Senate. So far, efforts to reconcile the two pieces of legislation have snagged and recent reports... Read more…

Cerebras Systems Thinks Forward on AI Chips as it Claims Performance Win

June 22, 2022

Cerebras Systems makes the largest chip in the world, but is already thinking about its upcoming AI chips as learning models continue to grow at breakneck speed. The company’s latest Wafer Scale Engine chip is indeed the size of a wafer, and is made using TSMC’s 7nm process. The next chip will pack in more cores to handle the fast-growing compute needs of AI, said Andrew Feldman, CEO of Cerebras Systems. Read more…

AMD’s MI300 APUs to Power Exascale El Capitan Supercomputer

June 21, 2022

Additional details of the architecture of the exascale El Capitan supercomputer were disclosed today by Lawrence Livermore National Laboratory’s (LLNL) Terri Read more…

IDC Perspective on Integration of Quantum Computing and HPC

June 20, 2022

The insatiable need to compress time to insights from massive and complex datasets is fueling the demand for quantum computing integration into high performance computing (HPC) environments. Such an integration would allow enterprises to accelerate and optimize current HPC applications and processes by simulating and emulating them on today’s noisy... Read more…

Q&A with Intel’s Jeff McVeigh, an HPCwire Person to Watch in 2022

June 17, 2022

HPCwire presents our interview with Jeff McVeigh, vice president and general manager, Super Compute Group, Intel Corporation, and an HPCwire 2022 Person to Watc Read more…

Nvidia, Intel to Power Atos-Built MareNostrum 5 Supercomputer

June 16, 2022

The long-troubled, hotly anticipated MareNostrum 5 supercomputer finally has a vendor: Atos, which will be supplying a system that includes both Nvidia and Inte Read more…

D-Wave Debuts Advantage2 Prototype; Seeks User Exploration and Feedback

June 16, 2022

Starting today, D-Wave Systems is providing access to a 500-plus-qubit prototype of its forthcoming 7000-qubit Advantage2 quantum annealing computer, which is d Read more…

AMD Opens Up Chip Design to the Outside for Custom Future

June 15, 2022

AMD is getting personal with chips as it sets sail to make products more to the liking of its customers. The chipmaker detailed a modular chip future in which customers can mix and match non-AMD processors in a custom chip package. "We are focused on making it easier to implement chips with more flexibility," said Mark Papermaster, chief technology officer at AMD during the analyst day meeting late last week. Read more…

Nvidia R&D Chief on How AI is Improving Chip Design

April 18, 2022

Getting a glimpse into Nvidia’s R&D has become a regular feature of the spring GTC conference with Bill Dally, chief scientist and senior vice president of research, providing an overview of Nvidia’s R&D organization and a few details on current priorities. This year, Dally focused mostly on AI tools that Nvidia is both developing and using in-house to improve... Read more…

Royalty-free stock illustration ID: 1919750255

Intel Says UCIe to Outpace PCIe in Speed Race

May 11, 2022

Intel has shared more details on a new interconnect that is the foundation of the company’s long-term plan for x86, Arm and RISC-V architectures to co-exist in a single chip package. The semiconductor company is taking a modular approach to chip design with the option for customers to cram computing blocks such as CPUs, GPUs and AI accelerators inside a single chip package. Read more…

The Final Frontier: US Has Its First Exascale Supercomputer

May 30, 2022

In April 2018, the U.S. Department of Energy announced plans to procure a trio of exascale supercomputers at a total cost of up to $1.8 billion dollars. Over the ensuing four years, many announcements were made, many deadlines were missed, and a pandemic threw the world into disarray. Now, at long last, HPE and Oak Ridge National Laboratory (ORNL) have announced that the first of those... Read more…

AMD/Xilinx Takes Aim at Nvidia with Improved VCK5000 Inferencing Card

March 8, 2022

AMD/Xilinx has released an improved version of its VCK5000 AI inferencing card along with a series of competitive benchmarks aimed directly at Nvidia’s GPU line. AMD says the new VCK5000 has 3x better performance than earlier versions and delivers 2x TCO over Nvidia T4. AMD also showed favorable benchmarks against several Nvidia GPUs, claiming its VCK5000 achieved... Read more…

Top500: Exascale Is Officially Here with Debut of Frontier

May 30, 2022

The 59th installment of the Top500 list, issued today from ISC 2022 in Hamburg, Germany, officially marks a new era in supercomputing with the debut of the first-ever exascale system on the list. Frontier, deployed at the Department of Energy’s Oak Ridge National Laboratory, achieved 1.102 exaflops in its fastest High Performance Linpack run, which was completed... Read more…

Newly-Observed Higgs Mode Holds Promise in Quantum Computing

June 8, 2022

The first-ever appearance of a previously undetectable quantum excitation known as the axial Higgs mode – exciting in its own right – also holds promise for developing and manipulating higher temperature quantum materials... Read more…

Nvidia Launches Hopper H100 GPU, New DGXs and Grace Superchips

March 22, 2022

The battle for datacenter dominance keeps getting hotter. Today, Nvidia kicked off its spring GTC event with new silicon, new software and a new supercomputer. Speaking from a virtual environment in the Nvidia Omniverse 3D collaboration and simulation platform, CEO Jensen Huang introduced the new Hopper GPU architecture and the H100 GPU... Read more…

PsiQuantum’s Path to 1 Million Qubits

April 21, 2022

PsiQuantum, founded in 2016 by four researchers with roots at Bristol University, Stanford University, and York University, is one of a few quantum computing startups that’s kept a moderately low PR profile. (That’s if you disregard the roughly $700 million in funding it has attracted.) The main reason is PsiQuantum has eschewed the clamorous public chase for... Read more…

Leading Solution Providers

Contributors

ISC 2022 Booth Video Tours

AMD
AWS
DDN
Dell
Intel
Lenovo
Microsoft
PENGUIN SOLUTIONS

Intel Reiterates Plans to Merge CPU, GPU High-performance Chip Roadmaps

May 31, 2022

Intel reiterated it is well on its way to merging its roadmap of high-performance CPUs and GPUs as it shifts over to newer manufacturing processes and packaging technologies in the coming years. The company is merging the CPU and GPU lineups into a chip (codenamed Falcon Shores) which Intel has dubbed an XPU. Falcon Shores... Read more…

AMD Opens Up Chip Design to the Outside for Custom Future

June 15, 2022

AMD is getting personal with chips as it sets sail to make products more to the liking of its customers. The chipmaker detailed a modular chip future in which customers can mix and match non-AMD processors in a custom chip package. "We are focused on making it easier to implement chips with more flexibility," said Mark Papermaster, chief technology officer at AMD during the analyst day meeting late last week. Read more…

India Launches Petascale ‘PARAM Ganga’ Supercomputer

March 8, 2022

Just a couple of weeks ago, the Indian government promised that it had five HPC systems in the final stages of installation and would launch nine new supercomputers this year. Now, it appears to be making good on that promise: the country’s National Supercomputing Mission (NSM) has announced the deployment of “PARAM Ganga” petascale supercomputer at Indian Institute of Technology (IIT)... Read more…

Nvidia Dominates MLPerf Inference, Qualcomm also Shines, Where’s Everybody Else?

April 6, 2022

MLCommons today released its latest MLPerf inferencing results, with another strong showing by Nvidia accelerators inside a diverse array of systems. Roughly fo Read more…

AMD’s MI300 APUs to Power Exascale El Capitan Supercomputer

June 21, 2022

Additional details of the architecture of the exascale El Capitan supercomputer were disclosed today by Lawrence Livermore National Laboratory’s (LLNL) Terri Read more…

Nvidia, Intel to Power Atos-Built MareNostrum 5 Supercomputer

June 16, 2022

The long-troubled, hotly anticipated MareNostrum 5 supercomputer finally has a vendor: Atos, which will be supplying a system that includes both Nvidia and Inte Read more…

Industry Consortium Forms to Drive UCIe Chiplet Interconnect Standard

March 2, 2022

A new industry consortium aims to establish a die-to-die interconnect standard – Universal Chiplet Interconnect Express (UCIe) – in support of an open chipl Read more…

Covid Policies at HPC Conferences Should Reflect HPC Research

June 6, 2022

Supercomputing has been indispensable throughout the Covid-19 pandemic, from modeling the virus and its spread to designing vaccines and therapeutics. But, desp Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire