The Green500’s Crystal Anniversary Sees MN-3 Crystallize Its Winning Streak

By Oliver Peckham

December 2, 2021

“This is the 30th Green500,” said Wu Feng, custodian of the Green500 list, at the list’s SC21 birds-of-a-feather session. “You could say 15 years of Green500, which makes it, I guess, the crystal anniversary.” Indeed, HPCwire marked the 15th anniversary of the Green500 – which ranks supercomputers by flops-per-watt, rather than just by flops – earlier this year with its 29th list. But strangely, six months later, quite a bit has changed in the landscape: the exascale era is (somewhat) officially here; the list has reached a significant milestone; and its reigning champion is doing anything but resting on its laurels.

Exascale growing pains

Looming over the Top500 list this year are the conspicuous absences of the two now-confirmed Chinese exascale systems, Sunway’s OceanLight (reportedly landing at 1.3 exaflops peak) and Tianhe-3 (reportedly landing at 1.7 exaflops peak). But these systems didn’t dominate the Top500, instead sitting it out entirely, sparking concerns from the HPC community: if the top two systems don’t report to the Top500, how useful can the list be?

The Green500 – which began to operate under the umbrella of the Top500 some years ago – accordingly suffered the same fate this cycle, with both of the now-known exascale systems absent from that list as well. While the Green500 historically does not have the same level of participation as the Top500, the omission of the exascale systems still stings for reasons beyond the optics of participation.

The Green500 organizers are consistently interested in the flops-per-watt of not just the most efficient systems, but also the most powerful systems; Fugaku, for instance, is the current top-ranked system on the Top500 and places relatively well (26th) on the Green500 – but its November rating of 15.4 gigaflops per watt falls well short of Green500-topper MN-3, which delivers more than twice the efficiency at 39.4 gigaflops per watt. Without knowing the efficiency of the world’s most powerful systems, though, it becomes harder to suss out the divide between the most powerful systems and the most efficient systems.

Approaching key thresholds

Nevertheless, MN-3’s chart-topping achievement – nearly 40 gigaflops per watt – marks a significant waypoint for supercomputer efficiency. “If you were to naively extrapolate [that efficiency] linearly to an exascale supercomputer, you’d get a 25-megawatt supercomputer for an exaflop,” Feng said. “If you reach … 50 gigaflops per watt – which, right now we’re almost at 40 gigaflops per watt – the 50 gigaflops per watt signifies being able to have an exascale supercomputer in a 20-megawatt envelope.” This 20-megawatt envelope has long been an ambitious target of the U.S.’ plans for exascale, and one that Feng had, to date, been quite bearish about.

At SC21, he sounded decidedly more optimistic. “Back in 2008, the projected power extrapolation [for exascale systems] was on the order of gigawatts,” Feng said. “But you can see how it’s come down, and right now the MN-3 … extrapolated out for an exascale supercomputer would come in at 25.4 megawatts.” (It should also be noted that about a month after the June Green500 list, HPCwire reported that the U.S.’ first exascale system, Frontier, is likely to meet the 20-megawatt target.)

Still, he noted, the top ten systems on the Top500 were “quite a ways away from exascale in either a 40-, 30-, or 20-megawatt envelope” – and the median of the Green500 list was far from those targets, too. “The rate at which the energy efficiency is improving at the top end of the Green500 list is much faster than what you see the median is improving,” Feng said.

Image courtesy of Wu Feng.

This growth in efficiency has largely been driven by accelerated systems. “If you look at 2021, the median energy efficiency of a heterogeneous accelerated system is far outstripping the energy efficiency of homogeneous supercomputers,” Feng explained, adding that “ten out of ten in the top ten” Green500 systems were accelerated. AMD is also continuing its broad winning streak, nabbing eight of the top ten spots.

The dominance of MN-3

MN-3, though, is one of the two Intel systems in the top ten – but the system, operated by Preferred Networks, relies predominantly on the MN-Core accelerator, a custom accelerator designed by Preferred Networks for deep learning (and manufactured on TSMC’s 12nm process) that delivers a theoretical peak of 32.8 double-precision teraflops (131 single-precision or 524 half-precision teraflops) in a 500-watt wrapper.

Image courtesy of Yusuke Doi.

MN-3, which contains nearly 200 of those MN-Core boards, is not a newcomer to the Green500 list – far from it. The system, which initially reported a Linpack rating of 1.6 petaflops for the Top500 in June 2020, topped that cycle’s Green500 list with (at the time) 21.1 gigaflops per watt. That November, it slipped to second place, but increased its reported efficiency to 26.04 gigaflops per watt. By June 2021, it was back on top of the list with 29.7; and by last month, it had demolished its previous records with 39.4, easily winning the crown for a third time. Over that same time, its Linpack rating increased from that initial 1.6 Linpack petaflops to 2.2, and it jumped from 393rd on the Top500 to 301st.

Image courtesy of Yusuke Doi.

In an interview with HPCwire at SC21, Yusuke Doi – vice president of computing infrastructure for Preferred Networks – attributed the system’s dramatic improvements to the learning process inherent in working with new, in-house hardware. “If you use an Nvidia GPU, you have a lot of experience on that architecture,” he said. “Our architecture is the very first one, so we have to know about the processor itself to make it run efficiently. So it takes time!” In fact, he said, he was surprised when MN-3 placed in the top spot on the Green500 that early in its lifetime, when its efficiency was still – in their eyes – relatively low.

Preferred Networks – a newcomer to the scene – is still working out how to leverage MN-3 and the MN-Core accelerator beyond in-house use for deep learning. “We need plenty of computing resources [to accelerate our own business],” Doi said. “Our [business] strategy is not yet decided. One reason we came here [to SC21] is to find business strategies that fit to this field, because initially we are focusing on our internal usage, [but] we are open to people who want to use this kind of accelerator.”

But no, Preferred Networks won’t just keep improving MN-3 ad infinitum: they expect that their next accelerator, MN-Core 2, will reach running state in 2023, at which time Doi said they would have a “totally different system” to showcase the new hardware.

The future of the Green500

From the beginning, the Green500 has been aimed at “raising the awareness and encouraging the reporting of energy efficiency of supercomputers,” Feng said. “We wanted to do this to the point that we were driving energy efficiency – or greenness, if you will – as a first-order design constraint that’s on par with performance.”

Over time, the Green500 organizers and the Energy-Efficient HPC Working Group (EEHPCWG) have worked together to evolve the Green500’s reporting methodology in service of those goals. The Green500 now uses three levels of reporting, from level one (which requires only average power use across one phase on part of a system) to level three (which requires full, real energy use across the entire run on the whole system).

Feng reported that eight sites made higher-quality submissions this year, including high-profile submitters like Lawrence Berkeley National Laboratory and JAMSTEC. Preferred Networks, for its part, initially reported at level two but opted to increase its reporting to level three. “At that time, we were not satisfied because we got number one position at that time [and] we thought the number one system should be a more precise measurement,” Doi said.

Now, the Green500 seems pulled in two directions: in the face of major systems opting out of even the Top500, the less-ubiquitous Green500 might feel a need to allow low-difficulty reporting wherever possible; but on the other hand, some of its more prominent adopters have been calling for more intensive requirements, such as carbon footprint reporting or mandatory power reporting for Top500 submissions.

For the time being, it is unclear which direction the Green500 will lean toward. The EEHPCWG is planning to begin a process of reviewing – and possibly revising – the Green500 methodology based on the recommendations for improvements received over the past years. “We’re hoping that the revised document will ultimately be adopted by the Green500 and the Top500,” said Natalie Bates, lead of the EEHPCWG.

Header image: Wu Feng (left) presenting a Green500 winner certificate to Yusuke Doi (right) at SC21.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Supercomputer Analysis Shows the Atmospheric Reach of the Tonga Eruption

January 21, 2022

On Saturday, an enormous eruption on the volcanic islands of Hunga Tonga and Hunga Haʻapai shook the Pacific Ocean. The explosion, which could be heard six thousand miles away in Alaska, caused tsunamis across the entir Read more…

NSB Issues US State of Science and Engineering 2022 Report

January 20, 2022

This week the National Science Board released its biannual U.S. State of Science and Engineering 2022 report, as required by the NSF Act. Broadly, the report presents a near-term view of S&E based mostly on 2019 data. To a large extent, this year’s edition echoes trends from the last few reports. The U.S. is still a world leader in R&D spending and S&E education... Read more…

Researchers Achieve 99 Percent Quantum Accuracy with Silicon-Embedded Qubits 

January 20, 2022

Researchers in Australia and the U.S. have made exciting headway in the quantum computing arms race. A multi-institutional team including the University of New South Wales and Sandia National Laboratory announced that th Read more…

Trio of Supercomputers Powers Estimate of Carbon in Earth’s Outer Core

January 20, 2022

Carbon is one of the essential building blocks of life on Earth, and it—along with hydrogen, nitrogen and oxygen—is one of the key elements researchers look for when they search for habitable planets and work to unde Read more…

Multiverse Targets ‘Quantum Computing for the Masses’

January 19, 2022

The race to deliver quantum computing solutions that shield users from the underlying complexity of quantum computing is heating up quickly. One example is Multiverse Computing, a European company, which today launched the second financial services product in its Singularity product group. The new offering, Fair Price, “delivers a higher accuracy in fair price calculations for financial... Read more…

AWS Solution Channel

shutterstock 718231072

Accelerating drug discovery with Amazon EC2 Spot Instances

This post was contributed by Cristian Măgherușan-Stanciu, Sr. Specialist Solution Architect, EC2 Spot, with contributions from Cristian Kniep, Sr. Developer Advocate for HPC and AWS Batch at AWS, Carlos Manzanedo Rueda, Principal Solutions Architect, EC2 Spot at AWS, Ludvig Nordstrom, Principal Solutions Architect at AWS, Vytautas Gapsys, project group leader at the Max Planck Institute for Biophysical Chemistry, and Carsten Kutzner, staff scientist at the Max Planck Institute for Biophysical Chemistry. Read more…

Students at SC21: Out in Front, Alongside and Behind the Scenes

January 19, 2022

The Supercomputing Conference (SC) is one of the biggest international conferences dedicated to high-performance computing, networking, storage and analysis. SC21 was a true ‘hybrid’ conference, with a total of 380 o Read more…

Supercomputer Analysis Shows the Atmospheric Reach of the Tonga Eruption

January 21, 2022

On Saturday, an enormous eruption on the volcanic islands of Hunga Tonga and Hunga Haʻapai shook the Pacific Ocean. The explosion, which could be heard six tho Read more…

NSB Issues US State of Science and Engineering 2022 Report

January 20, 2022

This week the National Science Board released its biannual U.S. State of Science and Engineering 2022 report, as required by the NSF Act. Broadly, the report presents a near-term view of S&E based mostly on 2019 data. To a large extent, this year’s edition echoes trends from the last few reports. The U.S. is still a world leader in R&D spending and S&E education... Read more…

Multiverse Targets ‘Quantum Computing for the Masses’

January 19, 2022

The race to deliver quantum computing solutions that shield users from the underlying complexity of quantum computing is heating up quickly. One example is Multiverse Computing, a European company, which today launched the second financial services product in its Singularity product group. The new offering, Fair Price, “delivers a higher accuracy in fair price calculations for financial... Read more…

Students at SC21: Out in Front, Alongside and Behind the Scenes

January 19, 2022

The Supercomputing Conference (SC) is one of the biggest international conferences dedicated to high-performance computing, networking, storage and analysis. SC Read more…

Q-Ctrl – Tackling Quantum Hardware’s Noise Problems with Software

January 13, 2022

Implementing effective error mitigation and correction is a critical next step in advancing quantum computing. While a lot of attention has been given to effort Read more…

Nvidia Defends Arm Acquisition Deal: a ‘Once-in-a-Generation Opportunity’

January 13, 2022

GPU-maker Nvidia is continuing to try to keep its proposed acquisition of British chip IP vendor Arm Ltd. alive, despite continuing concerns from several governments around the world. In its latest action, Nvidia filed a 29-page response to the U.K. government to point out a list of potential benefits of the proposed $40 billion deal. Read more…

Nvidia Buys HPC Cluster Management Company Bright Computing

January 10, 2022

Graphics chip powerhouse Nvidia today announced that it has acquired HPC cluster management company Bright Computing for an undisclosed sum. Unlike Nvidia’s bid to purchase semiconductor IP company Arm, which has been stymied by regulatory challenges, the Bright deal is a straightforward acquisition that aims to expand... Read more…

SC21 Panel on Programming Models – Tackling Data Movement, DSLs, More

January 6, 2022

How will programming future systems differ from current practice? This is an ever-present question in computing. Yet it has, perhaps, never been more pressing g Read more…

IonQ Is First Quantum Startup to Go Public; Will It be First to Deliver Profits?

November 3, 2021

On October 1 of this year, IonQ became the first pure-play quantum computing start-up to go public. At this writing, the stock (NYSE: IONQ) was around $15 and its market capitalization was roughly $2.89 billion. Co-founder and chief scientist Chris Monroe says it was fun to have a few of the company’s roughly 100 employees travel to New York to ring the opening bell of the New York Stock... Read more…

US Closes in on Exascale: Frontier Installation Is Underway

September 29, 2021

At the Advanced Scientific Computing Advisory Committee (ASCAC) meeting, held by Zoom this week (Sept. 29-30), it was revealed that the Frontier supercomputer is currently being installed at Oak Ridge National Laboratory in Oak Ridge, Tenn. The staff at the Oak Ridge Leadership... Read more…

AMD Launches Milan-X CPU with 3D V-Cache and Multichip Instinct MI200 GPU

November 8, 2021

At a virtual event this morning, AMD CEO Lisa Su unveiled the company’s latest and much-anticipated server products: the new Milan-X CPU, which leverages AMD’s new 3D V-Cache technology; and its new Instinct MI200 GPU, which provides up to 220 compute units across two Infinity Fabric-connected dies, delivering an astounding 47.9 peak double-precision teraflops. “We're in a high-performance computing megacycle, driven by the growing need to deploy additional compute performance... Read more…

Intel Reorgs HPC Group, Creates Two ‘Super Compute’ Groups

October 15, 2021

Following on changes made in June that moved Intel’s HPC unit out of the Data Platform Group and into the newly created Accelerated Computing Systems and Graphics (AXG) business unit, led by Raja Koduri, Intel is making further updates to the HPC group and announcing... Read more…

Nvidia Buys HPC Cluster Management Company Bright Computing

January 10, 2022

Graphics chip powerhouse Nvidia today announced that it has acquired HPC cluster management company Bright Computing for an undisclosed sum. Unlike Nvidia’s bid to purchase semiconductor IP company Arm, which has been stymied by regulatory challenges, the Bright deal is a straightforward acquisition that aims to expand... Read more…

D-Wave Embraces Gate-Based Quantum Computing; Charts Path Forward

October 21, 2021

Earlier this month D-Wave Systems, the quantum computing pioneer that has long championed quantum annealing-based quantum computing (and sometimes taken heat fo Read more…

Killer Instinct: AMD’s Multi-Chip MI200 GPU Readies for a Major Global Debut

October 21, 2021

AMD’s next-generation supercomputer GPU is on its way – and by all appearances, it’s about to make a name for itself. The AMD Radeon Instinct MI200 GPU (a successor to the MI100) will, over the next year, begin to power three massive systems on three continents: the United States’ exascale Frontier system; the European Union’s pre-exascale LUMI system; and Australia’s petascale Setonix system. Read more…

Three Chinese Exascale Systems Detailed at SC21: Two Operational and One Delayed

November 24, 2021

Details about two previously rumored Chinese exascale systems came to light during last week’s SC21 proceedings. Asked about these systems during the Top500 media briefing on Monday, Nov. 15, list author and co-founder Jack Dongarra indicated he was aware of some very impressive results, but withheld comment when asked directly if he had... Read more…

Leading Solution Providers

Contributors

Lessons from LLVM: An SC21 Fireside Chat with Chris Lattner

December 27, 2021

Today, the LLVM compiler infrastructure world is essentially inescapable in HPC. But back in the 2000 timeframe, LLVM (low level virtual machine) was just getting its start as a new way of thinking about how to overcome shortcomings in the Java Virtual Machine. At the time, Chris Lattner was a graduate student of... Read more…

2021 Gordon Bell Prize Goes to Exascale-Powered Quantum Supremacy Challenge

November 18, 2021

Today at the hybrid virtual/in-person SC21 conference, the organizers announced the winners of the 2021 ACM Gordon Bell Prize: a team of Chinese researchers leveraging the new exascale Sunway system to simulate quantum circuits. The Gordon Bell Prize, which comes with an award of $10,000 courtesy of HPC pioneer Gordon Bell, is awarded annually... Read more…

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from Julialang.org. I Read more…

Top500: No Exascale, Fugaku Still Reigns, Polaris Debuts at #12

November 15, 2021

No exascale for you* -- at least, not within the High-Performance Linpack (HPL) territory of the latest Top500 list, issued today from the 33rd annual Supercomputing Conference (SC21), held in-person in St. Louis, Mo., and virtually, from Nov. 14–19. "We were hoping to have the first exascale system on this list but that didn’t happen," said Top500 co-author... Read more…

TACC Unveils Lonestar6 Supercomputer

November 1, 2021

The Texas Advanced Computing Center (TACC) is unveiling its latest supercomputer: Lonestar6, a three peak petaflops Dell system aimed at supporting researchers Read more…

Nvidia Defends Arm Acquisition Deal: a ‘Once-in-a-Generation Opportunity’

January 13, 2022

GPU-maker Nvidia is continuing to try to keep its proposed acquisition of British chip IP vendor Arm Ltd. alive, despite continuing concerns from several governments around the world. In its latest action, Nvidia filed a 29-page response to the U.K. government to point out a list of potential benefits of the proposed $40 billion deal. Read more…

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

Intel Launches 10nm ‘Ice Lake’ Datacenter CPU with Up to 40 Cores

April 6, 2021

The wait is over. Today Intel officially launched its 10nm datacenter CPU, the third-generation Intel Xeon Scalable processor, codenamed Ice Lake. With up to 40 Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire