“This is the 30th Green500,” said Wu Feng, custodian of the Green500 list, at the list’s SC21 birds-of-a-feather session. “You could say 15 years of Green500, which makes it, I guess, the crystal anniversary.” Indeed, HPCwire marked the 15th anniversary of the Green500 – which ranks supercomputers by flops-per-watt, rather than just by flops – earlier this year with its 29th list. But strangely, six months later, quite a bit has changed in the landscape: the exascale era is (somewhat) officially here; the list has reached a significant milestone; and its reigning champion is doing anything but resting on its laurels.

Exascale growing pains

Looming over the Top500 list this year are the conspicuous absences of the two now-confirmed Chinese exascale systems, Sunway’s OceanLight (reportedly landing at 1.3 exaflops peak) and Tianhe-3 (reportedly landing at 1.7 exaflops peak). But these systems didn’t dominate the Top500, instead sitting it out entirely, sparking concerns from the HPC community: if the top two systems don’t report to the Top500, how useful can the list be?

The Green500 – which began to operate under the umbrella of the Top500 some years ago – accordingly suffered the same fate this cycle, with both of the now-known exascale systems absent from that list as well. While the Green500 historically does not have the same level of participation as the Top500, the omission of the exascale systems still stings for reasons beyond the optics of participation.

The Green500 organizers are consistently interested in the flops-per-watt of not just the most efficient systems, but also the most powerful systems; Fugaku, for instance, is the current top-ranked system on the Top500 and places relatively well (26th) on the Green500 – but its November rating of 15.4 gigaflops per watt falls well short of Green500-topper MN-3, which delivers more than twice the efficiency at 39.4 gigaflops per watt. Without knowing the efficiency of the world’s most powerful systems, though, it becomes harder to suss out the divide between the most powerful systems and the most efficient systems.

Approaching key thresholds

Nevertheless, MN-3’s chart-topping achievement – nearly 40 gigaflops per watt – marks a significant waypoint for supercomputer efficiency. “If you were to naively extrapolate [that efficiency] linearly to an exascale supercomputer, you’d get a 25-megawatt supercomputer for an exaflop,” Feng said. “If you reach … 50 gigaflops per watt – which, right now we’re almost at 40 gigaflops per watt – the 50 gigaflops per watt signifies being able to have an exascale supercomputer in a 20-megawatt envelope.” This 20-megawatt envelope has long been an ambitious target of the U.S.’ plans for exascale, and one that Feng had, to date, been quite bearish about.

At SC21, he sounded decidedly more optimistic. “Back in 2008, the projected power extrapolation [for exascale systems] was on the order of gigawatts,” Feng said. “But you can see how it’s come down, and right now the MN-3 … extrapolated out for an exascale supercomputer would come in at 25.4 megawatts.” (It should also be noted that about a month after the June Green500 list, HPCwire reported that the U.S.’ first exascale system, Frontier, is likely to meet the 20-megawatt target.)

Still, he noted, the top ten systems on the Top500 were “quite a ways away from exascale in either a 40-, 30-, or 20-megawatt envelope” – and the median of the Green500 list was far from those targets, too. “The rate at which the energy efficiency is improving at the top end of the Green500 list is much faster than what you see the median is improving,” Feng said.

This growth in efficiency has largely been driven by accelerated systems. “If you look at 2021, the median energy efficiency of a heterogeneous accelerated system is far outstripping the energy efficiency of homogeneous supercomputers,” Feng explained, adding that “ten out of ten in the top ten” Green500 systems were accelerated. AMD is also continuing its broad winning streak, nabbing eight of the top ten spots.

The dominance of MN-3

MN-3, though, is one of the two Intel systems in the top ten – but the system, operated by Preferred Networks, relies predominantly on the MN-Core accelerator, a custom accelerator designed by Preferred Networks for deep learning (and manufactured on TSMC’s 12nm process) that delivers a theoretical peak of 32.8 double-precision teraflops (131 single-precision or 524 half-precision teraflops) in a 500-watt wrapper.

MN-3, which contains nearly 200 of those MN-Core boards, is not a newcomer to the Green500 list – far from it. The system, which initially reported a Linpack rating of 1.6 petaflops for the Top500 in June 2020, topped that cycle’s Green500 list with (at the time) 21.1 gigaflops per watt. That November, it slipped to second place, but increased its reported efficiency to 26.04 gigaflops per watt. By June 2021, it was back on top of the list with 29.7; and by last month, it had demolished its previous records with 39.4, easily winning the crown for a third time. Over that same time, its Linpack rating increased from that initial 1.6 Linpack petaflops to 2.2, and it jumped from 393rd on the Top500 to 301st.

In an interview with HPCwire at SC21, Yusuke Doi – vice president of computing infrastructure for Preferred Networks – attributed the system’s dramatic improvements to the learning process inherent in working with new, in-house hardware. “If you use an Nvidia GPU, you have a lot of experience on that architecture,” he said. “Our architecture is the very first one, so we have to know about the processor itself to make it run efficiently. So it takes time!” In fact, he said, he was surprised when MN-3 placed in the top spot on the Green500 that early in its lifetime, when its efficiency was still – in their eyes – relatively low.

Preferred Networks – a newcomer to the scene – is still working out how to leverage MN-3 and the MN-Core accelerator beyond in-house use for deep learning. “We need plenty of computing resources [to accelerate our own business],” Doi said. “Our [business] strategy is not yet decided. One reason we came here [to SC21] is to find business strategies that fit to this field, because initially we are focusing on our internal usage, [but] we are open to people who want to use this kind of accelerator.”

But no, Preferred Networks won’t just keep improving MN-3 ad infinitum: they expect that their next accelerator, MN-Core 2, will reach running state in 2023, at which time Doi said they would have a “totally different system” to showcase the new hardware.

The future of the Green500

From the beginning, the Green500 has been aimed at “raising the awareness and encouraging the reporting of energy efficiency of supercomputers,” Feng said. “We wanted to do this to the point that we were driving energy efficiency – or greenness, if you will – as a first-order design constraint that’s on par with performance.”

Over time, the Green500 organizers and the Energy-Efficient HPC Working Group (EEHPCWG) have worked together to evolve the Green500’s reporting methodology in service of those goals. The Green500 now uses three levels of reporting, from level one (which requires only average power use across one phase on part of a system) to level three (which requires full, real energy use across the entire run on the whole system).

Feng reported that eight sites made higher-quality submissions this year, including high-profile submitters like Lawrence Berkeley National Laboratory and JAMSTEC. Preferred Networks, for its part, initially reported at level two but opted to increase its reporting to level three. “At that time, we were not satisfied because we got number one position at that time [and] we thought the number one system should be a more precise measurement,” Doi said.

Now, the Green500 seems pulled in two directions: in the face of major systems opting out of even the Top500, the less-ubiquitous Green500 might feel a need to allow low-difficulty reporting wherever possible; but on the other hand, some of its more prominent adopters have been calling for more intensive requirements, such as carbon footprint reporting or mandatory power reporting for Top500 submissions.

For the time being, it is unclear which direction the Green500 will lean toward. The EEHPCWG is planning to begin a process of reviewing – and possibly revising – the Green500 methodology based on the recommendations for improvements received over the past years. “We’re hoping that the revised document will ultimately be adopted by the Green500 and the Top500,” said Natalie Bates, lead of the EEHPCWG.

Header image: Wu Feng (left) presenting a Green500 winner certificate to Yusuke Doi (right) at SC21.