15 Years Later, the Green500 Continues Its Push for Energy Efficiency as a First-Order Concern in HPC

By Oliver Peckham

July 15, 2021

The Green500 list, which ranks the most energy-efficient supercomputers in the world, has virtually always faced an uphill battle. As Wu Feng – custodian of the Green500 list and an associate professor at Virginia Tech – tells it, “noone” cared about energy efficiency in the early 2000s, when the seeds of the Green500 were being planted. Nevertheless, in 2007, the Green500 launched its first list, ranking the participating supercomputers by their flops per watt. 29 lists and nearly 15 years later, the Green500 organizers (now having joined forces with the Top500) continue to use the list to advocate for a future for supercomputing that treats energy efficiency as a first-order concern, even as organizations hesitate to put their increasingly large systems through the efficiency gauntlet. At ISC 2021, Feng discussed the state of the Green500 and the supercomputers it tracks.

A more ambitious (and more challenging) reporting landscape

Over the years, the Green500 has developed three levels of energy efficiency reporting: levels one, two and three. Level one submissions (which, when the Green500 launched, were all submissions) have the most forgiving requirements, allowing system operators to assess efficiency using a small subset of the system during just its core phase, with further leniency allowed in meter accuracy.

The other two levels are intended to push system operators toward more accurate reporting and more active consideration of energy efficiency. Level two submissions must use a larger portion of the system – including all its subsystems – and measure the average power of a full run with higher accuracy, while level three submissions assess the efficiency of the entire system and its energy use over a full run with exacting precision (<=0.5%).

Image courtesy of Wu Feng.

Feng said that making a level two or three report comes with advantages: system operators will then have access to more accurate information, allowing them to leverage that knowledge and data in system operations (or even procurement). Furthermore, some of them had reported that the barrier to entry for a level three report was relatively low.

Reporting at different levels also has a small correlation with the values reported, Feng said: “Usually, the level one reading for flops per watt will be a smidgen higher than for level two and level three.” Comparing the Green500 reporting to miles-per-gallon ratings for a car, he said that “instead of 29 miles per gallon and 22 miles per gallon highway and city, it might be 28.96 miles per gallon and 21.96 miles per gallon … There’s higher correlation between level two and level three than [there is for] level two and level three relative to level one, but they’re all still pretty close.”

But this push for greater accuracy, of course, has its impediments: for instance, Feng explained, a level three run might indeed pose little difficulty for an organization – “but only if you have the appropriate infrastructure in place – otherwise, it can be very difficult to do.”

“[For a level three run,] you have to have the entire supercomputer instrumented – all the nodes,” Feng told HPCwire. For increasingly massive systems, the number of meters required for a level three report can span into the thousands. If you have a massive system without the appropriate instrumentation, making a more accurate submission to the Green500 can be a very tall order. (Nvidia, for instance, had been unable to make level three submissions.)

As a result, despite level two and three reporting offering more accuracy – and more benefits for system operators – it is unfeasible to require those higher standards, particularly considering that even a level one report is optional for submission to the Top500, despite the lists joining forces five years ago. (Feng did say that one proposal had suggested that the organizers only award Green500 systems if they had made a level two or higher report.) “We started with level one mainly to try and get people to start reporting,” he said. “Because if the barrier to entry is that you have to do a level three measurement, we would have even fewer entries in the Green500.”

But beyond the push for level two and three reporting, the broader Green500 list faces some general resistance. “Just doing a Top500 run is labor-intensive, and then we’re adding onto it the measurement of power,” Feng said. “So it’s hard for a lot of these institutions to be commandeering, if you will, their supercomputers for such a long period of time.”

Furthermore, Feng explained that many Top500 systems choose not to report to the Green500 because they know or suspect that their energy efficiency rating will not be competitive. “By and large, most of the measurements [on the Green500 list] are of the supercomputers that are more energy efficient,” he said. “You see a smattering of ones farther down … but that’s probably because they were higher up earlier on and then they just kept reporting the number.”

These tendencies have created a dichotomous Green500 list that exhibits both promising and worrying reporting trends. “We had a high of approximately 350 submissions early on when everything was level one,” Feng said. “[But] the level of reporting has come down somewhat.” Indeed, the most recent edition had 181 submissions – a far cry from that high water mark, which Feng said was largely fueled by IBM’s strong interest and investment in energy efficiency.

On the other hand, the average reporting quality of the respondents is going up, as are the raw numbers of respondents choosing level two and three reporting. The 29 level two submissions – more than 16 percent of overall submissions – are at least a four-year high, as are the 14 level three submissions. The higher-tier levels are also overrepresented among the most efficient supercomputers, with only five of the Green500’s top ten using level one reporting.

The top ten most energy-efficient systems on the June 2021 Green500 list. The “power quality level” column on the right denotes the reporting level: level one, two or three: Image courtesy of Wu Feng.

Increasing efficiency, especially at the high end

Trends in the systems themselves are promising, as well: systems are broadly approaching 30-gigaflops-per-watt efficiency, with heterogeneous systems “really delivering and driving on the energy efficiency front” and accelerators dominating the list. These trends are accentuated in the higher echelons of system efficiency: “The median has been improving steadily,” Feng said, “[but] it’s the top end of the list that’s been improving much more rapidly.” The Green500’s top ten also showed a strong 60 percent turnover from November’s list, drawing a sharp contrast to the relatively stable Top500 list it accompanied.

Despite that strong turnover, Feng remains bearish about the ambitious 20MW energy efficiency goal that has long loomed over U.S. plans for exascale supercomputing. Throughout his talk, Feng extrapolated from various trends to an exaflops: the imminent 30-gigaflops-per-watt efficiency of the Green500, for instance, “linearly extrapolates to an exascale computer in approximately 33MW”; based on the top system on the Top500, the extrapolation was closer to 68MW. Whether any of the three planned U.S. exascale systems will hit the 20MW (50-gigaflops-per-watt) target remains a subject of debate in the broader community, of course – and for more on that, read HPCwire’s feature on the Frontier system.

Image courtesy of Wu Feng.

Now, Feng and the Green500 press on, working to push the envelope of energy efficiency in a supercomputing status quo that regularly pushes into tens of megawatts per system.

“The goal of the list is we seek to raise awareness and encourage energy efficiency of supercomputers, and we seek to drive energy efficiency as a first-order design constraint that is on par with performance,” Feng said. Later, he added: “The problem is that … frankly, here in the United States, we’re in a society that is all about performance.”

To view the full June 2021 Green500 list, click here.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Verifying the Universe with Exascale Computers

July 30, 2021

The ExaSky project, one of the critical Earth and Space Science applications being solved by the US Department of Energy’s (DOE’s) Exascale Computing Project (ECP), is preparing to use the nation’s forthcoming exas Read more…

What’s After Exascale? The Internet of Workflows Says HPE’s Nicolas Dubé

July 29, 2021

With the race to exascale computing in its final leg, it’s natural to wonder what the Post Exascale Era will look like. Nicolas Dubé, VP and chief technologist for HPE’s HPC business unit, agrees and shared his vision at Supercomputing Frontiers Europe 2021 held last week. The next big thing, he told the virtual audience at SFE21, is something that will connect HPC and (broadly) all of IT – into what Dubé calls The Internet of Workflows. Read more…

How UK Scientists Developed Transformative, HPC-Powered Coronavirus Sequencing System

July 29, 2021

In November 2020, the COVID-19 Genomics UK Consortium (COG-UK) won the HPCwire Readers’ Choice Award for Best HPC Collaboration for its CLIMB-COVID sequencing project. Launched in March 2020, CLIMB-COVID has now resulted in the sequencing of over 675,000 coronavirus genomes – an increasingly critical task as variants like Delta threaten the tenuous prospect of a return to normalcy in much of the world. Read more…

KAUST Leverages Mixed Precision for Geospatial Data

July 28, 2021

For many computationally intensive tasks, exacting precision is not necessary for every step of the entire task to obtain a suitably precise result. The alternative is mixed-precision computing: using high precision wher Read more…

Oak Ridge Supercomputer Enables Next-Gen Jet Turbine Research

July 27, 2021

Air travel is notoriously carbon-inefficient, with many airlines going as far as to offer purchasable carbon offsets to ease the guilt over large-footprint travel. But even over just the last decade, major aircraft model Read more…

AWS Solution Channel

Data compression with increased performance and lower costs

Many customers associate a performance cost with data compression, but that’s not the case with Amazon FSx for Lustre. With FSx for Lustre, data compression reduces storage costs and increases aggregate file system throughput. Read more…

IBM and University of Tokyo Roll Out Quantum System One in Japan

July 27, 2021

IBM and the University of Tokyo today unveiled an IBM Quantum System One as part of the IBM-Japan quantum program announced in 2019. The system is the second IBM Quantum System One assembled outside the U.S. and follows Read more…

What’s After Exascale? The Internet of Workflows Says HPE’s Nicolas Dubé

July 29, 2021

With the race to exascale computing in its final leg, it’s natural to wonder what the Post Exascale Era will look like. Nicolas Dubé, VP and chief technologist for HPE’s HPC business unit, agrees and shared his vision at Supercomputing Frontiers Europe 2021 held last week. The next big thing, he told the virtual audience at SFE21, is something that will connect HPC and (broadly) all of IT – into what Dubé calls The Internet of Workflows. Read more…

How UK Scientists Developed Transformative, HPC-Powered Coronavirus Sequencing System

July 29, 2021

In November 2020, the COVID-19 Genomics UK Consortium (COG-UK) won the HPCwire Readers’ Choice Award for Best HPC Collaboration for its CLIMB-COVID sequencing project. Launched in March 2020, CLIMB-COVID has now resulted in the sequencing of over 675,000 coronavirus genomes – an increasingly critical task as variants like Delta threaten the tenuous prospect of a return to normalcy in much of the world. Read more…

IBM and University of Tokyo Roll Out Quantum System One in Japan

July 27, 2021

IBM and the University of Tokyo today unveiled an IBM Quantum System One as part of the IBM-Japan quantum program announced in 2019. The system is the second IB Read more…

Intel Unveils New Node Names; Sapphire Rapids Is Now an ‘Intel 7’ CPU

July 27, 2021

What's a preeminent chip company to do when its process node technology lags the competition by (roughly) one generation, but outmoded naming conventions make it seem like it's two nodes behind? For Intel, the response was to change how it refers to its nodes with the aim of better reflecting its positioning within the leadership semiconductor manufacturing space. Intel revealed its new node nomenclature, and... Read more…

Will Approximation Drive Post-Moore’s Law HPC Gains?

July 26, 2021

“Hardware-based improvements are going to get more and more difficult,” said Neil Thompson, an innovation scholar at MIT’s Computer Science and Artificial Intelligence Lab (CSAIL). “I think that’s something that this crowd will probably, actually, be already familiar with.” Thompson, speaking... Read more…

With New Owner and New Roadmap, an Independent Omni-Path Is Staging a Comeback

July 23, 2021

Put on a shelf by Intel in 2019, Omni-Path faced a uncertain future, but under new custodian Cornelis Networks, OmniPath is looking to make a comeback as an independent high-performance interconnect solution. A "significant refresh" – called Omni-Path Express – is coming later this year according to the company. Cornelis Networks formed last September as a spinout of Intel's Omni-Path division. Read more…

Chameleon’s HPC Testbed Sharpens Its Edge, Presses ‘Replay’

July 22, 2021

“One way of saying what I do for a living is to say that I develop scientific instruments,” said Kate Keahey, a senior fellow at the University of Chicago a Read more…

Summer Reading: “High-Performance Computing Is at an Inflection Point”

July 21, 2021

At last month’s 11th International Symposium on Highly Efficient Accelerators and Reconfigurable Technologies (HEART), a group of researchers led by Martin Schulz of the Leibniz Supercomputing Center (Munich) presented a “position paper” in which they argue HPC architectural landscape... Read more…

AMD Chipmaker TSMC to Use AMD Chips for Chipmaking

May 8, 2021

TSMC has tapped AMD to support its major manufacturing and R&D workloads. AMD will provide its Epyc Rome 7702P CPUs – with 64 cores operating at a base cl Read more…

Intel Launches 10nm ‘Ice Lake’ Datacenter CPU with Up to 40 Cores

April 6, 2021

The wait is over. Today Intel officially launched its 10nm datacenter CPU, the third-generation Intel Xeon Scalable processor, codenamed Ice Lake. With up to 40 Read more…

Berkeley Lab Debuts Perlmutter, World’s Fastest AI Supercomputer

May 27, 2021

A ribbon-cutting ceremony held virtually at Berkeley Lab's National Energy Research Scientific Computing Center (NERSC) today marked the official launch of Perlmutter – aka NERSC-9 – the GPU-accelerated supercomputer built by HPE in partnership with Nvidia and AMD. Read more…

Ahead of ‘Dojo,’ Tesla Reveals Its Massive Precursor Supercomputer

June 22, 2021

In spring 2019, Tesla made cryptic reference to a project called Dojo, a “super-powerful training computer” for video data processing. Then, in summer 2020, Tesla CEO Elon Musk tweeted: “Tesla is developing a [neural network] training computer called Dojo to process truly vast amounts of video data. It’s a beast! … A truly useful exaflop at de facto FP32.” Read more…

Google Launches TPU v4 AI Chips

May 20, 2021

Google CEO Sundar Pichai spoke for only one minute and 42 seconds about the company’s latest TPU v4 Tensor Processing Units during his keynote at the Google I Read more…

CentOS Replacement Rocky Linux Is Now in GA and Under Independent Control

June 21, 2021

The Rocky Enterprise Software Foundation (RESF) is announcing the general availability of Rocky Linux, release 8.4, designed as a drop-in replacement for the soon-to-be discontinued CentOS. The GA release is launching six-and-a-half months after Red Hat deprecated its support for the widely popular, free CentOS server operating system. The Rocky Linux development effort... Read more…

Iran Gains HPC Capabilities with Launch of ‘Simorgh’ Supercomputer

May 18, 2021

Iran is said to be developing domestic supercomputing technology to advance the processing of scientific, economic, political and military data, and to strengthen the nation’s position in the age of AI and big data. On Sunday, Iran unveiled the Simorgh supercomputer, which will deliver.... Read more…

HPE Launches Storage Line Loaded with IBM’s Spectrum Scale File System

April 6, 2021

HPE today launched a new family of storage solutions bundled with IBM’s Spectrum Scale Erasure Code Edition parallel file system (description below) and featu Read more…

Leading Solution Providers

Contributors

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from Julialang.org. I Read more…

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

GTC21: Nvidia Launches cuQuantum; Dips a Toe in Quantum Computing

April 13, 2021

Yesterday Nvidia officially dipped a toe into quantum computing with the launch of cuQuantum SDK, a development platform for simulating quantum circuits on GPU-accelerated systems. As Nvidia CEO Jensen Huang emphasized in his keynote, Nvidia doesn’t plan to build... Read more…

Microsoft to Provide World’s Most Powerful Weather & Climate Supercomputer for UK’s Met Office

April 22, 2021

More than 14 months ago, the UK government announced plans to invest £1.2 billion ($1.56 billion) into weather and climate supercomputing, including procuremen Read more…

Quantum Roundup: IBM, Rigetti, Phasecraft, Oxford QC, China, and More

July 13, 2021

IBM yesterday announced a proof for a quantum ML algorithm. A week ago, it unveiled a new topology for its quantum processors. Last Friday, the Technical Univer Read more…

Q&A with Jim Keller, CTO of Tenstorrent, and an HPCwire Person to Watch in 2021

April 22, 2021

As part of our HPCwire Person to Watch series, we are happy to present our interview with Jim Keller, president and chief technology officer of Tenstorrent. One of the top chip architects of our time, Keller has had an impactful career. Read more…

AMD-Xilinx Deal Gains UK, EU Approvals — China’s Decision Still Pending

July 1, 2021

AMD’s planned acquisition of FPGA maker Xilinx is now in the hands of Chinese regulators after needed antitrust approvals for the $35 billion deal were receiv Read more…

Senate Debate on Bill to Remake NSF – the Endless Frontier Act – Begins

May 18, 2021

The U.S. Senate today opened floor debate on the Endless Frontier Act which seeks to remake and expand the National Science Foundation by creating a technology Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire