HPC Under the Covers: Linpack, Exascale & the Top500

By Tiffany Trader

June 28, 2018

HPCers can get painted as a monolithic bunch by outsiders, but internecine disagreements abound over the HPCest of HPC jargon, as was evident at ISC this week.

Ask four HPC leaders about Linpack’s relevance, get four distinct answers — and that’s just what happened at the Monday Top500 panel. During the panel, moderator Horst Simon (Top500 co-author and deputy lab director of Lawrence Berkeley National Lab) asked panelists Yutong Lu, Steve Conway, Thomas Schulthess and Steve Scott about the limitations surrounding Linpack and what needs to be changed at the Top500.

Yutong Lu, National Supercomputing Center in Guangzhou, China, and ISC 2019 Program Chair:

“I think the performance for the supercomputer will be the eternal target because people will always ask and care about how much faster the supercomputer could run, and what’s the highest performance that can be reached. But I think that the metrics could be changed. If you look back 20-or-more years, the computational power was the bottleneck of the full system, so the HPL was a good benchmark at that time and continued to be over the past 20 years. But now we all note that the data access and ability have become the bottleneck of the system, so we obviously need some new benchmarks to measure that part. That will be something we need to change.”

Steve Conway, COO of Hyperion Research:

“The Top500 is great as a census of elements affecting large supercomputers over time, but it’s often been interpreted–as it was never intended to be–as a predictor of performance over a spectrum of HPC code. One thing that could be valuable is a warning like on the cigarette label that says ‘this could be fatal if you use it as a predictor.’ But I was very pleased to see the attention paid to HPCG and the Green500 and the inclusion of those lists. My only recommendation would be to give those equal promotional strength.”

Thomas Schulthess, Director of the Swiss National Supercomputing Centre (CSCS):

ISC 2018 Panel: Top500’s Relevance after 25 Years

“I have quite a different opinion. The relevance today is clearly from a political point of view and a funding point of view. From an application performance point of view the story is very different. It even comes to the point where the Top500 may actually be a distraction if you have certain goals on the application side. And let me give you an example: there is the TaihuLight system in Wuxi, and the Piz Daint system that I have a lot of authority over. When you look at the flops, TaihuLight is on top with a factor of five difference. When you look at how the benchmark from the weather and climate community performs–the baroclinic instability test–then the order is reversed and the performance of Piz Daint is about two to three times faster.

“We’ve been thinking about this quite a bit…and it turns out that flops is not a good metric to design systems against. It may be good to track and look back retroactively, but not looking into the future. The conclusion is that we need a metric that relates to a scientific goal: so simulated years per day for the given size of the problem. And it is very important that the size of the problem factors in. Remember in the Top500 the HPL we do the size of the problem to maximize this metric of flops. We can do the same with HPCG. It turns out from an engineering point of view that this is not good. If you’re paid to do something, you’re not going to change your target just to maximize some number. That’s a really bad idea.

“We need to set goals. In weather and climate I think we have very clear goals that everybody can relate to, and I wish that the scientific community could come together behind a few goals rather than everybody wanting their own goal to be the metric. So not just some performance metric, but the size of the problem needs to be set, and can be varied over time. But we need to compare apples to apples, and not apples to oranges. And the last point that is really missing in the Top500 is the algorithmic or the method side. Changing algorithms in the history of computing is just as important as changing architectures.”

Steve Scott, Cray SVP and Chief Technology Officer:

“From a scientific perspective I couldn’t agree with you more. From a practical perspective I can’t agree with you at all.

“I would love to see simulated years per day as a much more interesting and useful metric, but there’s no way that you could do that. And you can’t really change the metric that the list uses because it sort of invalidates that historical record aspect. So we have to count on people that are actually doing these procurements and fielding these big systems to be sophisticated people who understand what’s really important and that Linpack is not that thing; and I absolutely think that the Top500–despite all of the good that it’s done–has caused some bad behavior. People have made decisions to get to a higher ranking on that list. And then there’ve been other people who have said ‘I’m going to buy a supercomputer and I’m not even going to put it on the list, because I don’t endorse the metric.’ I think the reality is that you’re not going to be able to change the Top500 benchmark. I like the idea of augmenting it with some things, and there’ve been some attempts with HPCG and the HPCC benchmarks. So we can augment it; I don’t think we can change it.

“I think that the HPL performance is becoming more and more disjointed from real application performance as we go forward, and memory bandwidth and interconnects and other things matter a lot more. Architectural aspects matter. As we get closer to the end of the CMOS era and we may change the way we do computing or go to completely different architectures, it may become even more strained to the point where we have to do something. But in the meantime I’m not sure there’s a whole lot we can do other that continue on the current path.”

Is it Exascale?

At ISC and on #HPC Twitter, discussion has also turned to the “true meaning” of exascale; take this tweet thread for example:

The discussion was further unpacked in this fun exchange from the ISC Analyst Crossfire put on by Intersect360 Research CEO Addison Snell with panelists Depei Qian (Sun Yat-Sen University & Beihang University), Stephan Schenk (BASF SE), Alex Bouzari (DDN), and Ian Colle (HPC at Amazon Web Services).

“Exascale is a term that’s driving me nuts because it has no exact definition,” said Snell, who proceeded to proffer variations of potential exascale definitions to get panelists’ quick takes.

The panelists all agreed that “exa-levels of something non-computational, like an exabyte of storage under one namespace or if you could magically have an exabit/sec of bandwidth” were not exascale, with one panelist offering that “exascale is whatever gets politicians to fund industry.”

The concept of whether 10 to the 18th flops per second at reduced/mixed precision should be called exascale drew one yes, another reference to funding, and consensus from the other panelists, the moderator and yours truly that that was moving the goalposts.

As for whether 10 to the 18th flops theoretical peak with no Linpack or other benchmark or application gets you to exascale, the panelists were unanimous in that it does not, with a comment that “the only thing Linpack does is it gets funding from politicians,” and another that “if our focus is on doing real work, no.”

In the final scenario, Snell asked whether exaflops for a loosely coupled non-HPC application like SETI@home counts as exascale. That drew three no’s and another nod to the market opportunity.

HPC Secrets

The open secrets of HPC are in the crosshairs this week, as illustrated by Andrew Jones’ article published by our friends at the Top500 News.

The increased prevalence of IT/Web-scale systems (close to half the list now) means it’s not in verity a list of 500 supercomputers or HPC clusters. But it was so-called list stuffing via duplicate systems (or large deployments parceled so as to optimize system share) that came to wider attention this week when Lenovo claimed 117 of the 500 machines, becoming the largest Top500 provider as measured by number of systems. It needs to be said that Lenovo didn’t invent the practice–but they have mastered it (the company lists 56 duplicate entries). It should also be said that they have not, to our knowledge, broken any rules.

A search through the annals of the list shows duplicate serial entries exist going back to at least 2010. The practice slowed down after Lenovo purchased IBM’s x86 business in 2014 and ramped up again a year later as Lenovo (and other vendors) figured out how to amplify their list presence, both through system slicing or increased benchmarking of Web/IT machines. The effect shows up as a dip in 10G Ethernet (rise in InfiniBand) and the subsequent climb of 10G Ethernet in that timeframe.

Source: Mellanox June 2018 Top500 analysis slidies

My read is that there was a collectively accepted threshold at which cloud/IT systems and creative system splicing were tolerated, and now that line may be breached. How or whether the problem will be addressed is not clear to me. It’s not as simple as removing anonymous submissions, since anonymity is requisite for industrial representation (as Jones pointed out). Hundreds of NDA site visits obviously aren’t feasible. Top500 Co-author Eric Strohmeier indicated in the Top500 press briefing on Monday that his group knows “reasonably well” who is using the anonymously listed systems; but what is more challenging is tracking down whether a submission is really configured the way it is claimed. Due to some game-playing in the past, system configurations must now be frozen, so that if systems are parceled up into small increments to get a high system count, those systems cannot then be reconfigured into a larger system to keep from falling off.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

InfiniBand Still Tops in Supercomputing

July 19, 2018

In the competitive global HPC landscape, system and processor vendors, nations and end user sites certainly get a lot of attention--deservedly so--but more than ever, the network plays a crucial role. While fast, perform Read more…

By Tiffany Trader

HPC for Life: Genomics, Brain Research, and Beyond

July 19, 2018

During the past few decades, the life sciences have witnessed one landmark discovery after another with the aid of HPC, paving the way toward a new era of personalized treatments based on an individual’s genetic makeup Read more…

By Warren Froelich

WCRP’s New Strategic Plan for Climate Research Highlights the Importance of HPC

July 19, 2018

As climate modeling increasingly leverages exascale computing and researchers warn of an impending computing gap in climate research, the World Climate Research Programme (WCRP) is developing its new Strategic Plan – and high-performance computing is slated to play a critical role. Read more…

By Oliver Peckham

HPE Extreme Performance Solutions

Introducing the First Integrated System Management Software for HPC Clusters from HPE

How do you manage your complex, growing cluster environments? Answer that big challenge with the new HPC cluster management solution: HPE Performance Cluster Manager. Read more…

IBM Accelerated Insights

Are Your Software Licenses Impeding Your Productivity?

In my previous article, Improving chip yield rates with cognitive manufacturing, I highlighted the costs associated with semiconductor manufacturing, and how cognitive methods can yield benefits in both design and manufacture.  Read more…

U.S. Exascale Computing Project Releases Software Technology Progress Report

July 19, 2018

As is often noted the race to exascale computing isn’t just about hardware. This week the U.S. Exascale Computing Project (ECP) released its latest Software Technology (ST) Capability Assessment Report detailing progress so far. Read more…

By John Russell

InfiniBand Still Tops in Supercomputing

July 19, 2018

In the competitive global HPC landscape, system and processor vendors, nations and end user sites certainly get a lot of attention--deservedly so--but more than Read more…

By Tiffany Trader

HPC for Life: Genomics, Brain Research, and Beyond

July 19, 2018

During the past few decades, the life sciences have witnessed one landmark discovery after another with the aid of HPC, paving the way toward a new era of perso Read more…

By Warren Froelich

D-Wave Breaks New Ground in Quantum Simulation

July 16, 2018

Last Friday D-Wave scientists and colleagues published work in Science which they say represents the first fulfillment of Richard Feynman’s 1982 notion that Read more…

By John Russell

AI Thought Leaders on Capitol Hill

July 14, 2018

On Thursday, July 12, the House Committee on Science, Space, and Technology heard from four academic and industry leaders – representatives from Berkeley Lab, Argonne Lab, GE Global Research and Carnegie Mellon University – on the opportunities springing from the intersection of machine learning and advanced-scale computing. Read more…

By Tiffany Trader

HPC Serves as a ‘Rosetta Stone’ for the Information Age

July 12, 2018

In an age defined and transformed by its data, several large-scale scientific instruments around the globe might be viewed as a ‘mother lode’ of precious data. With names seemingly created for a ‘techno-speak’ glossary, these interferometers, cyclotrons, sequencers, solenoids, satellite altimeters, and cryo-electron microscopes are churning out data in previously unthinkable and seemingly incomprehensible quantities -- billions, trillions and quadrillions of bits and bytes of electro-magnetic code. Read more…

By Warren Froelich

Tsinghua Powers Through ISC18 Field

July 10, 2018

Tsinghua University topped all other competitors at the ISC18 Student Cluster Competition with an overall score of 88.43 out of 100. This gives Tsinghua their s Read more…

By Dan Olds

HPE, EPFL Launch Blue Brain 5 Supercomputer

July 10, 2018

HPE and the Ecole Polytechnique Federale de Lausannne (EPFL) Blue Brain Project yesterday introduced Blue Brain 5, a new supercomputer built by HPE, which displ Read more…

By John Russell

Pumping New Life into HPC Clusters, the Case for Liquid Cooling

July 10, 2018

High Performance Computing (HPC) faces some daunting challenges in the coming years as traditional, industry-standard systems push the boundaries of data center Read more…

By Scott Tease

Leading Solution Providers

SC17 Booth Video Tours Playlist

Altair @ SC17

Altair

AMD @ SC17

AMD

ASRock Rack @ SC17

ASRock Rack

CEJN @ SC17

CEJN

DDN Storage @ SC17

DDN Storage

Huawei @ SC17

Huawei

IBM @ SC17

IBM

IBM Power Systems @ SC17

IBM Power Systems

Intel @ SC17

Intel

Lenovo @ SC17

Lenovo

Mellanox Technologies @ SC17

Mellanox Technologies

Microsoft @ SC17

Microsoft

Penguin Computing @ SC17

Penguin Computing

Pure Storage @ SC17

Pure Storage

Supericro @ SC17

Supericro

Tyan @ SC17

Tyan

Univa @ SC17

Univa

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This