HPC Under the Covers: Linpack, Exascale & the Top500

By Tiffany Trader

June 28, 2018

HPCers can get painted as a monolithic bunch by outsiders, but internecine disagreements abound over the HPCest of HPC jargon, as was evident at ISC this week.

Ask four HPC leaders about Linpack’s relevance, get four distinct answers — and that’s just what happened at the Monday Top500 panel. During the panel, moderator Horst Simon (Top500 co-author and deputy lab director of Lawrence Berkeley National Lab) asked panelists Yutong Lu, Steve Conway, Thomas Schulthess and Steve Scott about the limitations surrounding Linpack and what needs to be changed at the Top500.

Yutong Lu, National Supercomputing Center in Guangzhou, China, and ISC 2019 Program Chair:

“I think the performance for the supercomputer will be the eternal target because people will always ask and care about how much faster the supercomputer could run, and what’s the highest performance that can be reached. But I think that the metrics could be changed. If you look back 20-or-more years, the computational power was the bottleneck of the full system, so the HPL was a good benchmark at that time and continued to be over the past 20 years. But now we all note that the data access and ability have become the bottleneck of the system, so we obviously need some new benchmarks to measure that part. That will be something we need to change.”

Steve Conway, COO of Hyperion Research:

“The Top500 is great as a census of elements affecting large supercomputers over time, but it’s often been interpreted–as it was never intended to be–as a predictor of performance over a spectrum of HPC code. One thing that could be valuable is a warning like on the cigarette label that says ‘this could be fatal if you use it as a predictor.’ But I was very pleased to see the attention paid to HPCG and the Green500 and the inclusion of those lists. My only recommendation would be to give those equal promotional strength.”

Thomas Schulthess, Director of the Swiss National Supercomputing Centre (CSCS):

ISC 2018 Panel: Top500’s Relevance after 25 Years

“I have quite a different opinion. The relevance today is clearly from a political point of view and a funding point of view. From an application performance point of view the story is very different. It even comes to the point where the Top500 may actually be a distraction if you have certain goals on the application side. And let me give you an example: there is the TaihuLight system in Wuxi, and the Piz Daint system that I have a lot of authority over. When you look at the flops, TaihuLight is on top with a factor of five difference. When you look at how the benchmark from the weather and climate community performs–the baroclinic instability test–then the order is reversed and the performance of Piz Daint is about two to three times faster.

“We’ve been thinking about this quite a bit…and it turns out that flops is not a good metric to design systems against. It may be good to track and look back retroactively, but not looking into the future. The conclusion is that we need a metric that relates to a scientific goal: so simulated years per day for the given size of the problem. And it is very important that the size of the problem factors in. Remember in the Top500 the HPL we do the size of the problem to maximize this metric of flops. We can do the same with HPCG. It turns out from an engineering point of view that this is not good. If you’re paid to do something, you’re not going to change your target just to maximize some number. That’s a really bad idea.

“We need to set goals. In weather and climate I think we have very clear goals that everybody can relate to, and I wish that the scientific community could come together behind a few goals rather than everybody wanting their own goal to be the metric. So not just some performance metric, but the size of the problem needs to be set, and can be varied over time. But we need to compare apples to apples, and not apples to oranges. And the last point that is really missing in the Top500 is the algorithmic or the method side. Changing algorithms in the history of computing is just as important as changing architectures.”

Steve Scott, Cray SVP and Chief Technology Officer:

“From a scientific perspective I couldn’t agree with you more. From a practical perspective I can’t agree with you at all.

“I would love to see simulated years per day as a much more interesting and useful metric, but there’s no way that you could do that. And you can’t really change the metric that the list uses because it sort of invalidates that historical record aspect. So we have to count on people that are actually doing these procurements and fielding these big systems to be sophisticated people who understand what’s really important and that Linpack is not that thing; and I absolutely think that the Top500–despite all of the good that it’s done–has caused some bad behavior. People have made decisions to get to a higher ranking on that list. And then there’ve been other people who have said ‘I’m going to buy a supercomputer and I’m not even going to put it on the list, because I don’t endorse the metric.’ I think the reality is that you’re not going to be able to change the Top500 benchmark. I like the idea of augmenting it with some things, and there’ve been some attempts with HPCG and the HPCC benchmarks. So we can augment it; I don’t think we can change it.

“I think that the HPL performance is becoming more and more disjointed from real application performance as we go forward, and memory bandwidth and interconnects and other things matter a lot more. Architectural aspects matter. As we get closer to the end of the CMOS era and we may change the way we do computing or go to completely different architectures, it may become even more strained to the point where we have to do something. But in the meantime I’m not sure there’s a whole lot we can do other that continue on the current path.”

Is it Exascale?

At ISC and on #HPC Twitter, discussion has also turned to the “true meaning” of exascale; take this tweet thread for example:

The discussion was further unpacked in this fun exchange from the ISC Analyst Crossfire put on by Intersect360 Research CEO Addison Snell with panelists Depei Qian (Sun Yat-Sen University & Beihang University), Stephan Schenk (BASF SE), Alex Bouzari (DDN), and Ian Colle (HPC at Amazon Web Services).

“Exascale is a term that’s driving me nuts because it has no exact definition,” said Snell, who proceeded to proffer variations of potential exascale definitions to get panelists’ quick takes.

The panelists all agreed that “exa-levels of something non-computational, like an exabyte of storage under one namespace or if you could magically have an exabit/sec of bandwidth” were not exascale, with one panelist offering that “exascale is whatever gets politicians to fund industry.”

The concept of whether 10 to the 18th flops per second at reduced/mixed precision should be called exascale drew one yes, another reference to funding, and consensus from the other panelists, the moderator and yours truly that that was moving the goalposts.

As for whether 10 to the 18th flops theoretical peak with no Linpack or other benchmark or application gets you to exascale, the panelists were unanimous in that it does not, with a comment that “the only thing Linpack does is it gets funding from politicians,” and another that “if our focus is on doing real work, no.”

In the final scenario, Snell asked whether exaflops for a loosely coupled non-HPC application like [email protected] counts as exascale. That drew three no’s and another nod to the market opportunity.

HPC Secrets

The open secrets of HPC are in the crosshairs this week, as illustrated by Andrew Jones’ article published by our friends at the Top500 News.

The increased prevalence of IT/Web-scale systems (close to half the list now) means it’s not in verity a list of 500 supercomputers or HPC clusters. But it was so-called list stuffing via duplicate systems (or large deployments parceled so as to optimize system share) that came to wider attention this week when Lenovo claimed 117 of the 500 machines, becoming the largest Top500 provider as measured by number of systems. It needs to be said that Lenovo didn’t invent the practice–but they have mastered it (the company lists 56 duplicate entries). It should also be said that they have not, to our knowledge, broken any rules.

A search through the annals of the list shows duplicate serial entries exist going back to at least 2010. The practice slowed down after Lenovo purchased IBM’s x86 business in 2014 and ramped up again a year later as Lenovo (and other vendors) figured out how to amplify their list presence, both through system slicing or increased benchmarking of Web/IT machines. The effect shows up as a dip in 10G Ethernet (rise in InfiniBand) and the subsequent climb of 10G Ethernet in that timeframe.

Source: Mellanox June 2018 Top500 analysis slidies

My read is that there was a collectively accepted threshold at which cloud/IT systems and creative system splicing were tolerated, and now that line may be breached. How or whether the problem will be addressed is not clear to me. It’s not as simple as removing anonymous submissions, since anonymity is requisite for industrial representation (as Jones pointed out). Hundreds of NDA site visits obviously aren’t feasible. Top500 Co-author Eric Strohmeier indicated in the Top500 press briefing on Monday that his group knows “reasonably well” who is using the anonymously listed systems; but what is more challenging is tracking down whether a submission is really configured the way it is claimed. Due to some game-playing in the past, system configurations must now be frozen, so that if systems are parceled up into small increments to get a high system count, those systems cannot then be reconfigured into a larger system to keep from falling off.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Is Data Science the Fourth Pillar of the Scientific Method?

April 18, 2019

Nvidia CEO Jensen Huang revived a decade-old debate last month when he said that modern data science (AI plus HPC) has become the fourth pillar of the scientific method. While some disagree with the notion that statistic Read more…

By Alex Woodie

At ASF 2019: The Virtuous Circle of Big Data, AI and HPC

April 18, 2019

We've entered a new phase in IT -- in the world, really -- where the combination of big data, artificial intelligence, and high performance computing is pushing the bounds of what's possible in business and science, in w Read more…

By Alex Woodie with Doug Black and Tiffany Trader

Google Open Sources TensorFlow Version of MorphNet DL Tool

April 18, 2019

Designing optimum deep neural networks remains a non-trivial exercise. “Given the large search space of possible architectures, designing a network from scratch for your specific application can be prohibitively expens Read more…

By John Russell

HPE Extreme Performance Solutions

HPE and Intel® Omni-Path Architecture: How to Power a Cloud

Learn how HPE and Intel® Omni-Path Architecture provide critical infrastructure for leading Nordic HPC provider’s HPCFLOW cloud service.

powercloud_blog.jpgFor decades, HPE has been at the forefront of high-performance computing, and we’ve powered some of the fastest and most robust supercomputers in the world. Read more…

IBM Accelerated Insights

Bridging HPC and Cloud Native Development with Kubernetes

The HPC community has historically developed its own specialized software stack including schedulers, filesystems, developer tools, container technologies tuned for performance and large-scale on-premises deployments. Read more…

Interview with 2019 Person to Watch Michela Taufer

April 18, 2019

Today, as part of our ongoing HPCwire People to Watch focus series, we are highlighting our interview with 2019 Person to Watch Michela Taufer. Michela -- the General Chair of SC19 -- is an ACM Distinguished Scientist. Read more…

By HPCwire Editorial Team

At ASF 2019: The Virtuous Circle of Big Data, AI and HPC

April 18, 2019

We've entered a new phase in IT -- in the world, really -- where the combination of big data, artificial intelligence, and high performance computing is pushing Read more…

By Alex Woodie with Doug Black and Tiffany Trader

Interview with 2019 Person to Watch Michela Taufer

April 18, 2019

Today, as part of our ongoing HPCwire People to Watch focus series, we are highlighting our interview with 2019 Person to Watch Michela Taufer. Michela -- the Read more…

By HPCwire Editorial Team

Intel Gold U-Series SKUs Reveal Single Socket Intentions

April 18, 2019

Intel plans to jump into the single socket market with a portion of its just announced Cascade Lake microprocessor line according to one media report. This isn Read more…

By John Russell

BSC Researchers Shrink Floating Point Formats to Accelerate Deep Neural Network Training

April 15, 2019

Sometimes calculating solutions as precisely as a computer can wastes more CPU resources than is necessary. A case in point is with deep learning. In early stag Read more…

By Ken Strandberg

Intel Extends FPGA Ecosystem with 10nm Agilex

April 11, 2019

The insatiable appetite for higher throughput and lower latency – particularly where edge analytics and AI, network functions, or for a range of datacenter ac Read more…

By Doug Black

Nvidia Doubles Down on Medical AI

April 9, 2019

Nvidia is collaborating with medical groups to push GPU-powered AI tools into clinical settings, including radiology and drug discovery. The GPU leader said Monday it will collaborate with the American College of Radiology (ACR) to provide clinicians with its Clara AI tool kit. The partnership would allow radiologists to leverage AI techniques for diagnostic imaging using their own clinical data. Read more…

By George Leopold

Digging into MLPerf Benchmark Suite to Inform AI Infrastructure Decisions

April 9, 2019

With machine learning and deep learning storming into the datacenter, the new challenge is optimizing infrastructure choices to support diverse ML and DL workfl Read more…

By John Russell

AI and Enterprise Datacenters Boost HPC Server Revenues Past Expectations – Hyperion

April 9, 2019

Building on the big year of 2017 and spurred in part by the convergence of AI and HPC, global revenue for high performance servers jumped 15.6 percent last year Read more…

By Doug Black

The Case Against ‘The Case Against Quantum Computing’

January 9, 2019

It’s not easy to be a physicist. Richard Feynman (basically the Jimi Hendrix of physicists) once said: “The first principle is that you must not fool yourse Read more…

By Ben Criger

Why Nvidia Bought Mellanox: ‘Future Datacenters Will Be…Like High Performance Computers’

March 14, 2019

“Future datacenters of all kinds will be built like high performance computers,” said Nvidia CEO Jensen Huang during a phone briefing on Monday after Nvidia revealed scooping up the high performance networking company Mellanox for $6.9 billion. Read more…

By Tiffany Trader

ClusterVision in Bankruptcy, Fate Uncertain

February 13, 2019

ClusterVision, European HPC specialists that have built and installed over 20 Top500-ranked systems in their nearly 17-year history, appear to be in the midst o Read more…

By Tiffany Trader

Intel Reportedly in $6B Bid for Mellanox

January 30, 2019

The latest rumors and reports around an acquisition of Mellanox focus on Intel, which has reportedly offered a $6 billion bid for the high performance interconn Read more…

By Doug Black

It’s Official: Aurora on Track to Be First US Exascale Computer in 2021

March 18, 2019

The U.S. Department of Energy along with Intel and Cray confirmed today that an Intel/Cray supercomputer, "Aurora," capable of sustained performance of one exaf Read more…

By Tiffany Trader

Looking for Light Reading? NSF-backed ‘Comic Books’ Tackle Quantum Computing

January 28, 2019

Still baffled by quantum computing? How about turning to comic books (graphic novels for the well-read among you) for some clarity and a little humor on QC. The Read more…

By John Russell

IBM Quantum Update: Q System One Launch, New Collaborators, and QC Center Plans

January 10, 2019

IBM made three significant quantum computing announcements at CES this week. One was introduction of IBM Q System One; it’s really the integration of IBM’s Read more…

By John Russell

Deep500: ETH Researchers Introduce New Deep Learning Benchmark for HPC

February 5, 2019

ETH researchers have developed a new deep learning benchmarking environment – Deep500 – they say is “the first distributed and reproducible benchmarking s Read more…

By John Russell

Leading Solution Providers

SC 18 Virtual Booth Video Tour

Advania @ SC18 AMD @ SC18
ASRock Rack @ SC18
DDN Storage @ SC18
HPE @ SC18
IBM @ SC18
Lenovo @ SC18 Mellanox Technologies @ SC18
NVIDIA @ SC18
One Stop Systems @ SC18
Oracle @ SC18 Panasas @ SC18
Supermicro @ SC18 SUSE @ SC18 TYAN @ SC18
Verne Global @ SC18

IBM Bets $2B Seeking 1000X AI Hardware Performance Boost

February 7, 2019

For now, AI systems are mostly machine learning-based and “narrow” – powerful as they are by today's standards, they're limited to performing a few, narro Read more…

By Doug Black

The Deep500 – Researchers Tackle an HPC Benchmark for Deep Learning

January 7, 2019

How do you know if an HPC system, particularly a larger-scale system, is well-suited for deep learning workloads? Today, that’s not an easy question to answer Read more…

By John Russell

Arm Unveils Neoverse N1 Platform with up to 128-Cores

February 20, 2019

Following on its Neoverse roadmap announcement last October, Arm today revealed its next-gen Neoverse microarchitecture with compute and throughput-optimized si Read more…

By Tiffany Trader

France to Deploy AI-Focused Supercomputer: Jean Zay

January 22, 2019

HPE announced today that it won the contract to build a supercomputer that will drive France’s AI and HPC efforts. The computer will be part of GENCI, the Fre Read more…

By Tiffany Trader

Intel Launches Cascade Lake Xeons with Up to 56 Cores

April 2, 2019

At Intel's Data-Centric Innovation Day in San Francisco (April 2), the company unveiled its second-generation Xeon Scalable (Cascade Lake) family and debuted it Read more…

By Tiffany Trader

Oil and Gas Supercloud Clears Out Remaining Knights Landing Inventory: All 38,000 Wafers

March 13, 2019

The McCloud HPC service being built by Australia’s DownUnder GeoSolutions (DUG) outside Houston is set to become the largest oil and gas cloud in the world th Read more…

By Tiffany Trader

Intel Extends FPGA Ecosystem with 10nm Agilex

April 11, 2019

The insatiable appetite for higher throughput and lower latency – particularly where edge analytics and AI, network functions, or for a range of datacenter ac Read more…

By Doug Black

UC Berkeley Paper Heralds Rise of Serverless Computing in the Cloud – Do You Agree?

February 13, 2019

Almost exactly ten years to the day from publishing of their widely-read, seminal paper on cloud computing, UC Berkeley researchers have issued another ambitious examination of cloud computing - Cloud Programming Simplified: A Berkeley View on Serverless Computing. The new work heralds the rise of ‘serverless computing’ as the next dominant phase of cloud computing. Read more…

By John Russell

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This