Every few years, the relevance of the TOP500 list is called into question. Community insiders understand that this single-metric test is not representative of most supercomputing workloads. Still no open science system had ever opted out of the list until NCSA made the bold move to keep Blue Waters’ Linpack results private.

In an interview with HPCwire, Blue Waters Project Director Bill Kramer provided a detailed account of the NCSA’s decision-making process and what they perceive to be the list’s flaws. While some may consider their stance controversial, it has nevertheless sparked a dialogue, which is what Kramer was hoping for.

In response to that interview, we received a letter from Council on Competitiveness President and CEO Deborah L. Wince-Smith offering an alternative point-of-view. We’ve published that letter below (edited only for style).

Dear Editor:

The Council on Competitiveness has long championed high-performance computing (HPC) as a basis for increased innovation and productivity, and as a competitive advantage for the United States in the global marketplace. We closely follow developments in the advancement of high-end supercomputing; hence are somewhat concerned by the comments made by Blue Waters Project Director Bill Kramer in his recent interview with HPCwire, November 16th, 2012 (“Blue Waters Opts Out of TOP500“).

The larger HPC community and we agree the Linpack benchmark used as the metric for the TOP500 list is limited as a measure of overall HPC system performance, but we came away from reading the article with the conclusion that those who choose to “opt in” to the TOP500 are less interested in sustained performance on mission critical applications than they are in achieving a high Linpack result.

From our external vantage point, sustained performance on scientific and engineering applications is clearly central to U.S. Department of Energy (DOE) academic and research missions, and is the primary driver for the development of ever more capable HPC platforms and solutions. As an example, we were pleased to see teams from Argonne and Lawrence Livermore each achieve truly astonishing performance of over 11 petaflops sustained on the 20 petaflops peak Sequoia system (TOP500 #2) on two very challenging, scientific applications, which HPCwire published on November 28th, 2012 (“DOE Labs Set Records with IBM Blue Gene/Q“).

From the Council’s perspective, the idea of the Sustained Petascale Performance test is interesting, especially if we had specifics on how this might be applied to the larger HPC community and industry in particular. Ongoing discussions on how best to evolve the standard that TOP500 represents are welcome, but at the same time, we should not deprive the community of this very important benchmark.

The fleeting glory of a high TOP500 ranking is not what motivates the HPC community, but these benchmarks do focus attention and conversation on how we are doing and where we might improve, particularly in light of growing off-shore competition for HPC leadership.

Deborah L. Wince-Smith
President and CEO
Council on Competitiveness
202-682-4292
www.compete.org