Thomas Sterling: ‘I Think We Will Never Reach Zettaflops’

By Nicole Hemsoth

May 7, 2012

As supercomputing makes its way through the petascale era, the future of the technology has never seemed so uncertain.  HPC veteran Thomas Sterling, Professor of Informatics & Computing at Indiana University, takes us through some of the most critical  developments in high performance computing, explaining why the transition to exascale is going to be very different than the ones in the past and how the United States is losing its leadership in HPC innovation.

HPCwire: Do you have a sense that other regions, China and Europe in particular, are closing the HPC leadership gap that the US has enjoyed for so long? If so, do you think this is more the result of technology democratization rather than government policy choices?

Thomas Sterling: It is clear that other regions are not closing the HPC leadership gap; they are widening it. Through a series of actions in both the EU and Asia the momentum is shifting overseas where once it was entrenched within the US. The Europeans through their EESI and forthcoming EESI2 efforts are making dramatic strides in planning towards a EU-dominant exascale trajectory. The Russians as well are putting in to place a tightly coordinated exascale program combining Moscow State University, T-Platforms, and government, not simply to duplicate prior US methods but to innovate beyond them.

Asia today represents the biggest surge in top performing machines with the 10 petaflops Kei machine in Kobe Japan and the 2.6 petaflops Tianhe-1A system in Tianjin. Perhaps more defining, if somewhat less powerful, is the Sunway BlueLight, a petaflops-class machine built entirely of Chinese microprocessors. Less clear is the degree to which all of these machines are being applied effectively to end-purpose applications, but it is only a matter of time until these strengths push other science and industry objectives beyond the US sphere of influence.

The reason is a mix of both technology democratization and government policy. Neither is working in favor of the US. While the US will be deploying 10 and 20 petaflops machines over the next couple of years, it is clear that the momentum in innovation is off-shore. What may reverse this distressing trend is the new — no pun intended — energy at DOE in driving towards exascale through innovative advancements in software, programming methods, application parallel algorithms, and eventually at least to some degree in hardware.

HPCwire: How is the transition to exascale different from the other HPC milestone transitions — terascale and petascale — that we’ve passed through?

Sterling: The transition to exascale is different from the two previous tri-decade transitions through which we have passed, and in two fundamental ways: one related to the past, and the other the future. By the early 1990’s, the “killer micro,” cheap DRAM, and the emergence of system area networks manifest as MPPs (for example, Intel Touchstone Delta) and commodity clusters (for example, my own Beowulf project). These combined with the foundational Communicating Sequential Processes execution model reflected by the message-passing programming model established a formula to match weak-scaling workloads to VLSI component technologies.

At about 11 year intervals this delivered teraflops-scale computing, ASCI Red in 1999, and petaflops-scale, Roadrunner in 2008. However, this highly successful strategy is unlikely to facilitate the realization of exascale computing, except perhaps for some specialized and carefully crafted workloads. This is because the means adopted by this approach to address key factors of performance degradation will no longer prove adequate.

For example, the fine-grained instruction level parallelism and coarse-grained concurrent processes will not provide sufficient efficient parallelism to meet the billion-plus-way parallelism requirement of exascale. Static resource allocation and task scheduling is insufficient to provide the necessary efficiency or scalability as well as the introspective techniques required for reliability and power management. I expect the need for new programming models, which may include but not be limited to variations of previous techniques, will be essential for communicating between user applications and underlying execution systems.

As I have asserted in the past, a new execution model as an embodiment of a paradigm shift will drive this transition from old systems to new. We have done this before in the case of scalar to vector and SIMD, and again from these to message passing, MPPs, and clusters. We are now simply — or not so simply — facing another phase shift in HPC system programming, structure, and operation.

Exascale is also different because unlike previous milestones, it is unlikely that we will face yet another one in the future. These words may be thrown back in my face, but I think we will never reach zettaflops, at least not by doing discrete floating point operations. We are reaching the anvil of the technology S-curve and will be approaching an asymptote of single program performance due to a combination of factors including atomic granularity at nanoscale.

Of course I anticipate something else will be devised that is beyond my imagination, perhaps something akin to quantum computing, metaphoric computing, or biological computing. But whatever it is, it won’t be what we’ve been doing for the last seven decades. That is another unique aspect of the exascale milestone and activity. For a number, I’m guessing about 64 exaflops to be the limit, depending on the amount of pain we are prepared to tolerate.

HPCwire: What’s the biggest hardware challenge to attain exascale computing?

Sterling: The usual response to this question is either “power” or “resilience” and these are certainly critical challenges to attaining exascale. Depending on the analysis of choice, without innovative ways of managing vertical and lateral data movement power estimates based on anticipated technology trends suggest an order of magnitude greater power demand than is considered practical. Single point failure modes of systems comprising hundreds of millions of cores will exhibit mean-time-to-interrupt on the order of minutes or many seconds, much less than the expected time to service a checkpoint or restart cycle using conventional methods.

While both are clearly important, I think the biggest hardware challenge is architecture. This may surprise many of our colleagues because there is a general expectation that the system architecture is likely to be an evolutionary extension of the current mix of multicore sockets and GPU accelerators. This view is driven by the number one concern, which is parallelism and the need to expose and exploit it. Not only will the system architecture have to provide sufficient hardware concurrency of on the order of a billion or more simultaneous actions for the throughput requirement, it will have to use more of it as a latency mitigating method requiring additional architecture change.

Further, it will have to incorporate mechanisms to reduce overheads in order to make effective use of finer granularity tasks (e.g., lightweight user threads) such as the instantiation of remote actions. Support for advanced forms of global address spaces, their management, and address translation will be required in support of randomly distributed global data (e.g., dynamic graphs). New mechanisms for efficient semantically rich synchronization and continuation (control object) migration to manage locality of control will be part of future designs if they are to succeed at unprecedented scale.

Additional hardware mechanisms will be required for fault tolerance including error detection, isolation, in-memory checkpointing, and recovery through reconfiguration. Power reduction will demand active sensor and control hardware mechanisms to continuously adjust energy usage based on application demands. New processor cores and their relationship to memory (for example, processor in memory) for superior bandwidth, reduced latency, and lower power will further drive hardware innovation needs.

HPCwire: How about software challenges?

Sterling: Every advance in hardware will require corresponding changes in software. But the software challenge extends beyond this supporting role. Perhaps most critical is the development of performance-oriented runtime system software for scalable computing. Such software will include dynamic scheduling for lightweight user threads, message-driven computation for moving the work to the data, global address space management, and again efficient support for powerful synchronization objects like the futures construct to eliminate the use of global barriers and enable asynchrony control through dynamic adaptation.

The ParalleX execution model, as well as the HPX-3 prototype runtime system and the ETI SWARM that embody many of its principles, are two examples that support these goals, even on today’s conventional parallel distributed system architectures. But they are only a beginning as the needs for a new generation of fault tolerance and energy management control will be required, too. With a billion cores, their memory hierarchy, and layered communications, a new scalable and robust operating system will be needed. A new software architecture is required to provide a context in which both runtime and operating system need to be mutually designed.

One major challenge is a new interface and protocol definition between the runtime and OS that enables a unique dynamic for a symbiotic relationship of mutual and interactive support. The presence of a performance runtime system also imposes new demands and class of functionality on future compilers that now play a very different role given the existence of a runtime and the exploitation of introspective techniques. These changes percolate up to affect the need for new application programming interfaces. In combination this suggests possibly an entirely new software stack for exascale computing implying that it is not too early to be investing in and conducting research in these areas already.

HPCwire: Do you think the industry will provide suitable manycore hardware and software products that can be applied to high performance computing — for exascale, but also for HPC in general?

Sterling: This is a complicated question with the answer depending on what is meant by “the industry,” “suitable,” and “products.” I am not sanguine about the current path and offerings as incrementally extended to exascale, and industry roadmaps that assume this approach will have shrinking impact on the total range of problems that will eventually apply exascale capability to their solution space. I don’t think we as a community know enough at this point to establish what the right hardware/software machine is for general purpose exascale or even if such a system is possible within the constraints of parallelism, energy, and reliability.

Therefore claims that particular vendors have it under control are of limited value at best. The design space is just too complicated, prior methods for scaling to Moore’s Law apply to a decreasing degree, whole new modalities demanding advanced runtime components are yet to be derived but are essential, and generality is already diminishing to a worrisome degree for such assertions to have meaningful validity.

Nonetheless, industry will deliver the systems that will be used in the next decade. There is no other choice. It is clear that vendors would prefer not to have to retool and this is true for users as well. To do so will involve a degree of disruption that would be best avoided if it were possible. And for a portion of the overall workload, even at exascale, this may prove to be possible. But such systems are a placebo to an ailing HPC community that if not in triage, is already showing symptoms of underlying conditions that require attention.

The big hurdle is when industry fully embraces the need to address the system wide parallel computing challenge at the processor core level, refactors the physical and logical relationship between cores and memory banks for minimum latency and maximum bandwidth, and transitions from static to dynamic execution models and system software. I do expect this to happen but not without strong push from the user mission-critical agencies.

Thomas Sterling will be delivering the Wednesday keynote at this year’s International Supercomputing Conference (ISC’12), which will take place in Hamburg, Germany from June 17-21. His presentation will examine the achievements over the past 12 months in high performance computing.

Related Articles

The Power to Flop

The Bumpy Road to Exascale: A Q&A with Thomas Sterling

Taking a Disruptive Approach to Exascale

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

GTC21: Dell Building Cloud Native Supercomputers at U Cambridge and Durham

April 14, 2021

In conjunction with GTC21, Dell Technologies today announced new supercomputers at universities across DiRAC (Distributed Research utilizing Advanced Computing) in the UK with plans to explore use of Nvidia BlueField DPU Read more…

The Role and Potential of CPUs in Deep Learning

April 14, 2021

Deep learning (DL) applications have unique architectural characteristics and efficiency requirements. Hence, the choice of computing system has a profound impact on how large a piece of the DL pie a user can finally enj Read more…

GTC21: Nvidia Launches cuQuantum; Dips a Toe in Quantum Computing

April 13, 2021

Yesterday Nvidia officially dipped a toe into quantum computing with the launch of cuQuantum SDK, a development platform for simulating quantum circuits on GPU-accelerated systems. As Nvidia CEO Jensen Huang emphasized i Read more…

Nvidia Aims Clara Healthcare at Drug Discovery, Imaging via DGX

April 12, 2021

Nvidia Corp. continues to expand its Clara healthcare platform with the addition of computational drug discovery and medical imaging tools based on its DGX A100 platform, related InfiniBand networking and its AGX develop Read more…

Nvidia Serves Up Its First Arm Datacenter CPU ‘Grace’ During Kitchen Keynote

April 12, 2021

Today at Nvidia’s annual spring GPU technology conference, held virtually once more due to the ongoing pandemic, the company announced its first ever Arm-based CPU, called Grace in honor of the famous American programmer Grace Hopper. Read more…

AWS Solution Channel

Volkswagen Passenger Cars Uses NICE DCV for High-Performance 3D Remote Visualization


Volkswagen Passenger Cars has been one of the world’s largest car manufacturers for over 70 years. The company delivers more than 6 million automobiles to global customers every year, from 50 production locations on five continents. Read more…

Nvidia Debuts BlueField-3 – Its Next DPU with Big Plans for an Expanded Role

April 12, 2021

Nvidia today announced its next generation data processing unit (DPU) – BlueField-3 – adding more substance to its evolving concept of the DPU as a full-fledged partner to CPUs and GPUs in delivering advanced computi Read more…

GTC21: Dell Building Cloud Native Supercomputers at U Cambridge and Durham

April 14, 2021

In conjunction with GTC21, Dell Technologies today announced new supercomputers at universities across DiRAC (Distributed Research utilizing Advanced Computing) Read more…

The Role and Potential of CPUs in Deep Learning

April 14, 2021

Deep learning (DL) applications have unique architectural characteristics and efficiency requirements. Hence, the choice of computing system has a profound impa Read more…

Nvidia Serves Up Its First Arm Datacenter CPU ‘Grace’ During Kitchen Keynote

April 12, 2021

Today at Nvidia’s annual spring GPU technology conference, held virtually once more due to the ongoing pandemic, the company announced its first ever Arm-based CPU, called Grace in honor of the famous American programmer Grace Hopper. Read more…

Nvidia Debuts BlueField-3 – Its Next DPU with Big Plans for an Expanded Role

April 12, 2021

Nvidia today announced its next generation data processing unit (DPU) – BlueField-3 – adding more substance to its evolving concept of the DPU as a full-fle Read more…

Nvidia’s Newly DPU-Enabled SuperPod Is a Multi-Tenant, Cloud-Native Supercomputer

April 12, 2021

At GTC 2021, Nvidia has announced an upgraded iteration of its DGX SuperPods, calling the new offering “the first cloud-native, multi-tenant supercomputer.” Read more…

Tune in to Watch Nvidia’s GTC21 Keynote with Jensen Huang – Recording Now Available

April 12, 2021

Join HPCwire right here on Monday, April 12, at 8:30 am PT to see the Nvidia GTC21 keynote from Nvidia’s CEO, Jensen Huang, livestreamed in its entirety. Hosted by HPCwire, you can click to join the Huang keynote on our livestream to hear Nvidia’s expected news and... Read more…

The US Places Seven Additional Chinese Supercomputing Entities on Blacklist

April 8, 2021

As tensions between the U.S. and China continue to simmer, the U.S. government today added seven Chinese supercomputing entities to an economic blacklist. The U Read more…

Habana’s AI Silicon Comes to San Diego Supercomputer Center

April 8, 2021

Habana Labs, an Intel-owned AI company, has partnered with server maker Supermicro to provide high-performance, high-efficiency AI computing in the form of new Read more…

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from I Read more…

Intel Launches 10nm ‘Ice Lake’ Datacenter CPU with Up to 40 Cores

April 6, 2021

The wait is over. Today Intel officially launched its 10nm datacenter CPU, the third-generation Intel Xeon Scalable processor, codenamed Ice Lake. With up to 40 Read more…

CERN Is Betting Big on Exascale

April 1, 2021

The European Organization for Nuclear Research (CERN) involves 23 countries, 15,000 researchers, billions of dollars a year, and the biggest machine in the worl Read more…

Programming the Soon-to-Be World’s Fastest Supercomputer, Frontier

January 5, 2021

What’s it like designing an app for the world’s fastest supercomputer, set to come online in the United States in 2021? The University of Delaware’s Sunita Chandrasekaran is leading an elite international team in just that task. Chandrasekaran, assistant professor of computer and information sciences, recently was named... Read more…

HPE Launches Storage Line Loaded with IBM’s Spectrum Scale File System

April 6, 2021

HPE today launched a new family of storage solutions bundled with IBM’s Spectrum Scale Erasure Code Edition parallel file system (description below) and featu Read more…

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

Saudi Aramco Unveils Dammam 7, Its New Top Ten Supercomputer

January 21, 2021

By revenue, oil and gas giant Saudi Aramco is one of the largest companies in the world, and it has historically employed commensurate amounts of supercomputing Read more…

Quantum Computer Start-up IonQ Plans IPO via SPAC

March 8, 2021

IonQ, a Maryland-based quantum computing start-up working with ion trap technology, plans to go public via a Special Purpose Acquisition Company (SPAC) merger a Read more…

Leading Solution Providers


Can Deep Learning Replace Numerical Weather Prediction?

March 3, 2021

Numerical weather prediction (NWP) is a mainstay of supercomputing. Some of the first applications of the first supercomputers dealt with climate modeling, and Read more…

Livermore’s El Capitan Supercomputer to Debut HPE ‘Rabbit’ Near Node Local Storage

February 18, 2021

A near node local storage innovation called Rabbit factored heavily into Lawrence Livermore National Laboratory’s decision to select Cray’s proposal for its CORAL-2 machine, the lab’s first exascale-class supercomputer, El Capitan. Details of this new storage technology were revealed... Read more…

New Deep Learning Algorithm Solves Rubik’s Cube

July 25, 2018

Solving (and attempting to solve) Rubik’s Cube has delighted millions of puzzle lovers since 1974 when the cube was invented by Hungarian sculptor and archite Read more…

African Supercomputing Center Inaugurates ‘Toubkal,’ Most Powerful Supercomputer on the Continent

February 25, 2021

Historically, Africa hasn’t exactly been synonymous with supercomputing. There are only a handful of supercomputers on the continent, with few ranking on the Read more…

The History of Supercomputing vs. COVID-19

March 9, 2021

The COVID-19 pandemic poses a greater challenge to the high-performance computing community than any before. HPCwire's coverage of the supercomputing response t Read more…

AMD Launches Epyc ‘Milan’ with 19 SKUs for HPC, Enterprise and Hyperscale

March 15, 2021

At a virtual launch event held today (Monday), AMD revealed its third-generation Epyc “Milan” CPU lineup: a set of 19 SKUs -- including the flagship 64-core, 280-watt 7763 part --  aimed at HPC, enterprise and cloud workloads. Notably, the third-gen Epyc Milan chips achieve 19 percent... Read more…

HPE Names Justin Hotard New HPC Chief as Pete Ungaro Departs

March 2, 2021

HPE CEO Antonio Neri announced today (March 2, 2021) the appointment of Justin Hotard as general manager of HPC, mission critical solutions and labs, effective Read more…

Microsoft, HPE Bringing AI, Edge, Cloud to Earth Orbit in Preparation for Mars Missions

February 12, 2021

The International Space Station will soon get a delivery of powerful AI, edge and cloud computing tools from HPE and Microsoft Azure to expand technology experi Read more…

  • arrow
  • Click Here for More Headlines
  • arrow