With the exascale era underway, the HPC community is already turning its attention to zettascale computing, the next of the 1,000-fold performance leaps that have occurred about once a decade. With this in mind, the ISC 2024 organizers asked Kathy Yelick, Vice Chancellor for Research and the Robert S. Pepper Distinguished Professor of Electrical Engineering and Computer Sciences at the University of California, Berkeley, to deliver the opening keynote talk, titled “Beyond Exascale Computing.” HPCwire asked Steve Conway, senior analyst at Intersect360 Research, to interview Dr. Yelick ahead of her talk.

HPCWIRE: We’re all looking forward to your keynote talk in Hamburg. The Exascale era has just started. How important is it for the HPC community to begin addressing the beyond-exascale challenges right away? Are they even harder than the challenges the HPC community faced getting to Exascale?

Yelick: The HPC community needs to start thinking immediately about the post-exascale challenges as there are plenty of computing demands that will go beyond Exascale, and the problems are definitely harder than they were getting to Exascale. Exascale was already harder than some of the previous milestones because of the broad adoption of GPUs that have been disruptive to the software stack, from compilers and libraries to applications. Architectural innovations of this kind were important in getting to Exascale, but there is no underlying exponential here – most architectural innovations are a one-time improvement. And when they require software changes that propagate up the stack, you need an all-hands-on-deck approach. GPUs and wide SIMD units both reflect more parallelism within a compute node, and arguably, one could try to squeeze out more parallelism of that kind, but there are natural limits based on what is available in the applications.

HPCWIRE: I assume you’ll discuss the challenge of scaling with little or no help from Moore’s Law. GPUs have functioned as booster rockets for approaching Exascale on Linpack and user applications. Will another breakthrough technology be needed to get beyond Exascale, or will multiple incremental advances be enough?

Yelick: You’re right that we don’t expect much help from increasing transistor density and certainly not from an increase in clock speeds that we saw before 2005. These used to be key pieces in delivering each factor of 1000x. Density is not completely dead, as there are plans for 2-nanometer devices, but we are reaching atomic limits, and the cost of new fabrication facilities continues to grow. The question is whether we will see breakthroughs in energy efficiency, and I don’t want to discount innovations such as EUV lithography and 3D designs, which have already helped get us to where we are today. Researchers will continue working on more energy-efficient device technologies without increased clock speeds. The main levers for performance will be increased parallelism, improved data movement performance, and better computational efficiency through specialization, narrowed data types, and reduced complexity.

HPCWWIRE: Can you say more about the parallelism challenge?

Increased hardware parallelism is only useful if we can discover and express it in the applications. HPC has historically benefited from weak scaling, which increases the problem size with the parallelism. That has permitted higher-resolution climate models, more atoms in molecular dynamics simulations, and larger physical domains in general. However, it runs into limits for problems that scale superlinearly because equally scaling the problem size and parallelism can lead to impractical long runtimes. GPU parallelism and wider SIMD units are necessarily addressing strong scaling since memory capacity does not scale with parallelism, but that is likely to run into limits at the application level instead. Even when that parallelism exists, it may require more clever algorithms and the right programming mechanisms.

The question moving beyond Exascale is whether there are problems that have sufficient untapped parallelism and efficiency opportunities or if data movement costs can be improved to realize 1000x in application benefits.

HPCWIRE: Another well-known challenge is power use. AMD’s Lisa Su estimated a year ago that if built with today’s technology, a Zettascale system would need 21 gigawatts of power, the equivalent of 21 nuclear plants. Governments around the world have set aggressive goals for decarbonization and sustainability. Can you comment on the power challenge?

Yelick: It is always the case that a simple 1000x scaling of existing technology requires 1000x more power, cooling, and space, not to mention raising serious reliability concerns due to the number of components and connections between them. The same argument was made in the Petascale era, where we were looking at 2-megawatt systems and were concerned about a 2-gigawatt Exascale platform. In the end, the combination of device improvements and architectural efficiencies combined with increased parallelism in hardware and software resulted in 100x efficiency improvement, so with Frontier at OLCF, we have an Exascale system of about 20 (more precisely 21) megawatts. Bill Dally and Peter Kogge also have a very nice paper summarizing the 2008 DARPA Exascale study and how things played out in the first systems. We need at least, the same level of efficiency improvements to hit the next performance milestone, hopefully without another 10x in power.

HPCWIRE: How important do you think renewables and other alternative energy sources will be for moving beyond Exascale? Critically important or not so much?

Yelick: Given that 10x power increase going from Petascale to Exascale, the HPC community has a responsibility to consider a shift to renewable energy sources for their computing centers, but this will only have an impact if it results in an overall increase in renewables and not simply shifting from other sectors. I also want to emphasize that the energy-efficient computer designs we discussed earlier will not reduce energy use in computing. There is an insatiable demand for computing in science, business, entertainment, and government, so any benefits from improved efficiency will result in more computing. Rather than having a fixed demand level, computing is limited by budgets, infrastructure, and access.

HPCWIRE: In procurements for leadership-class supercomputers, there’s a shift toward more flexible, modular system architectures, such as a general-purpose simulation module networked to an AI module and a quantum computing module. Will it be difficult to exploit multiple modules for a single application? Will it still make sense to use a single metric for the performance of the whole system?

Yelick: There has always been some interest in optimizing systems for a given workload, whether data-intensive or algorithm-specific, as in a molecular dynamics machine. In the past, the operational advantages of having a single system architecture and the lack of a static, predictable workload in most HPC centers had limited heterogeneity. We already see changes in things like memory configurations or a system with a mixture of GPU-accelerated and CPU-only nodes. The desire to run full-system computations, including the Linpack benchmark, works against heterogeneity. Given the push for specialization, we may see more system heterogeneity, but it will be hard to use within a single application. We are seeing more complex workflows that integrate multiple coarse-grained computations, which may use a mix of architecture types. I see quantum computing not so much as an accelerator or replacement as an entirely complementary computing model.

HPCWIRE: Some users have argued that each new generation of leadership-class supercomputers has had less breadth of applicability than the prior generation because it requires more parallelism to be found in the most important applications. Is that a problem?

Yelick: The U.S. Exascale Computing Program demonstrated a large and diverse set of applications that effectively use an Exascale system, and we’ve seen similar successful applications from programs worldwide. This effort didn’t happen on its own – there was a lot of hard work on algorithms, software technology, libraries, and application development behind these. In addition to moving some of the established HPC applications in fields like chemistry, astronomy, and climate science to deliver new capabilities at Exascale, there are also new applications in genomics, scientific imaging, optimization, and clean energy. It is also the case that plenty of applications do not run at scale or are limited by data movement or insufficient fine-grained parallelism. Some of these are fundamental to the application, others are due to lack of investment in a particular code, and still others may make use of the Exascale architecture but not at scale. I don’t see this as a problem since there are high-priority problems that can be solved with these systems.

HPCWIRE: Can you briefly describe DOE activities aimed at getting beyond Exascale computing?

Yelick: I recently chaired a U.S. National Academies study on post-exascale computing for the National Nuclear Security Agency (NNSA), but I think many of the conclusions apply to DOE more broadly and to science and engineering organizations that rely on high-end computing. One of the main findings is that the current approach used for Exascale will not be sufficient for the next generation, citing the kinds of technology and architectural issues described above. The topic of AI hardware is also called out since it is an area of innovation and commercial investments. DOE may leverage that commercial ecosystem by mapping current methods to AI hardware features or exploring new AI-based approaches. DOE can also try to influence AI hardware by looking for ways in which sparsity support can benefit both AI and non-AI applications. Finally, they take advantage of the growing support for hardware specialization, developing their features for science or engineering workloads.

Steve Conway, senior analyst at Intersect360 Research, has closely tracked AI progress for more than a decade. Conway has spoken and published widely on this topic, including an AI primer for senior U.S. military leaders co-authored with Johns Hopkins University Applied Physics Laboratory.