This year’s International Supercomputing Conference (ISC) kicked off yesterday in Hamburg, Germany, with a keynote from Dan Reed, presidential professor at the University of Utah and chair of the National Science Board.

Reed opened by explaining that computing’s “free lunch” was over with the end of Moore’s law and highlighting the “incredibly rapid transition” faced by computing through factors like AI, chiplets, energy pressures and shifting economics.

“We’re increasingly in a situation where, the path we’re on, if we want more performance, it means putting more Euros in the slot,” Reed said. “There’s a political limit to the maximum amount we can spend. … I’m convinced we may run out of money before we run out of physics.”

And there wasn’t much room to branch out, either. “The world we’re in now is actually a monoculture,” Reed said, citing a Top500 list replete with accelerated x86 systems powered by AMD, Intel and Nvidia. “The diversity of the ecosystem has largely disappeared.”

This, he said, wasn’t always the case.

A full stop to the punctuated equilibrium?

“The history of high-performance computing is one of punctuated equilibrium,” Reed said. “It’s this way, it’s this way, it’s this way – and then it’s something different. And then it’s that way, that way, that way – and then it’s something different. We’re in the middle of one of those phase transitions right now.”

Reed pointed out the shift to exploiting multicore architectures as the most recent example of this phenomenon – but in 2017, he said, the curve again “ran out of gas.” Reed said the HPC community was trying to “cheat death” at the end of Moore’s law, with transistors, thread performance and clock frequency plateauing but costs and energy footprints continuing to grow. (“I keep thinking we’re going to plateau on power, but people are now talking about packages that have a thermal dissipation of a kilowatt,” Reed said, drawing concerned murmurs from the audience.)

But Reed wondered whether the era of faster and faster computing – the era of “cheating death” – might be coming to an end. He drew parallels to the evolution of commercial flight, which stopped speeding up with the rise and fall of the supersonic Concorde jet, which was not economically viable. To date, supersonic commercial travel hasn’t returned. “The locus of innovation changed,” Reed said. “Don’t misunderstand – we continued to innovate, but we shifted, because the economics did not really support supersonic transport.”

So yes, Reed argued, we theoretically could create more and more powerful computers at higher and higher costs. But: “There’s no scenario I can see in which we’re willing to stand up a €50 billion supercomputer – not unless it is some core national security issue that drives topics we can’t talk about here. Certainly, science is never going to drive that story forward. … There have to be other drivers that make it work.”

And, to be clear, there is a need to “make it work.” Reed cited a U.S. National Academies study conducted on behalf of the National Nuclear Security Administration (NNSA) that concluded that business-as-usual computing would not be adequate for the post-exascale era.

Money, money, money

Reed argued that the metaphorical “head of the dog” had shifted from technical computing to AI and hyperscalers. “That’s where the money is,” he said, showing a graph of market caps for traditional computing companies – e.g. Atos, Lenovo, HPE – against those of the AI and cloud giants.

“There’s a good chance that ChatGPT’s training required over 3×10^25 flops,” Reed said. “That’s months to years over exascale-class computing. Think about any computational model we would run that we could justify devoting a calendar year to running that model. That’s a pretty short list – not because there aren’t problems that might require that, but because the political dynamics of sharing a machine across domains are very real. The economics are what make this work.”

Moreover, Reed said, those giants are starting to wonder why they’re buying open-market processors and accelerators when they have the money to design and optimize hardware for their own specific workloads. He ran through a series of processors designed for the cloud, noting that only one – the Ampere One – was purchasable with money. “The hardware that the hyperscalers are developing is not available for purchase,” he said, calling it a “sea change” in computing.

Reed said this made sense, with fewer and fewer vendors likely to accept the financial, technical and political risks of a paradigm that relied on ever-larger machines powered by untested hardware. In the U.S., he said, the list was basically down to just one. And what if they were to exit?

“What happens if we show up and nobody wants to play?” Reed asked.

A change in worldview

In the past, when computing slowed, it had always been clear what the next trend would be. “It’s less obvious what bandwagon to jump onto now,” Reed said. (“Quantum… I don’t know,” he added. “We’ve got a long way to go before we get to thousands of reliable qubits.”)

As he concluded his keynote, Reed mostly focused on the lens that he believed the HPC community needed to use in its introspection. “We have to think about a different world where the locus of innovation has shifted,” he said. “Although we still drive innovation, we’re not the primary driver anymore.”

This change in status, he said, was accompanied by a shifting manufacturing landscape and intense geopolitical competition.

“Despite all that, we’re surrounded by opportunities – opportunities in new materials, and devices, and architectures, and algorithms and software,” Reed said (“You can build a chiplet for the price of a startup!”). “What we have to do is seize the opportunity and let go of some of our existing worldview.”

Reed referenced the Sapir-Whorf hypothesis, which argues that the nature of a language affects how its speakers view and interact with the world. The HPC community, he suggested, needed to shift its perspective. In particular, he said, the community needed to see where the money in computing was going, follow it, and consider different models of collaboration and partnerships.

“We have to follow the money,” he said. “But we also have to follow the culture and the technology.”