May 20, 2021 — Jason Hill sometimes jokes he knocks on wood for a living.

“In the past year, I’ve worn a hole in my desk,” he said.

As risk manager for the Oak Ridge Leadership Computing Facility (OLCF), Hill spends his days—and some sleepless nights—pondering not what kind of scientific breakthroughs Oak Ridge National Laboratory’s supercomputers can make possible but what can stand in the way. The countdown to launch for Frontier, the nation’s first exascale supercomputer, increased those concerns—and that was before a viral pandemic shut down the world.

“I definitely didn’t have that on the bingo card,” Hill said. “When we first talked about the coronavirus and what might happen, it was in February 2020, and the virus was still seen as just a problem overseas. Some team members at the time wondered, ‘Why are we dedicating so much time to this?’ But shortly thereafter came the directive to work from home, and now here we are over a year later.”

The planning—and yes, worrying—paid off. The Frontier system remains on track for delivery by late 2021, more than a year into the greatest worldwide disease outbreak in a century and amid shocks to supply lines from Albania to Zimbabwe.

“We got ahead of the problem, if only by a couple of weeks,” Hill said. “That’s why we plan, for the situations we expect and don’t expect. That we’ve been able to stay on track is a huge credit to the design and construction teams who are part of this amazing group of people. My job as the risk manager is to just help them do theirs the best way they can and try to eliminate uncertainty along the way.”

The power of exascale speed, capable of over a quintillion calculations per second—more than 10¹⁸, or a billion billion—promises opportunities to help solve some of the greatest challenges in science when the HPE Cray system opens to full user operations in 2022. Teams of scientists will be lined up, waiting to tackle challenges such as screening new cancer treatments, building better nuclear reactors, and probing the origins of the universe.

Hill’s priorities tend to be more immediate. Will the system arrive on time? Will unexpected costs sap the budget? Which parts of the plan could stumble—and which solutions will be effective? And once all the parts that comprise Frontier finally arrive, will they fit together and work as intended when the requisite dignitaries show up to cut the ribbon and flip the switch?

“Even during the pandemic, the greatest risk has been and probably always will be the scale of the system,” Hill said. “The first time it’s ever put together as one machine is when it shows up here at our facility. The application teams can’t fine-tune the codes until we have all the hardware, so until it’s all put together, we don’t have the ability to measure full system performance.”

Constant questions and point-by-point planning can cut down the guesswork. Hill approaches each project like a detective sizing up a case, noting the good and the bad, the known and the unknown.

He and other members of the OLCF team draw on decades of experience in building up and maintaining supercomputer systems from Jaguar to Titan to Summit, the nation’s fastest such machine until Frontier powers up. Hill spent a decade as an on-call system administrator, helping maintain the file systems supporting those computers and answering his share of late-night calls and emails.

“I’ve been one of the people receiving, configuring, and operating computer systems throughout my career, and that experience helps me ask leading questions of the project team that is executing the work but may not speak the same language as the team responsible for creating the project’s performance baseline,” he said. “We start with what project controls call the iron triangle—cost, schedule, and scope—and build out from there. How certain are we on each of these? Where can we find events that will affect the project’s ability to deliver on any of these goals?

“Some people choose to see risk management as a roadblock to their work, but it’s really just about helping achieve the end goal. We look at the risks to the project’s completion, but we also look at opportunities—a risk that can benefit the project, too. As the project moves along, events become more certain, and estimates are fine-tuned as we go. Then we can have multiple risk-mitigation plans locked and ready to go if and when an event occurs.”

Having the team’s full support just makes the process smoother.