Nuclear waste storage sites are a subject of intense controversy and debate; nobody wants the radioactive remnants in their backyard. Now, a collaboration between Berkeley Lab, Pacific Northwest National University (PNNL), Brown University and Nvidia has yielded new insights for nuclear waste remediation using the joint power of supercomputing and deep learning. Their work will be presented at the SC19 Deep Learning on Supercomputers workshop.
The researchers leveraged physics-informed generative adversarial networks (“GANs”), which have been used to analyze everything from human faces to synthetic universes and particle physics.
“In science we know the laws of physics and observation principles – mass, momentum, energy, etc.,” said George Karniadakis, a professor of applied mathematics at Brown and co-author of the research. “The concept of physics-informed GANs is to encode prior information from the physics into the neural network. This allows you to go well beyond the training domain, which is very important in applications where the conditions can change.”
“George and his group at Brown have pioneered the approach of incorporating physics into GANs and using them to synthesize data – in this case, subsurface flow fields,” explained Prabhat, co-author of the paper and head of the Data and Analytics Services Team at Berkeley Lab’s National Energy Research Scientific Computing Center.
The research addressed the “Hanford Site,” which served as a plutonium production site for the Manhattan Project, then as the site of a plutonium production reactor and eight other nuclear reactors. This went on for nearly 50 years, after which point tens of millions of gallons of hazardous waste in underground tanks and 100 square miles of contaminated groundwater lingered alongside a major river in south-central Washington state.
For the last three decades, cleanup has been underway. Sensor-equipped wells have been collecting data about the site’s groundwater quality and geology – but they’re not quite enough. “Estimating the Hanford Site properties from data only would require more than a million measurements,” said Alex Tartakovsky, co-author of the paper and a computational mathematician at PNNL, “and in practice we have maybe a thousand. The laws of physics help us compensate for the lack of data.”
They trained the GAN on the Summit supercomputer, which (as of the June 2019 Top500 list) remains the world’s fastest publicly-ranked supercomputer at 148.6 Linpack petaflops. The team achieved peak and sustained performance of 1.2 exaflops, scaling to 27,504 of Summit’s Nvidia V100 GPUs and 4,584 of its nodes.
“Achieving such a massive scale and performance required full stack optimization and multiple strategies to extract maximum parallelism,” said Houston. “At the chip level, we optimized the structure and design of the neural network to maximize Tensor Core utilization via cuDNN support in TensorFlow. At the node level, we used NCCL and NVLink for high-speed data exchange. And at the system level, we optimized Horovod and MPI not only to combine the data and models but to handle adversary parallel strategies. To maximize utilization of our GPUs, we had to shard the data and then distribute it to align with the parallelization technique.”
This physics-informed GAN, trained by HPC, allowed the researchers to quantify their uncertainties about the subsurface flow in the site. To start, they used synthetic data to validate the accuracy of their model. “The initial purpose of this project was to estimate the accuracy of the methods, so we used synthetic data instead of real measurements,” Tartakovsky said. “This allowed us to estimate the performance of the physics-informed GANS as a function of the number of measurements.”
For now, the work remains focused on validation, but the researchers hope to scale these practices to real-world data and conditions in future studies. They also tout the applicability of their deep learning research for future DOE research in other fields.
“This is a new high-water mark for GAN architectures,” Prabhat said. “We wanted to create an inexpensive surrogate for a very costly simulation, and what we were able to show here is that a physics-constrained GAN architecture can produce spatial fields consistent with our knowledge of physics. In addition, this exemplar project brought together experts from subsurface modeling, applied mathematics, deep learning, and HPC. As the DOE considers broader applications of deep learning – and, in particular, GANs – to simulation problems, I expect multiple research teams to be inspired by these results.”
About the research
The paper discussed in this article — “Highly-scalable, physics-informed GANs for learning solutions of stochastic PDEs” — will be presented at SC19. It was written by Liu Yang, Sean Treichler, Thorsten Kurth, Keno Fischer, David Barajas-Solano, Josh Romero, Valentin Churavy, Alexandre Tartakovsky, Michael Houston, Prabhat and George Karniadakis and can be accessed here.
Read the original article discussing this research by Kathy Kincade here.