The race to blend deep learning and first-principle simulation to speed up solutions and scale up problems tackled is one of the most exciting research areas in computational science today. This year’s ACM Gordon Bell Prize winner announced today at SC20 makes significant progress in that direction.

The work by a team of researchers from China and the U.S.  – Pushing the limit of molecular dynamics with ab initio accuracy to 100 million atoms with machine learning – used a machine learning protocol while retaining rigorous ab initio accuracy. The authors and their affiliations are listed at the end of the article. Here’s an excerpt from their paper:

“We report that a machine learning-based simulation protocol (Deep Potential Molecular Dynamics), while retaining ab initio accuracy, can simulate more than 1 nanosecond-long trajectory of over 100 million atoms per day, using a highly optimized code (GPU DeePMD-kit) on the Summit supercomputer. Our code can efficiently scale up to the entire Summit supercomputer, attaining 91 PFLOPS in double precision (45.5% of the peak) and 162/275 PFLOPS in mixed-single/half precision.

“The great accomplishment of this work is that it opens the door to simulating unprecedented size and time scales with ab initio accuracy. It also poses new challenges to the next-generation supercomputer for a better integration of machine learning and physical modeling,” they write.

Not only is the work impressive on its own, but it has implication for the forthcoming of exascale machines. “The optimization strategy presented in this paper can also be applied to other many-core architectures. For example, it can be easily converted to the Heterogeneous-compute Interface for Portability (HIP) programming model to run on the next exascale supercomputer Frontier, which will be based on AMD GPUs,” wrote the researchers[i].

Frontier (at OLCF) is now expected to the first U.S. exascale system brought online.

In presenting the award, Bronis R. de Supinski, chair of the 2020 Gordon Bell Award committee and CTO of Lawrence Livermore National Laboratory, praised the innovative approach and said “The work achieved outstanding levels of performance on the summit supercomputer at Oak Ridge National Laboratory, in addition to showing that the overall approach significantly lowers time to solution for computationally demanding problems.”

It is best to read their paper directly. Here’s what the researchers say about their key contributions:

“To effectively harness the computing power offered by the heterogeneous system architecture of Summit, our goal is to migrate to GPUs almost all computational tasks and a significant amount of communication tasks. Due to the relatively limited size of the computational granularity in the DP model, a straightforward GPU implementation encounters many bottlenecks and is thus not efficient. As such, our main algorithmic innovations are the following:

• We increase the computational granularity of DeePMD by introducing a new data layout for the neighbor list that avoids branching in the computation of the embedding matrix.
• The elements in the new data structure of the neighbor list are compressed into 64-bit integers for more efficient GPU optimization of the customized TensorFlow operators.
• We develop mixed-precision computation for the DP model. Computationally intensive tasks are performed with single or half precision without reducing the accuracy of the physical observables.”

This year there were six finalists for the Gordon Bell Prize each impressive in its own right. One ran a successful simulation Square Kilometer Array project to simulate its data processing workflow on Summit. The data flow for SKA will be immense. Another looked at accelerating graph-based datamining (also on Summit) with an eye towards scanning of bio-research literature and a third GBP finalist ran high-resolution weather simulations on Fugaku. Below are two slides from the team’s SC20 presentation and here is a link to the video.

Here are summaries of the other five Gordon Bell Prize finalists with links to their papers:

A 1024-member ensemble data assimilation with 3.5-km mesh global weather simulations

Numerical weather prediction (NWP) supports our daily lives. Weather models require higher spatiotemporal resolutions to prepare for extreme weather disasters and reduce the uncertainty of predictions. The accuracy of the initial state of the weather simulation is also critical; thus, we need more advanced data assimilation (DA) technology. By combining resolution and ensemble size, we have achieved the world’s largest weather DA experiment using a global cloud-resolving model and an ensemble Kalman filter method. The number of grid points was ~4.4 trillion, and 1.3 PiB of data was passed from the model simulation part to the DA part. We adopted a data-centric application design and approximate computing to speed up the overall system of DA. Our DA system, named NICAM-LETKF, scales to 131,072 nodes (6,291,456 cores) of the supercomputer Fugaku with a sustained performance of 29 PFLOPS and 79 PFLOPS for the simulation and DA parts, respectively. (link to paper)

Processing full-scale square kilometre array data on the summit supercomputer

This work presents a workflow for simulating and processing the full-scale low-frequency telescope data of the Square Kilometre Array (SKA) Phase 1. The SKA project will enter the construction phase soon, and once completed, it will be the world’s largest radio telescope and one of the world’s largest data generators. The authors used Summit to mimic an end-to-end SKA workflow, simulating a dataset of a typical 6 hour observation and then processing that dataset with an imaging pipeline. This workflow was deployed and run on 4,560 compute nodes, and used 27,360 GPUs to generate 2.6 PB of data. This was the first time that radio astronomical data were processed at this scale. Results show that the workflow has the capability to process one of the key SKA science cases, an Epoch of Reionization observation. This analysis also helps reveal critical design factors for the next-generation radio telescopes and the required dedicated processing facilities. (link to paper)

Toward realization of numerical towing-tank tests by wall-resolved large eddy simulation based on 32 billion grid finite-element computation

To realize numerical towing-tank tests by substantially shortening the time to the solution, a general-purpose Finite-Element flow solver, named FrontFlow/blue (FFB), has been fully optimized so as to achieve maximum possible sustained memory throughputs with three of its four hot kernels. A single-node sustained performance of 179.0 GFLOPS, which corresponds to 5.3% of the peak performance, has been achieved on Fugaku, the next flagship computer of Japan. A weak-scale benchmark test has confirmed that FFB runs with a parallel efficiency of over 85% up to 5,505,024 compute cores, and an overall sustained performance of 16.7 PFLOPS has been achieved. As a result, the time needed for large-eddy simulation using 32 billion grids has been significantly reduced from almost two days to only 37 min., or by a factor of 71. This has clearly indicated that a numerical towing-tank could actually be built for ship hydrodynamics within a few years. (link to paper)

Accelerating large-scale excited-state GW calculations on leadership HPC systems

Large-scale GW calculations are the state-of-the-art approach to accurately describe many-body excited-state phenomena in complex materials. This is critical for novel device design but due to their extremely high computational cost, these calculations often run at a limited scale. In this paper, we present algorithm and implementation advancements made in the materials science code BerkeleyGW to scale calculations to the order of over 10,000 electrons utilizing the entire Summit at OLCF. Excellent strong and weak scaling is observed, and a 105.9 PFLOP/s double-precision performance is achieved on 27,648 V100 GPUs, reaching 52.7% of the peak. This work for the first time demonstrates the possibility to perform GW calculations at such scale within minutes on current HPC systems, and leads the way for future efficient HPC software development in materials, physical, chemical, and engineering sciences. (link to paper)

Scalable knowledge graph analytics at 136 petaflop/s

We are motivated by newly proposed methods for data mining large-scale corpora of scholarly publications, such as the full biomedical literature, which may consist of tens of millions of papers spanning decades of research. In this setting, analysts seek to discover how concepts relate to one another. They construct graph representations from annotated text databases and then formulate the relationship-mining problem as one of computing all-pairs shortest paths (APSP), which becomes a significant bottleneck. In this context, we present a new high-performance algorithm and implementation of the Floyd-Warshall algorithm for distributed-memory parallel computers accelerated by GPUs, which we call dSnapshot (Distributed Accelerated Semiring All-Pairs Shortest Path). For our largest experiments, we ran dSnapshot on a connected input graph with millions of vertices using 4,096 nodes (24,576 GPUs) of the Oak Ridge National Laboratory’s Summit supercomputer system. We find dSnapshot achieves a sustained performance of 136×1015 floating-point operations per second (136 petaflop/s) at a parallel efficiency of 90% under weak scaling and, in absolute speed, 70% of the best possible performance given our computation (in the single-precision tropical semiring or “min-plus” algebra). Looking forward, we believe this novel capability will enable the mining of scholarly knowledge corpora when embedded and integrated into artificial intelligence-driven natural language processing workflows at scale. (link to paper)

[i] Authors for Pushing the limit of molecular dynamics with ab initio accuracy to 100 million atoms with machine learning: Weile Jia (UC Berkeley), Han Wang (Institute of Applied Physics and Computational Mathematics, Beijing), Mohan Chen (College of Engineering, Peking University), Denghui Lu (College of Engineering, Peking University), Lin Lin (UC Berkeley), Roberto Car (Princeton University), Weinan E (Princeton University), Linfeng Zhang (Princeton University)

Don’t forget to check our coverage of the winners and finalists for the 2020 Gordon Bell Special Prize for High Performance Computing-Based COVID-19 Research.