Datanami
EnterpriseAI
HPCwire Japan
QCwire
HPC & AI Wall StreetFeb. 1, 2021 — The Texascale Days event in December 2020 provided an opportunity for nine research groups to use large sections of the National Science Foundation-funded Frontera supercomputer at the Texas Advanced Computing Center (TACC) to solve problems that in many cases have never been attempted.
What follows are abbreviated first-person accounts of how some of the researchers used their one-day access to the Frontera system to run breakthrough simulations.
Rommie Amaro, professor and endowed chair of chemistry and biochemistry at the University of California San Diego and winner of the 2020 Gordon Bells Prize for COVID-19 Research
We used Texascale Days to continue our efforts to build and simulate the full SARS-CoV-2 envelope using all-atom molecular dynamics simulations with NAMD2.
The virion has ~300 million atoms with explicit solvent. A significant challenge of working with systems of this size and complexity is ensuring the stability of the membrane, which requires extensive equilibration of hundreds of millions of atoms.
We were grateful for the opportunity to use nearly 4,000 Frontera nodes to expedite our equilibration of the virion. The additional data we obtained from this run will help us with method development, as well as provide an improved understanding of how bilayer dynamics are impacted by surface pressure, the presence of the spike protein, and viral envelope curvature.
INFLUENCE OF GRAVITY WAVES ON STELLAR GASES
Paul Woodward, professor in astronomy, University of Minnesota – Twin Cities
In mid-December 2020, our research team was given access to the entire Frontera machine at TACC for 24 hours. We used this opportunity to perform special, highly resolved simulations of a main sequence star of 80 solar masses. We ran two such simulations side-by-side, using 3,672 nodes for each run, in order to see the effects of radiation diffusion on the results.
These simulations are a part of an investigation of core convection in the massive main sequence stars. We study the mixing of gases at the boundary of the central convection zone, and the behavior and observational signatures of internal gravity waves set up by the convection in the stably stratified outer envelopes of such stars. When they reach the stellar surface, these gravity waves can be detected by satellites like TESS that monitor the time-varying brightness of stars in searching for exoplanets.
Not only do these gravity waves carry to the surface information about the gas motions in the core of the star, but they can, in principle, cause material mixing and, for rotating stars, angular momentum transport in the envelope. Teasing such subtle effects out of our simulation data requires our code to deliver very high accuracy and to follow the behavior of the gas over a sufficiently long period. We are addressing both these requirements for our PPMstar code, and we could not hope to obtain the necessary fidelity of the simulation without the tremendous computing power of the Frontera system.
AI + SIMULATION DRUG DISCOVERY PIPELINE FOR COVID-19
Andre Merzky, senior research programmer, Rutgers University, Radical Group; Arvind Ramanathan, principal investigator, Argonne National Laboratory
We ran a similar drug screening pipeline to our last Texascale runs. Where the first Texascale run was used to ensure we could efficiently run at scale, the second was used to test different configurations of our software stack to ensure high resource utilization under different workloads. That again is something we cannot easily test at lower scale: results obtained at, say, 1,000 nodes do not easily translate into similar numbers at 8,000 nodes.
The results were mixed: we had to exclude a number of configurations which turned out to choke our communication system, but at the end managed to determine a configuration which allows us to run the pipeline at about 94% resource utilization in steady state, which we are really happy with (this is for production data, not benchmarks!).
In summary, we consider the pipeline stable at small and large scales now, and the production runs are currently faster than the preparatory work in the lab needed to feed data into the production pipeline. In other words: we are fast enough for the time being as not to slow down the overall research pipeline.
SCALING QUANTUM EXPRESSO TO THE LIMIT
Feliciano Giustino, professor of physics and director of the Center for Quantum Materials Engineering, Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin; Hyungjun Lee, lead software architect, Oden Institute
The Center for Quantum Materials Engineering at The University of Texas at Austin specializes in the computational modeling and design of functional materials at the atomic scale using quantum mechanics. One of our overarching aims is to harness the interaction between electrons and lattice vibrations (phonons) in materials design. To this aim, we lead the development of EPW, an open-source community code of the Quantum ESPRESSO simulation suite for predictive calculations of electron-phonon physics and related materials properties.
In 2020, we refactored EPW in preparation for exascale computing, and we extended the previous one-level MPI parallelization to a new hybrid two-level MPI and OpenMP parallelization. This hybrid hierarchical parallelization scheme allows us to reduce the overhead in MPI communications as well as the memory footprint of the calculations. During Texascale Days in December 2020, we had the opportunity to test this new hybrid parallelization scheme and assess the readiness of the EPW code for future exascale simulations on leadership-class supercomputers.
With exclusive access to Frontera, we benchmarked the EPW code with calculations on the superconductor MgB2 (magnesium diboride) on up to 7,840 nodes (out of a total of 8,008). We demonstrated very good strong-scaling performance, reaching 86% of the ideal speedup on 439,040 cores. These tests are documented on the project webpage.
MASSIVE TORNADOES – BUILDING TO THE FULL STORM
Leigh Orf, atmospheric scientist, Space Science and Engineering Center, University of Wisconsin
I simulate tornado-producing thunderstorms. The fidelity of the simulations goes up with increased resolution (a smaller spacing between grid points on a cubic lattice), specifically with regards to the behavior of the tornado. However, doubling the resolution leads to a 16x increase in the amount of resources required. Things get crazy fast, and it’s easy to use up all available resources.
As resolution is increased, the model’s time step is decreased in order to maintain stability. (This is a universal issue in fluid dynamics codes.) Not only do we need to do more calculations owing to more grid points, but we must march forward in tinier time increments to keep the model from blowing up.
I was given 7,700 nodes to run on for one day. I started the simulation from scratch in an environment that matches one of the big tornadoes that occurred during the 27 April 2011 southeastern U.S. outbreak. We made it from t=0 to t=1700 seconds, with a low I/O load since not much goes on early in these simulations.
Simulations generally need to go out to t=9000 seconds [150 minutes] or so to complete. What we have so far is a perfectly reasonable looking storm, early in its life cycle. Because it is so early in the simulation, nothing much of note has happened. It would take another four or so days at 7,700 nodes to get the simulation to the point where interesting thing happen, if they even do happen. (There is no guarantee that the simulated storm will produce a tornado — much like what happens with tornado warnings that are issued when no tornado is observed).
If this were a movie, we’d only be about 20 minutes into it! Hopefully, I’ll be able to continue the simulation during future Texascale Days.
EVOLVING THE EARLY UNIVERSITY IN 24 HOURS ON FRONTERA
Simeon Bird, professor of physics and astronomy, University of California, Riverside; Tiziana DiMatteo, principal investigator, Carnegie Mellon University
Read full first-person account.
TURBULENCE IN THE COSMOS — AND ON EARTH
Alexei Kritsuk, researcher, Center for Astrophysics and Space Sciences, University of California, San Diego; Michael Norman, principal investigator, UCSD
Most of the allocated 24-hour period was used for a production run, which employed 3,900 nodes or roughly half of the machine. This simulation evolved a Mach 1 turbulence model on a grid of 2048^3 points for approximately 3.3 dynamical times, which is sufficient to approach a statistically stationary regime of the turbulence.
These simulations are addressing important applications in astrophysics and cosmology such as interstellar turbulence and star formation, turbulence in the dilute intergalactic medium, and polarized galactic foregrounds to the cosmic microwave background radiation. They also address the basic physics of compressible turbulence, which has a wide range of potential applications in astrophysics, physics of the solar wind, magnetospheric physics, atmospheric sciences, and vehicle design.
In the near future, we plan to evolve the model further in time and collect statistics characterizing energy transfer across scales in compressible turbulence. The planned run shall not be the biggest in terms of the grid resolution, but will provide unprecedented scale separation and very accurate statistics of transonic turbulence thanks to a class of novel, computationally efficient high-order methods implemented in the code.
Source: Aaron Dubrow, TACC
