Near Munich, at the Leibniz Supercomputing Centre, magnificent work was afoot. The researchers (a team from Leibniz, Intel and the Australian National University) were producing the largest turbulence simulations in history: a series of interstellar reconstructions, some comprised of more than one trillion grid elements, revealing the stretches and folds of the universe by which stars are birthed.
Then, they visualized the results.
Turbulent flow simulations are often used to characterize systems like weather and industrial design, but they can also illuminate astrophysical elements such as cosmic ray particles, magnetic fields and the flames of supernovae. These researchers sought to illuminate the formation of stars by examining the interplay between supersonic turbulence, subsonic turbulence and magnetic fields.
The simulations at Leibniz, which ran up to a resolution of 100483, included both hydrodynamical (HD) and magneto-hydrodynamical (MHD) runs. They were run on the SuperMUC-NG supercomputer at Leibniz, a Lenovo system with nearly 6,500 nodes (outfitted with Intel 8174 Xeon Skylake CPUs), main memory of 719 TB, and a Linpack mark of 19.5 petaflops – placing it 9th on the most recent Top500 list.
In the course of this research, the team put SuperMUC-NG to work: “the 100483 HD simulation alone,” the researchers wrote, “required about 131 TB memory, over 23 TB of disk space per snapshot (with more than 100 snapshots produced during the simulation campaign), and 45 million CPU-h of computing time.”
The researchers were faced with correspondingly large challenges when visualizing the enormous datasets that resulted. Basing their approach on a custom version of VisIt and the Intel OSPRay rendering engine, they utilized ray tracing, which they described as “particularly well-suited” to run on highly parallel architectures like those of SuperMUC-NG due to “excellent parallel scalability.” They also performed the visualizations and post-processing on the same machine that ran the simulations, reducing the need to transfer data, adapt to new hardware or generally adjust the computational workflow.
The result: the first simulations to resolve the transition between supersonic and subsonic turbulence, known as the “sonic scale” – and a brilliant, otherworldly rendering of the turbulent flows and dense “filament” distributions that give birth to the “seeds of newborn stars.” The video below, provided by Leibniz, illustrates the outcome.
About the research
The research described in the article has been submitted to Parallel Computing as “Visualizing the world’s largest turbulence simulation.” It was written by Salvatore Cielo, Luigi Iapichino, Johannes Gunther, Christopher Federrath, Elisabeth Mayer and Markus Wiedemann. It can be accessed at this link.