Over a six month period in 2019, the ominously-named Last Journey simulation used massive supercomputing power to summon trillions of particles into the digital either, representing how mass travels across the universe. The goal: to understand the movements and formations of dark matter, which largely eludes researchers.
To do this, they ran a massive simulation, building on code from 2013’s Outer Rim simulation and using the Hardware/Hybrid Accelerated Cosmology Code (HACC) and the CosmoTools analysis framework to both run the simulation and extract information from the simulation while it was running. The researchers would progress the simulation through time to assess the effects of gravity: then, using CosmoTools, the researchers would extract information on the halos of dark matter forming throughout the simulated universe.
“When we know where the particles are at a certain point in time, we characterize the structures that have formed by using CosmoTools and store a subset of data to make further use down the line,” explained Adrian Pope, a core developer for both HACC and CosmoTools at Argonne, in an interview with Argonne’s Savannah Mitchem. “If we find a dense clump of particles, that indicates the location of a dark matter halo, and galaxies can form inside these dark matter halos.”
Rinse, lather, repeat.
“We would move particles, do analysis, move particles, do analysis,” said Pope. “At the end, we would go back through the subsets of data that we had carefully chosen to store and run additional analysis to gain more insight into the dynamics of structure formation, such as which halos merged together and which ended up orbiting each other.”
One of the largest cosmological simulations ever, the Last Journey used Argonne’s Mira supercomputer. Mira, an IBM Blue Gene/Q system, spanned 49,152 nodes, each with an IBM PowerPC A2 CPU, and an aggregate 768 TB of memory. Seven years after deployment in 2012, Mira placed 22nd on the November 2019 Top500 list with 8.59 Linpack petaflops.
Still, this marked the last journey for Mira itself. After a long production life, the system finally retired after the Last Journey simulation, in December 2019. Argonne is now setting its sights on exascale computing, with its Aurora supercomputer slated for delivery sometime next year.
“We’ve learned and adapted a lot during the lifespan of Mira, and this is an interesting opportunity to look back and look forward at the same time,” said Pope. “When preparing for simulations on exascale machines and a new decade of progress, we are refining our code and analysis tools, and we get to ask ourselves what we weren’t doing because of the limitations we have had until now.”
The data from the Last Journey will also be added to other large-scale cosmological experiments, including the Stage-4 ground-based cosmic microwave background experiment (CMB-S4), the Legacy Survey of Space and Time and others.
“This huge data set they are building will feed into many different efforts,” said Katherine Riley, director of science for the Argonne Leadership Computing Facility (ALCF). “In the end, that’s our primary mission – to help high-impact science get done. When you’re able to not only do something cool, but to feed an entire community, that’s a huge contribution that will have an impact for many years.”
To read the article from Argonne’s Savannah Mitchem discussing this research, click here.