Right after the Big Bang the universe was filled with hot, ionized hydrogen (H) and helium (He) gas, a bright torch if you will. By 400,000 years later the flame had dimmed as the gas de-ionized (atoms recaptured electrons) and radiation shifted to ‘invisible’ wavelengths. So started the Cosmic Dark Ages. Now a group of researchers have conducted the most complete simulation of the ‘re-lighting’ of the torch using Titan supercomputer (Cray) at Oak Ridge Leadership Computing Facility (OLCF) and relying heavily on its GPU (NVIDIA Tesla K20s) power.

“When the first galaxies and stars formed about 100 million years after the big bang, starlight leaked into the intergalactic medium, where most of the hydrogen and helium created in the big bang was still waiting, not yet formed into galaxies,” said Paul Shapiro, project leader and astronomy professor at the University of Texas at Austin. “As ultraviolet photons of starlight streamed across the universe, they struck these intergalactic atoms and photoionized [meaning light served as the catalyst for ionization] them, releasing fast-moving electrons that heated the gas to 10,000 kelvin.”

By the end of the first billion years, the universe was [again] filled with ionized gas – reionized, actually, since atoms were born in an ionized state a billion years before that but had since then recombined. “We call this transformation the ‘epoch of reionization.’ It was a kind of ‘second coming’ of ionization, a giant backlash by the first galaxies on the universe that created them,” said Shapiro.

The multi-institutional team including collaborators from the universities of Texas, Strasbourg, Zurich, Sussex, Madrid, and Potsdam and its first results are will soon be published in the Monthly Notices of the Royal Astronomical Society. A fuller account of the project is posted on the OLCF web site.

To solve the equations describing the time history of reionization, researchers needed to compute “snapshots” of the moving gas, stars, dark matter, and radiation to track the expanding surfaces of the ionized zones without blurring them – no small task given the radiation that causes these surfaces to expand travels at the speed of light, much faster than the gas, dark matter, and stars.

The simulation ran on Titan, which has GPUs on all of its 18, 688 nodes. This project required using 8,192 GPUs. “Titan is special in having such a large numbers of GPUs, and that is something that made our simulations a Titan-specific milestone,” Shapiro said.

In total the simulation generated data sets of more than 2 petabytes. The team stored 138 snapshots, one for every 10 million years of cosmic time, starting at a time about 10 million years after the big bang. Each snapshot created about 16 terabytes of raw data. To analyze this particularly large, challenging set of raw data, the team subsequently was awarded additional computing hours to continue using the OLCF’s analysis clusters Eos and Rhea.

Shapiro’s team found the simulated reionization of the local universe ended a bit later than observations suggest for the universe-at-large, but the predicted properties of galaxies, intergalactic gas, and the radiation backgrounds that filled the universe at such early times agreed with those observed once an adjustment was made for late reionization.

Link to the full OLCF article on the project: https://www.olcf.ornl.gov/2016/03/08/illuminating-the-universes-ignition/