Verifying the Universe with Exascale Computers

By Rob Farber

July 30, 2021

The ExaSky project, one of the critical Earth and Space Science applications being solved by the US Department of Energy’s (DOE’s) Exascale Computing Project (ECP), is preparing to use the nation’s forthcoming exascale supercomputers. Exascale machines will enable the ExaSky team to verify the gravitational influences, gas dynamics, and astrophysical inputs that they use to model the universe at unprecedented fidelity, as well as address forthcoming challenge problems to predict and replicate high-accuracy sky survey data.

Explaining his work for a general audience, Salman Habib, the director of Argonne’s Computational Science Division and an Argonne Distinguished Fellow, notes, “The ExaSky team is adapting our Lagrangian-based Hardware/Hybrid Accelerated Cosmology Code (HACC) and adaptive mesh refinement cosmology codes (Nyx) to run on GPU-accelerated exascale hardware. These machines will give us the ability to incorporate more complex physics arising from diverse inputs, such as the presence of massive neutrinos, models of star and galaxy formation, and several sources of astrophysical feedback, such as active galactic nuclei, galactic winds, and supernova explosions. These will be incorporated into both codes and run on larger grids with finer resolution. The idea is that the similar physical models in both codes should provide similar results at many different scales even though the two codes utilize different computational approaches. Obtaining similar results from both simulations helps validate our understanding of the physical processes that are occurring in nature. After that, we can add new features like star formation to replicate via simulation, observed sky survey data to verify our results and make the simulation come alive.”

Habib continues, “The ExaSky effort has wide impact, as it gives scientists a computational tool to assist in the verification of gravitational evolution, gas dynamics, and the subgrid models used in the ExaSky cosmological simulations when run at a very high dynamic range. ExaSky is an important application effort for addressing forthcoming DOE challenge problems.”

A Crisis in Cosmology?

Understanding the accelerated expansion of the universe is one of the scientific questions that the ExaSky team aims to investigate.

Observations of the universe confirm that the universe is expanding and the expansion rate is increasing with time. The underlying cause of this acceleration is not understood, and cosmologists refer to it generally as “dark energy,” a convenient shorthand coined 20 years ago for encapsulating this lack of understanding.

Salman Habib, Argonne National Laboratory

Cosmic acceleration and other similar insights were enabled by several observational advances coupled with improved theory and modeling. The current model of cosmology, which includes ingredients such as dark energy and dark matter (a form of matter that interacts gravitationally in the normal way but has very weak interactions, if any, with atomic matter), provides a very good description of astronomical and cosmological observations. Small discrepancies do exist, and there is an uncertainty as to whether these discrepancies indicate new physics—which would be very exciting—or are the result of measurement artifacts because cosmological measurements are often complex and difficult to obtain and measure. One such discrepancy is the so-called “Hubble tension,” which stipulates that the current rate of the universe’s expansion, as estimated by different techniques, shows a moderate level of disagreement. Another potential problem relates to how galaxies cluster. Galaxies are not randomly distributed in the universe but follow a well-measured statistical distribution. The measured clustering can be used to predict gravitational lensing (i.e., the distortion of shapes of background objects by intervening matter), but the measured lensing signal is too low.

If these discrepancies are signposts pointing to new discoveries, then they could result in an extremely exciting series of watershed moments that advance our understanding of the universe and uncover new aspects of the fundamental physics of matter and its interactions. Potential impacts include a possible modification of general relativity at large distances and the addition of new sectors to the Standard Model of particle physics.

Examining the Fundamental Properties of Matter

Consistent with the expansion of the universe as time moves forward, the energy density of the universe must increase as we go back in time. Thus, the universe functions as a sort of particle accelerator, allowing access to higher and higher energies the deeper into space and time we can look. Habib notes that scientists use this type of information to examine fundamental properties of matter, such as the mass of neutrinos. The analysis of current cosmological observations, such as the anisotropies in the temperature of the cosmic microwave background or the distribution of galaxies at large length scales, provides an upper bound on the sum of neutrino masses.

Habib believes that scientists can also use ExaSky simulations to examine other scientific problems, such as the nature of dark matter and the nature of primordial fluctuations in the cosmic microwave background. Succinctly, tiny temperature variations or fluctuations—at the part-per-million level in this afterglow radiation left over from the Big Bang—can offer great insight into the origin, evolution, and content of the universe.[i]

Simulating Data with Strict Observational Accuracy Requirements

Tying simulation to observed data is a necessary step in validating any computer model. The ExaSky team plans to verify its simulation results against data gathered from sky-survey observations.

The ExaSky page on the ECP website provides a more detailed description of the sky survey data and the challenge problems that are being addressed by the ExaSky team. A summary of this description is provided as follows.

  • The DOE-supported surveys have strict observational requirements. Systematic errors must be characterized and controlled, where possible, to the percent level or better. Sky surveys—such as the Cosmic Microwave Background-Stage 4, the Dark Energy Spectroscopic Instrument, and the Vera C. Rubin Observatory’s Legacy Survey of Space and Time—typically have sub-percent (i.e., statistical) requirements over the range of well-observed spatial scales.
  • The ExaSky team is actively developing its code so the team can compare its models against high-fidelity survey data through a series of challenge problems. This work in progress will be carried out with a new set of subgrid models for gas cooling, UV heating, star formation, and supernova, as well as feedback from active galactic nuclei.
  • These runs will eventually comprise a small number of very large cosmological simulations, using HACC, that will simultaneously address many science problems. Setting up the science challenge problem requires carefully configuring the initial conditions for multiple simulation species and multiple simulations: building subgrid models by matching against results from high-resolution galaxy formation astrophysics codes via a nested-box simulation approach, having a medium-scale set for parameter exploration, and—based on these results—designing and implementing the final large-scale challenge problem runs on exascale platforms.

Technical and scientific details of the challenge problems can be found in the ExaSky/HACC CoPA Tutorial presented at the ECP Annual Meeting on February 6, 2020.

Using GPUs from AMD, NVIDIA, and Intel

Habib notes that their codes are performing well on all platforms in preparation for the exascale future, including GPUs from AMD, NVIDIA, and Intel.

  • HACC: According to Habib, the HACC code performs well on all platforms. The precursor to HACC was designed to run on the Roadrunner hybrid dual-core AMD Opteron and IBM PowerXCell 8i system. Roadrunner was the first supercomputer to demonstrate a sustained 1 petaflop of performance running the Linpack benchmark in 2008.[ii] The core structure of the HACC code is massively parallel and designed to run extremely well on modern GPUs and many-core processors.
  • Nyx: Designed as a message passing interface code, Nyx has been adapted to run on GPUs.

Both codes have now been ported to Intel, NVIDIA, and AMD GPUs. The Heterogeneous-Computing Interface for Portability (HIP) translation layer was used to create code for the AMD GPUs. For the Intel GPUs, Habib notes that, “We program close to the metal and are using the new Intel GPU hardware and OneAPI software. We are doing well, but it’s not a direct translation from CUDA”.

A Strict Measure of Performance

To measure performance, the team uses a very stringent figure of merit (FOM), as shown in Figure 1. The FOM is a quantitative metric of an application’s scientific work rate. As the code is optimized to run faster and/or with more complex physics, the FOM increases.

Habib observes that the HACC code can approach peak floating-point performance on a device because much of the local particle interaction computation fits in the register memory. On GPUs, register memory is the only memory subsystem that can support peak floating-point performance.[iii] The register memory is implemented as an array of processor registers inside a register file on each of the GPU’s streaming multiprocessors. If a calculation exceeds the capacity of the GPU register file, then register spilling occurs when some of the computation is offloaded to slower memory. Using slower memory can incur a significant performance penalty, which would prevent an application from realizing peak floating-point performance.[iv]

Figure 1: Figure of merit used by the ExaSky team (source: https://ecpannualmeeting.com/assets/overview/sessions/ECP-CoPA-ExaSky-HACC-Pope-2020-02-06.pdf).

Habib noted, “Assuming codes scale from Summit to an exascale platform, FOM ratios of 20 of performant codes on Summit imply a factor of roughly 100 at the exascale, which is impressive.” The current scaling on Summit is shown in Figure 2. The projected FOM is a measured value of the FOM in which “projected” assumes that the scale-up on an exascale system will be successful.

Figure 2: Scaling figure of merit on Summit (source: Salman Habib, Argonne National Laboratory)

Summary

To date, the ExaSky team reports that it has successfully incorporated gas physics and subgrid models within its codes and has added advanced software technology to analyze simulation data. The team’s next steps include adding more physics and, once ready, testing the software on next-generation hardware as the systems come online.

[i] “Fluctuations in the Cosmic Microwave Background,” Wilkinson Microwave Anisotropy Probe Homepage, NASA, updated August 20, 2014. https://wmap.gsfc.nasa.gov/universe/bb_cosmo_fluct.html.

[ii] “Highlights – June 2008,” TOP500 List, updated June 18, 2008. https://top500.org/lists/top500/2008/06/.

[iii] Rob Farber, ed., CUDA Application Design and Development (Morgan Kaufmann, 2012), https://www.sciencedirect.com/book/9780123884268/cuda-application-design-and-development.

[iv] Sparsh Mittal, “A Survey of Techniques for Architecting and Managing GPU Register File,” IEEE Transactions on Parallel and Distributed Systems 28, no. 1 (January 2017): 16, https://doi.org/10.1109/TPDS.2016.2546249.


Rob Farber is a global technology consultant and author with an extensive background in HPC and machine learning technology development that he applies at national labs and commercial organizations. 

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

The Case for an Edge-Driven Future for Supercomputing

September 24, 2021

“Exascale only becomes valuable when it’s creating and using data that we care about,” said Pete Beckman, co-director of the Northwestern-Argonne Institute of Science and Engineering (NAISE), at the most recent HPC Read more…

Three Universities Team for NSF-Funded ‘ACES’ Reconfigurable Supercomputer Prototype

September 23, 2021

As Moore’s law slows, HPC developers are increasingly looking for speed gains in specialized code and specialized hardware – but this specialization, in turn, can make testing and deploying code trickier than ever. Now, researchers from Texas A&M University, the University of Illinois at Urbana... Read more…

Qubit Stream: Monte Carlo Advance, Infosys Joins the Fray, D-Wave Meeting Plans, and More

September 23, 2021

It seems the stream of quantum computing reports never ceases. This week – IonQ and Goldman Sachs tackle Monte Carlo on quantum hardware, Cambridge Quantum pushes chemistry calculations forward, D-Wave prepares for its Read more…

Asetek Announces It Is Exiting HPC to Protect Future Profitability

September 22, 2021

Liquid cooling specialist Asetek, well-known in HPC circles for its direct-to-chip cooling technology that is inside some of the fastest supercomputers in the world, announced today that it is exiting the HPC space amid multiple supply chain issues related to the pandemic. Although pandemic supply chain... Read more…

TACC Supercomputer Delves Into Protein Interactions

September 22, 2021

Adenosine triphosphate (ATP) is a compound used to funnel energy from mitochondria to other parts of the cell, enabling energy-driven functions like muscle contractions. For ATP to flow, though, the interaction between the hexokinase-II (HKII) enzyme and the proteins found in a specific channel on the mitochondria’s outer membrane. Now, simulations conducted on supercomputers at the Texas Advanced Computing Center (TACC) have simulated... Read more…

AWS Solution Channel

Introducing AWS ParallelCluster 3

Running HPC workloads, like computational fluid dynamics (CFD), molecular dynamics, or weather forecasting typically involves a lot of moving parts. You need a hundreds or thousands of compute cores, a job scheduler for keeping them fed, a shared file system that’s tuned for throughput or IOPS (or both), loads of libraries, a fast network, and a head node to make sense of all this. Read more…

The Latest MLPerf Inference Results: Nvidia GPUs Hold Sway but Here Come CPUs and Intel

September 22, 2021

The latest round of MLPerf inference benchmark (v 1.1) results was released today and Nvidia again dominated, sweeping the top spots in the closed (apples-to-apples) datacenter and edge categories. Perhaps more interesti Read more…

The Case for an Edge-Driven Future for Supercomputing

September 24, 2021

“Exascale only becomes valuable when it’s creating and using data that we care about,” said Pete Beckman, co-director of the Northwestern-Argonne Institut Read more…

Three Universities Team for NSF-Funded ‘ACES’ Reconfigurable Supercomputer Prototype

September 23, 2021

As Moore’s law slows, HPC developers are increasingly looking for speed gains in specialized code and specialized hardware – but this specialization, in turn, can make testing and deploying code trickier than ever. Now, researchers from Texas A&M University, the University of Illinois at Urbana... Read more…

Qubit Stream: Monte Carlo Advance, Infosys Joins the Fray, D-Wave Meeting Plans, and More

September 23, 2021

It seems the stream of quantum computing reports never ceases. This week – IonQ and Goldman Sachs tackle Monte Carlo on quantum hardware, Cambridge Quantum pu Read more…

Asetek Announces It Is Exiting HPC to Protect Future Profitability

September 22, 2021

Liquid cooling specialist Asetek, well-known in HPC circles for its direct-to-chip cooling technology that is inside some of the fastest supercomputers in the world, announced today that it is exiting the HPC space amid multiple supply chain issues related to the pandemic. Although pandemic supply chain... Read more…

TACC Supercomputer Delves Into Protein Interactions

September 22, 2021

Adenosine triphosphate (ATP) is a compound used to funnel energy from mitochondria to other parts of the cell, enabling energy-driven functions like muscle contractions. For ATP to flow, though, the interaction between the hexokinase-II (HKII) enzyme and the proteins found in a specific channel on the mitochondria’s outer membrane. Now, simulations conducted on supercomputers at the Texas Advanced Computing Center (TACC) have simulated... Read more…

The Latest MLPerf Inference Results: Nvidia GPUs Hold Sway but Here Come CPUs and Intel

September 22, 2021

The latest round of MLPerf inference benchmark (v 1.1) results was released today and Nvidia again dominated, sweeping the top spots in the closed (apples-to-ap Read more…

Why HPC Storage Matters More Now Than Ever: Analyst Q&A

September 17, 2021

With soaring data volumes and insatiable computing driving nearly every facet of economic, social and scientific progress, data storage is seizing the spotlight. Hyperion Research analyst and noted storage expert Mark Nossokoff looks at key storage trends in the context of the evolving HPC (and AI) landscape... Read more…

GigaIO Gets $14.7M in Series B Funding to Expand Its Composable Fabric Technology to Customers

September 16, 2021

Just before the COVID-19 pandemic began in March 2020, GigaIO introduced its Universal Composable Fabric technology, which allows enterprises to bring together Read more…

Ahead of ‘Dojo,’ Tesla Reveals Its Massive Precursor Supercomputer

June 22, 2021

In spring 2019, Tesla made cryptic reference to a project called Dojo, a “super-powerful training computer” for video data processing. Then, in summer 2020, Tesla CEO Elon Musk tweeted: “Tesla is developing a [neural network] training computer called Dojo to process truly vast amounts of video data. It’s a beast! … A truly useful exaflop at de facto FP32.” Read more…

Enter Dojo: Tesla Reveals Design for Modular Supercomputer & D1 Chip

August 20, 2021

Two months ago, Tesla revealed a massive GPU cluster that it said was “roughly the number five supercomputer in the world,” and which was just a precursor to Tesla’s real supercomputing moonshot: the long-rumored, little-detailed Dojo system. “We’ve been scaling our neural network training compute dramatically over the last few years,” said Milan Kovac, Tesla’s director of autopilot engineering. Read more…

Esperanto, Silicon in Hand, Champions the Efficiency of Its 1,092-Core RISC-V Chip

August 27, 2021

Esperanto Technologies made waves last December when it announced ET-SoC-1, a new RISC-V-based chip aimed at machine learning that packed nearly 1,100 cores onto a package small enough to fit six times over on a single PCIe card. Now, Esperanto is back, silicon in-hand and taking aim... Read more…

CentOS Replacement Rocky Linux Is Now in GA and Under Independent Control

June 21, 2021

The Rocky Enterprise Software Foundation (RESF) is announcing the general availability of Rocky Linux, release 8.4, designed as a drop-in replacement for the soon-to-be discontinued CentOS. The GA release is launching six-and-a-half months after Red Hat deprecated its support for the widely popular, free CentOS server operating system. The Rocky Linux development effort... Read more…

Intel Completes LLVM Adoption; Will End Updates to Classic C/C++ Compilers in Future

August 10, 2021

Intel reported in a blog this week that its adoption of the open source LLVM architecture for Intel’s C/C++ compiler is complete. The transition is part of In Read more…

Hot Chips: Here Come the DPUs and IPUs from Arm, Nvidia and Intel

August 25, 2021

The emergence of data processing units (DPU) and infrastructure processing units (IPU) as potentially important pieces in cloud and datacenter architectures was Read more…

AMD-Xilinx Deal Gains UK, EU Approvals — China’s Decision Still Pending

July 1, 2021

AMD’s planned acquisition of FPGA maker Xilinx is now in the hands of Chinese regulators after needed antitrust approvals for the $35 billion deal were receiv Read more…

Google Launches TPU v4 AI Chips

May 20, 2021

Google CEO Sundar Pichai spoke for only one minute and 42 seconds about the company’s latest TPU v4 Tensor Processing Units during his keynote at the Google I Read more…

Leading Solution Providers

Contributors

HPE Wins $2B GreenLake HPC-as-a-Service Deal with NSA

September 1, 2021

In the heated, oft-contentious, government IT space, HPE has won a massive $2 billion contract to provide HPC and AI services to the United States’ National Security Agency (NSA). Following on the heels of the now-canceled $10 billion JEDI contract (reissued as JWCC) and a $10 billion... Read more…

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from Julialang.org. I Read more…

Quantum Roundup: IBM, Rigetti, Phasecraft, Oxford QC, China, and More

July 13, 2021

IBM yesterday announced a proof for a quantum ML algorithm. A week ago, it unveiled a new topology for its quantum processors. Last Friday, the Technical Univer Read more…

Intel Launches 10nm ‘Ice Lake’ Datacenter CPU with Up to 40 Cores

April 6, 2021

The wait is over. Today Intel officially launched its 10nm datacenter CPU, the third-generation Intel Xeon Scalable processor, codenamed Ice Lake. With up to 40 Read more…

Frontier to Meet 20MW Exascale Power Target Set by DARPA in 2008

July 14, 2021

After more than a decade of planning, the United States’ first exascale computer, Frontier, is set to arrive at Oak Ridge National Laboratory (ORNL) later this year. Crossing this “1,000x” horizon required overcoming four major challenges: power demand, reliability, extreme parallelism and data movement. Read more…

Intel Unveils New Node Names; Sapphire Rapids Is Now an ‘Intel 7’ CPU

July 27, 2021

What's a preeminent chip company to do when its process node technology lags the competition by (roughly) one generation, but outmoded naming conventions make it seem like it's two nodes behind? For Intel, the response was to change how it refers to its nodes with the aim of better reflecting its positioning within the leadership semiconductor manufacturing space. Intel revealed its new node nomenclature, and... Read more…

Intel Announces Sapphire Rapids with HBM, Reveals Ponte Vecchio Form Factors

June 28, 2021

From the ISC 2021 Digital event, Intel announced it will offer Sapphire Rapids with integrated HBM, detailed new Xe-HPC GPU form factors, and introduced commercial support for DAOS (distributed application object storage). Intel also announced a new Ethernet solution, aimed at smaller-scale HPC. With integrated High Bandwidth Memory (HBM), the forthcoming Intel Xeon Scalable processors... Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire