NERSC Targets Exascale with Perlmutter and the Exascale Science Applications Program

October 18, 2021

Oct. 18, 2021 — With an eye toward the first generation of exascale computing, in 2021 the National Energy Research Scientific Computing Center (NERSC) at Lawrence Berkeley National Laboratory formally unveiled its newest supercomputer, Perlmutter.

Perlmutter features a heterogeneous architecture that makes it among the fastest supercomputers in the world for scientific simulation, data analysis, and artificial intelligence applications. It will be heavily used in studies of the climate, environment, clean energy technologies, semiconductors, microelectronics, and quantum information science and is expected to help reveal new discoveries in cosmology, microbiology, genetics, material sciences, “and pretty much any other field you can think of,” said Dr. Saul Perlmutter, the Nobel Prize-winning astrophysicist for whom the system is named.

The Perlmutter system, an HPE Cray EX supercomputer, is being delivered to NERSC in two phases. Phase 1, deployed in June 2021, includes 1,536 GPU-accelerated nodes, each containing four NVIDIA NVlink-connected A100 Tensor Core GPUs and one 3rd Gen AMD EPYC™ processor. Phase 1 also has a 35 PB all-flash Lustre file system that will provide very high-bandwidth storage. Phase 2, set to arrive later in 2021, will add 3,072 CPU-only nodes, each with two 3rd Gen AMD EPYC™ processors and 512 GB of memory per node.

To ensure that its users can readily utilize this new technology, for the last two years NERSC has been working with key development teams to prepare codes for Perlmutter and the coming exascale systems through its NERSC Exascale Science Applications Program (NESAP). NESAP is a collaborative effort that connects NERSC partners with code teams, vendors, and library and tools developers to prepare for advanced architectures and new computing systems.

Since NESAP was first established in 2014, the program has collaborated closely with ECP, incorporating six ECP applications into the initiative. Here are updates on five of those projects, including achievements to date and next goals.

EXAALT

The Exascale Atomistic Capability for Accuracy, Length, and Time (EXAALT) project is developing an exascale-scalable molecular dynamics simulation platform that will allow users to choose a point in accuracy, length, and time-space that is most appropriate for the problem at hand, trading the cost of one over the other. The project uses classical models like Spectral Neighbor Analysis Method (SNAP), a computationally intensive interatomic potential in the Large-scale Atomic/Molecular Massively Parallel Simulator (LAMMP), a molecular dynamics code with a focus on materials modeling.

A key focus of EXAALT is to improve the performance of SNAP potential in LAMPP on newer generation CPUs and GPUs. While SNAP had a GPU implementation, its performance relative to the peak performance showed a steady decline on new CPU and GPU hardware. To address this issue the team joined NESAP and attended the Center of Excellence hackathon in January 2019. Since then, they have made several advances:

  1. TestSNAP, a new stand-alone mini-app mimicking the workload of SNAP in LAMMPS, was created for rapid prototyping of various optimization strategies. This gave several collaborators an opportunity to contribute to the application without the need for any LAMMPS-specific knowledge.
  2. A refactor of the algorithm was performed that included kernel fission. Breaking one large GPU kernel into multiple smaller kernels allowed optimizations targeted towards specific stages of the algorithm. However, this increased the memory footprint of the algorithm since intermediate results now needed to be stored in between kernels.
  3. The team implemented new algorithmic improvements to reduce the memory overheads incurred due to the refactor, allowing the ECP KPP problem size of the refactored implementation to fit inside a single GPU memory.
  4. Scratch memory, the low-latency memory available on GPUs, was used to store intermediate results, which allowed the team to exploit symmetries in the algorithm and optimize by reducing the number of global read/write operations that arose from the new algorithm.
  5. All the optimizations tested in the mini-app are now part of the upstream LAMMPS/SNAP.

“Over the last three years, our new algorithm and implementation achieved a 26x speedup compared to the initial implementation of LAMMPS/SNAP from January 2019 on NVIDIA-V100 GPU,” said Rahulkumar Gayatri, an application performance specialist at NERSC whose current projects include EXAALT.

ExaBiome

The increasing sizes and complexities of datasets make a strong case for exascale-capable metagenome assemblers. However, the underlying algorithmic motifs are not well suited for GPU architectures. This poses a challenge because the majority of next-generation supercomputers will rely primarily on GPUs for most of their computational abilities.

The Exascale Solutions for Microbiome Analysis project (ExaBiome) – a collaborative effort that includes researchers from Berkeley Lab’s NERSC and CRD divisions, Los Alamos National Laboratory, and the Joint Genome Institute – was established to help address the challenges. The project is using machine learning algorithms and MetaHipMer, a high-performance metagenome assembler based on the whole genome assembler Meraculous, to study microbial diversity.

Among the project’s achievements to date, MetaHipMer is now GPU-ready. This was a challenging task because, unlike a typical science simulation, the algorithms used in DNA assembly do not work well on GPU architectures. As a result of close collaboration between scientists and performance engineers to redesign and re-implement some portions of the pipeline, three out of five of MetaHipMer’s modules can now take advantage of GPUs.

In addition, with the help of the MetaHipMer pipeline on the Summit system at OLCF, the ExaBiome team has assembled a metagenomic dataset of 25TB, the largest microbiome dataset ever assembled, and cut the total runtime in half on Summit. This JGI user data contained freshwater lake samples from a 17-year period, offering new insights into the evolution of microbial communities over time. Preparations are under way to assemble an even larger dataset that would require using an exascale machine like Frontier to its limits.

“GPU optimizations in MetaHipMer will benefit computations on Perlmutter as well as Frontier and Aurora,” said Muaaz Awan, an application performance specialist at NERSC and a member of the ExaBiome team. “We have already started the porting efforts and hope to complete them before Frontier becomes available for production runs.”

ExaFEL

The primary objective of the Data Analytics at the Exascale for Free Electron Lasers (ExaFEL) project – part of both ECP and NESAP for Data – is to leverage exascale computing to reduce, from weeks to minutes, the time needed to analyze x-ray diffraction data generated by experimental facilities such as the Linac Coherent Light Source. The objective of this data analysis is to understand the (yet unknown) molecular structure and function of samples, such as COVID-19 viral proteins, and the molecules involved in photosynthesis.

While ExaFEL covers a range of data projects, at present the focus is on porting two code bases to GPUs: multi-tiered iterative phasing (MTIP) for fluctuation x-ray scattering, and the computational crystallography toolbox (CCTBX), an open source code designed to advance automation of macromolecular structure determination. The project team is preparing both codes to make efficient use of next-generation supercomputers – initially Perlmutter, Frontier, and Aurora – and GPUs.  Because Perlmutter is using NVIDIA GPUs, Frontier is using AMD GPUs, and Aurora is using Intel GPUs, the team is working to ensure that the code they are developing is portable. For both CCTBX and MTIP, this involves using software packages such as Kokkos that make code portability relatively easier for developers.

In addition, the collaborators are looking for ways to off-load as many computational tasks to the GPUs as possible to make continuous efficient use of the hardware by distributing the data so the GPUs always have something to do, explained Johannes Blaschke, an applications performance specialist at NERSC and the NESAP liaison for the ExaFEL project. “We are trying to get a really weak signal out of a mountain of data, so we need to find ways for all of this complex data analysis to work at scale on a supercomputer,” he said.

Subsurface Applications

“An Exascale Subsurface Simulator of Coupled Flow, Transport, Reactions and Mechanics” is an ECP Subsurface project focused on developing a new multiphysics application code to model and solve flow, transport, reactions, and mechanics in wellbore cement to assess the risk of fracture evolution on wellbore failure.

In addition to developing this multiphysics capability, the project is developing a performance portability strategy for the application code Chombo-Crunch, a high-resolution pore scale subsurface simulator based on the Chombo software libraries supporting structured grid, adaptive, finite volume methods for numerical PDEs. Chombo-Crunch, the underlying application code, is one of the codes targeted by the NESAP program and incorporates AMR. For this effort the project is developing the Proto C++ portability layer. A Proto-enabled backend, Chombo4, is being developed for adoption of Proto in Chombo-Crunch.

Multiphysics coupling for these types of problems requires high resolution on large aspect ratio domains, requiring more computational resources than are available on current supercomputers. Also, new supercomputing architectures like Perlmutter and Frontier contain hardware accelerators that require software engineering of legacy application codes for efficient execution and optimized performance. Success of the ECP Subsurface application code development project will depend on performance portability development in Proto and Chombo4 that is supported by the NESAP program.

The project’s key achievement to date is the development of the Proto middleware approach for point-wise and stencil-based operations on regular and irregular grids in Chombo4 applications. The portability has been done on NVIDIA GPUs and AMD GPUs; so far, it has been demonstrated that Chombo4 applications with the Proto backend can execute on 4,096 NVIDIA V100 GPUs on Summit.

WDMApp

The goals of the ECP Whole Device Model Application (WDMApp) project are to predict performance of the edge plasma in ITER – the world’s largest fusion experiment – using a core-edge coupled kinetic-code framework and to achieve 50+ enhancements in the scientific problem size on exascale computers compared to what was achieved on Titan. For the core plasma that releases exhaust fusion heat to the edge plasma, the team is using the core-optimized code GENE or GEM. For the edge plasma that accepts the exhaust heat from the core plasma and builds up a pressure pedestal on which the core plasma sits on, the team is using the edge-optimized code XGC.

Due to the non-equilibrium multiscale nature of the edge plasma in complex geometry, XGC’s particle time-advance and Fokker-Planck collision kernels dominate the performance of the core-edge coupled WDMApp code. These two kernels must be ported and well-optimized on new architectures and programming models. Exhaust heat from core to edge is mostly carried by plasma turbulence. Kinetic coupling of the plasma turbulence across the core-edge coupling interface has been achieved to date in collisionless plasma, which evaluates the core-to-edge heat flow seamlessly. The coupled codes fully utilizes the V100 GPUs on Summit.

The project’s next goal is to include Fokker-Planck collisions in the coupled simulation so that the collisional modification of the turbulence and heat flux can be captured. The goal in the NESAP program is to port and optimize the Fokker-Planck kernel on A100 GPUs and optimize the coupled WDMApp simulation on Perlmutter with more realistic collision effects. The performance-dominant XGC code has been converted from Fortran to C++ and made to utilize Kokkos for portable GPU off-loading. The knowledge gained on the Perlmutter architecture will be utilized in achieving WDMApp’s exascale goal.

About NERSC and Berkeley Lab

The National Energy Research Scientific Computing Center (NERSC) is a U.S. Department of Energy Office of Science User Facility that serves as the primary high-performance computing center for scientific research sponsored by the Office of Science. Located at Lawrence Berkeley National Laboratory, the NERSC Center serves more than 7,000 scientists at national laboratories and universities researching a wide range of problems in combustion, climate modeling, fusion energy, materials science, physics, chemistry, computational biology, and other disciplines. Berkeley Lab is a DOE national laboratory located in Berkeley, California. It conducts unclassified scientific research and is managed by the University of California for the U.S. Department of Energy. »Learn more about computing sciences at Berkeley Lab.


Source: NERSC

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

SC21 Was Unlike Any Other — Was That a Good Thing?

December 3, 2021

For a long time, the promised in-person SC21 seemed like an impossible fever dream, the assurances of a prominent physical component persisting across years of canceled conferences, including two virtual ISCs and the virtual SC20. With the advent of the Delta variant, Covid surges in St. Louis and contention over vaccine requirements... Read more…

The Green500’s Crystal Anniversary Sees MN-3 Crystallize Its Winning Streak

December 2, 2021

“This is the 30th Green500,” said Wu Feng, custodian of the Green500 list, at the list’s SC21 birds-of-a-feather session. “You could say 15 years of Green500, which makes it, I guess, the crystal anniversary.” Indeed, HPCwire marked the 15th anniversary of the Green500 – which ranks supercomputers by flops-per-watt, rather than just by flops – earlier this year with... Read more…

AWS Arm-based Graviton3 Instances Now in Preview

December 1, 2021

Three years after unveiling the first generation of its AWS Graviton chip-powered instances in 2018, Amazon Web Services announced that the third generation of the processors – the AWS Graviton3 – will power all-new Amazon Elastic Compute 2 (EC2) C7g instances that are now available in preview. Debuting at the AWS re:Invent 2021... Read more…

Nvidia Dominates Latest MLPerf Results but Competitors Start Speaking Up

December 1, 2021

MLCommons today released its fifth round of MLPerf training benchmark results with Nvidia GPUs again dominating. That said, a few other AI accelerator companies participated and, one of them, Graphcore, even held a separ Read more…

HPC Career Notes: December 2021 Edition

December 1, 2021

In this monthly feature, we’ll keep you up-to-date on the latest career developments for individuals in the high-performance computing community. Whether it’s a promotion, new company hire, or even an accolade, we’ Read more…

AWS Solution Channel

Running a 3.2M vCPU HPC Workload on AWS with YellowDog

Historically, advances in fields such as meteorology, healthcare, and engineering, were achieved through large investments in on-premises computing infrastructure. Upfront capital investment and operational complexity have been the accepted norm of large-scale HPC research. Read more…

At SC21, Experts Ask: Can Fast HPC Be Green?

November 30, 2021

HPC is entering a new era: exascale is (somewhat) officially here, but Moore’s law is ending. Power consumption and other sustainability concerns loom over the enormous systems and chips of this new epoch, for both cost and compliance reasons. Reconciling the need to continue the supercomputer scale-up while reducing HPC’s environmental impacts... Read more…

SC21 Was Unlike Any Other — Was That a Good Thing?

December 3, 2021

For a long time, the promised in-person SC21 seemed like an impossible fever dream, the assurances of a prominent physical component persisting across years of canceled conferences, including two virtual ISCs and the virtual SC20. With the advent of the Delta variant, Covid surges in St. Louis and contention over vaccine requirements... Read more…

The Green500’s Crystal Anniversary Sees MN-3 Crystallize Its Winning Streak

December 2, 2021

“This is the 30th Green500,” said Wu Feng, custodian of the Green500 list, at the list’s SC21 birds-of-a-feather session. “You could say 15 years of Green500, which makes it, I guess, the crystal anniversary.” Indeed, HPCwire marked the 15th anniversary of the Green500 – which ranks supercomputers by flops-per-watt, rather than just by flops – earlier this year with... Read more…

Nvidia Dominates Latest MLPerf Results but Competitors Start Speaking Up

December 1, 2021

MLCommons today released its fifth round of MLPerf training benchmark results with Nvidia GPUs again dominating. That said, a few other AI accelerator companies Read more…

At SC21, Experts Ask: Can Fast HPC Be Green?

November 30, 2021

HPC is entering a new era: exascale is (somewhat) officially here, but Moore’s law is ending. Power consumption and other sustainability concerns loom over the enormous systems and chips of this new epoch, for both cost and compliance reasons. Reconciling the need to continue the supercomputer scale-up while reducing HPC’s environmental impacts... Read more…

Raja Koduri and Satoshi Matsuoka Discuss the Future of HPC at SC21

November 29, 2021

HPCwire's Managing Editor sits down with Intel's Raja Koduri and Riken's Satoshi Matsuoka in St. Louis for an off-the-cuff conversation about their SC21 experience, what comes after exascale and why they are collaborating. Koduri, senior vice president and general manager of Intel's accelerated computing systems and graphics (AXG) group, leads the team... Read more…

Jack Dongarra on SC21, the Top500 and His Retirement Plans

November 29, 2021

HPCwire's Managing Editor sits down with Jack Dongarra, Top500 co-founder and Distinguished Professor at the University of Tennessee, during SC21 in St. Louis to discuss the 2021 Top500 list, the outlook for global exascale computing, and what exactly is going on in that Viking helmet photo. Read more…

SC21: Larry Smarr on The Rise of Supernetwork Data Intensive Computing

November 26, 2021

Larry Smarr, founding director of Calit2 (now Distinguished Professor Emeritus at the University of California San Diego) and the first director of NCSA, is one of the seminal figures in the U.S. supercomputing community. What began as a personal drive, shared by others, to spur the creation of supercomputers in the U.S. for scientific use, later expanded into a... Read more…

Three Chinese Exascale Systems Detailed at SC21: Two Operational and One Delayed

November 24, 2021

Details about two previously rumored Chinese exascale systems came to light during last week’s SC21 proceedings. Asked about these systems during the Top500 media briefing on Monday, Nov. 15, list author and co-founder Jack Dongarra indicated he was aware of some very impressive results, but withheld comment when asked directly if he had... Read more…

IonQ Is First Quantum Startup to Go Public; Will It be First to Deliver Profits?

November 3, 2021

On October 1 of this year, IonQ became the first pure-play quantum computing start-up to go public. At this writing, the stock (NYSE: IONQ) was around $15 and its market capitalization was roughly $2.89 billion. Co-founder and chief scientist Chris Monroe says it was fun to have a few of the company’s roughly 100 employees travel to New York to ring the opening bell of the New York Stock... Read more…

Enter Dojo: Tesla Reveals Design for Modular Supercomputer & D1 Chip

August 20, 2021

Two months ago, Tesla revealed a massive GPU cluster that it said was “roughly the number five supercomputer in the world,” and which was just a precursor to Tesla’s real supercomputing moonshot: the long-rumored, little-detailed Dojo system. Read more…

Esperanto, Silicon in Hand, Champions the Efficiency of Its 1,092-Core RISC-V Chip

August 27, 2021

Esperanto Technologies made waves last December when it announced ET-SoC-1, a new RISC-V-based chip aimed at machine learning that packed nearly 1,100 cores onto a package small enough to fit six times over on a single PCIe card. Now, Esperanto is back, silicon in-hand and taking aim... Read more…

US Closes in on Exascale: Frontier Installation Is Underway

September 29, 2021

At the Advanced Scientific Computing Advisory Committee (ASCAC) meeting, held by Zoom this week (Sept. 29-30), it was revealed that the Frontier supercomputer is currently being installed at Oak Ridge National Laboratory in Oak Ridge, Tenn. The staff at the Oak Ridge Leadership... Read more…

AMD Launches Milan-X CPU with 3D V-Cache and Multichip Instinct MI200 GPU

November 8, 2021

At a virtual event this morning, AMD CEO Lisa Su unveiled the company’s latest and much-anticipated server products: the new Milan-X CPU, which leverages AMD’s new 3D V-Cache technology; and its new Instinct MI200 GPU, which provides up to 220 compute units across two Infinity Fabric-connected dies, delivering an astounding 47.9 peak double-precision teraflops. “We're in a high-performance computing megacycle, driven by the growing need to deploy additional compute performance... Read more…

Intel Reorgs HPC Group, Creates Two ‘Super Compute’ Groups

October 15, 2021

Following on changes made in June that moved Intel’s HPC unit out of the Data Platform Group and into the newly created Accelerated Computing Systems and Graphics (AXG) business unit, led by Raja Koduri, Intel is making further updates to the HPC group and announcing... Read more…

Intel Completes LLVM Adoption; Will End Updates to Classic C/C++ Compilers in Future

August 10, 2021

Intel reported in a blog this week that its adoption of the open source LLVM architecture for Intel’s C/C++ compiler is complete. The transition is part of In Read more…

Killer Instinct: AMD’s Multi-Chip MI200 GPU Readies for a Major Global Debut

October 21, 2021

AMD’s next-generation supercomputer GPU is on its way – and by all appearances, it’s about to make a name for itself. The AMD Radeon Instinct MI200 GPU (a successor to the MI100) will, over the next year, begin to power three massive systems on three continents: the United States’ exascale Frontier system; the European Union’s pre-exascale LUMI system; and Australia’s petascale Setonix system. Read more…

Leading Solution Providers

Contributors

Hot Chips: Here Come the DPUs and IPUs from Arm, Nvidia and Intel

August 25, 2021

The emergence of data processing units (DPU) and infrastructure processing units (IPU) as potentially important pieces in cloud and datacenter architectures was Read more…

D-Wave Embraces Gate-Based Quantum Computing; Charts Path Forward

October 21, 2021

Earlier this month D-Wave Systems, the quantum computing pioneer that has long championed quantum annealing-based quantum computing (and sometimes taken heat fo Read more…

HPE Wins $2B GreenLake HPC-as-a-Service Deal with NSA

September 1, 2021

In the heated, oft-contentious, government IT space, HPE has won a massive $2 billion contract to provide HPC and AI services to the United States’ National Security Agency (NSA). Following on the heels of the now-canceled $10 billion JEDI contract (reissued as JWCC) and a $10 billion... Read more…

The Latest MLPerf Inference Results: Nvidia GPUs Hold Sway but Here Come CPUs and Intel

September 22, 2021

The latest round of MLPerf inference benchmark (v 1.1) results was released today and Nvidia again dominated, sweeping the top spots in the closed (apples-to-ap Read more…

Ahead of ‘Dojo,’ Tesla Reveals Its Massive Precursor Supercomputer

June 22, 2021

In spring 2019, Tesla made cryptic reference to a project called Dojo, a “super-powerful training computer” for video data processing. Then, in summer 2020, Tesla CEO Elon Musk tweeted: “Tesla is developing a [neural network] training computer... Read more…

Three Chinese Exascale Systems Detailed at SC21: Two Operational and One Delayed

November 24, 2021

Details about two previously rumored Chinese exascale systems came to light during last week’s SC21 proceedings. Asked about these systems during the Top500 media briefing on Monday, Nov. 15, list author and co-founder Jack Dongarra indicated he was aware of some very impressive results, but withheld comment when asked directly if he had... Read more…

2021 Gordon Bell Prize Goes to Exascale-Powered Quantum Supremacy Challenge

November 18, 2021

Today at the hybrid virtual/in-person SC21 conference, the organizers announced the winners of the 2021 ACM Gordon Bell Prize: a team of Chinese researchers leveraging the new exascale Sunway system to simulate quantum circuits. The Gordon Bell Prize, which comes with an award of $10,000 courtesy of HPC pioneer Gordon Bell, is awarded annually... Read more…

Quantum Computer Market Headed to $830M in 2024

September 13, 2021

What is one to make of the quantum computing market? Energized (lots of funding) but still chaotic and advancing in unpredictable ways (e.g. competing qubit tec Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire