Last February, it seemed like 2020 was going to be the year of climate supercomputing. In nearly back-to-back announcements, the UK announced plans to create the most powerful supercomputer dedicated to weather and climate and the U.S. National Oceanic and Atmospheric Administration (NOAA) announced plans to triple its operational supercomputing capacity for weather and climate – but a few weeks later, of course, everything changed. Now, nearly a year later, it’s time for climate supercomputing to shine once more: HPE has announced plans to build a new flagship supercomputer for the National Center for Atmospheric Research (NCAR).

The system, which is expected to cost in excess of $35 million and deliver 19.87 peak petaflops, will be based on HPE’s Cray EX architecture. It will consist of 2,480 CPU-only nodes (each powered by two third-generation AMD Epyc Milan CPUs) and 82 GPU nodes (each powered by one Epyc Milan CPU and four Nvidia A100 “Ampere” GPUs) alongside eight Epyc Rome-based login nodes. (NCAR says that these GPUs will constitute 20 percent of the system’s sustained computing capacity.) In total, the system boasts 692 TB of memory. This hardware – rounded out by purpose-built HPE Slingshot-enabled HPC networking and 60 PB of Cray ClusterStor E1000 storage – will be augmented by an integrated software suite using the HPE Cray Programming Environment and Nvidia-provided AI and HPC software. The system will also be set up to burst to commercial cloud computing services. 

Like NCAR itself, the system will be funded by the National Science Foundation. The system will support NCAR’s work to provide the earth science community with state-of-the-art computing and data resources. HPE and NCAR provided some examples of research that will be enabled by the new system, including improved predictions of seasonal water supply, drought risk and flooding; better wildfire risk management; enhanced forecasting of climate change hazards and impacts; renewable energy optimization; air quality analysis; and even greater understanding of the dangers posed by solar storms. NCAR expects that the new supercomputer will help them to make some of these predictions up to a decade in advance.

“This new system … is a major step forward in supercomputing power, providing the scientific community with the most cutting-edge technology to better understand the Earth system,” said Anke Kamrath, director of the Computational and Information Systems Laboratory at NCAR. “The resulting research will lead to new insights into potential threats ranging from severe weather and solar storms to climate change, helping to advance the knowledge needed for improved predictions that will strengthen society’s resilience to potential disasters.”

NCAR’s current flagship system is the SGI-provided Cheyenne supercomputer, housed in NCAR’s Wyoming Supercomputing Center. Launched just over four years ago, Cheyenne contains 4,032 Intel Xeon Broadwell-based nodes, 313 TB of memory and Mellanox EDR InfiniBand networking. Its 4.8 Linpack petaflops (5.3 peak petaflops) placed it 60th on the most recent Top500 list, but HPE and NCAR anticipate that the new system will deliver more than three and a half times the performance with nearly 20 peak petaflops. Furthermore, despite that performance increase, they expect that the system will use just 40 percent more power than its predecessor.

“With its extra storage and a 3.5-fold capability improvement over the current NCAR supercomputer, Cheyenne, the new supercomputer will provide the necessary resources for our scientists to continue expanding their research in the atmospheric and geospatial sciences,” said Irfan Elahi, project director for the new system and director of NCAR’s High-Performance Computing Division. “To provide this capability, the new supercomputer is designed for highly energy-efficient operations.”

NCAR is aiming to install the new system system in the Wyoming Supercomputing Center later this year, but Cheyenne will remain in service until sometime in 2022, at which point the new supercomputer will fully replace it. 

“Researchers at the University of Wyoming will make great use of the new system to better understand areas of fundamental and economic interest impacted by flows in the atmosphere and underground,” said Ed Synakowski, vice president of research and economic development at the University of Wyoming. “The advances in computing that are captured in this upgrade, and the potential for impactful application of its results, are tremendous. We look forward to working with NCAR and the National Science Foundation in using this increased capacity to advance the fundamental science that determines so many issues of potentially high economic and social importance.”

The incoming supercomputer is as-yet unnamed, pending a contest among Wyoming schoolchildren.

Header image: the NCAR building in Boulder, Colorado.

Last week, researchers using Cheyenne announced results from an intensive wildfire data analysis. To learn more about that research, click here.