OLCF Reaches New Heights with AMD-Based 'Andes' Cluster

Dec. 11, 2020 — Staff members at the Oak Ridge Leadership Computing Facility (OLCF) have replaced the center’s Rhea data analysis cluster with a brand-new AMD-based system dubbed Andes.

For 6 faithful years, Rhea afforded OLCF users the opportunity to perform data analysis and visualizations of simulations performed on the OLCF’s former Cray XK7 Titan supercomputer. The center’s new data analysis system, deployed this week, features more nodes, more memory, a faster interconnect, and new EPYC CPUs by AMD.

“We are excited to provide a new platform that will meet the growing needs of our scientific user community,” said Clay England, high-performance computing (HPC) Linux systems task lead in the HPC Clusters Group of the Systems Section at the National Center for Computational Sciences (NCCS) that houses the OLCF.

At its inception in 2014, Rhea—an Intel system—contained only 196 nodes and 64 gigabytes (GB) of random-access memory (RAM) per node. Upgrades throughout the years brought it up to 512 nodes with 196 GB of RAM per node.

Andes packs an even bigger punch. At 704 nodes and 256 GB of RAM per node, Andes is the most powerful commodity cluster system the OLCF has ever deployed. It’s so powerful, in fact, and its chips draw so much heat, that it had to be installed with active water-cooled rear doors. Andes is also connected via a Mellanox InfiniBand interconnect with a water-cooled switch.

Specifications for the OLCF’s Rhea system compared with its new Andes cluster. Image Credit: Jason Smith, ORNL

Although Andes is new, England said users shouldn’t anticipate a huge learning curve. Most users experienced a variety of machine upgrades on Rhea that will carry over to Andes, including a file system upgrade from Lustre to IBM’s General Parallel File System.

“Rhea led users through a lot of changes over the years,” said Hong Liu, an HPC engineer in the User Assistance – Production Systems Group of the Operations Section at NCCS who wrote the Andes User Guide. “Many big changes on Rhea have already prepared users for this new system.”

One example is that Rhea moved from the Moab workload manager to the Slurm scheduler; Rhea was the first OLCF system to make this transition. The Slurm scheduler is also used on Andes, easing the transition.

Because nine high-memory NVIDIA K80 GPU nodes were recently added to Rhea, these nodes will be moved over to Andes. The transition will be accomplished in two phases to allow workflow testing on both systems.

Following a successful Andes early user test period, projects with current Rhea allocations have automatically received Andes allocations and access. To allow time to transfer workflows to Andes, Rhea will remain available for approximately 1 month.

The Andes User Guide is available here: https://docs.olcf.ornl.gov/systems/andes_user_guide.html

The OLCF is a US Department of Energy (DOE) Office of Science User Facility located at DOE’s Oak Ridge National Laboratory.

UT-Battelle LLC manages Oak Ridge National Laboratory for DOE’s Office of Science, the single largest supporter of basic research in the physical sciences in the United States. DOE’s Office of Science is working to address some of the most pressing challenges of our time. For more information, visit https://energy.gov/science.

Source: Rachel McDowell, OLCF