Livermore’s El Capitan Supercomputer to Debut HPE ‘Rabbit’ Near Node Local Storage

By Tiffany Trader

February 18, 2021

A near node local storage innovation called Rabbit factored heavily into Lawrence Livermore National Laboratory’s decision to select Cray’s proposal for its CORAL-2 machine, the lab’s first exascale-class supercomputer, El Capitan. Details of this new storage technology were revealed by Livermore Computing CTO Bronis de Supinksi at the Riken-CCS International Symposium, which took place earlier this week (Jan. 15).

A 2 exaflops supercomputer slated for delivery at Livermore in late 2022 or early 2023, El Capitan is a partnership between the Department of Energy lab and HPE (which acquired Cray in 2019). As has been previously disclosed, El Capitan will be powered by AMD CPUs and GPUs as part of the Cray EX architecture (formerly known as Shasta) with Slingshot networking.

Under a non recurring engineering (NRE) contract funded by Livermore, HPE is developing near node local storage technology that it calls its Rabbit program. De Supinski explains that NRE contracts allow procurers, such as Livermore, to improve elements of the overall system, and those innovations then flow to the broader HPC market through the partnering vendor’s portfolio. 

At its core, Rabbit is a 4U solution for node local storage that encompasses 18 SSDs (16 and 2 spares) and one (AMD Epyc) storage processor. HPE refers to Rabbit as near node local storage that combined with its custom Rabbit software supports a wide range of use cases, including resolving network bursts, optimizing input, and even running analysis processes.

De Supinski described the arrangement: “You have PCI connections from each of the compute nodes into what HPE calls the Rabbit-S boards, and that gives you a PCIe network that connects you to a fairly large number of SSDs and also to a Rabbit-P board, which has a separate AMD processor located on it that allows you to interact with the storage independent of the compute blades.”

Here’s what HPE expects the Rabbit design to look like:

“If we think about how Rabbit works, this is really like building a little nest of PCIe networks within the larger system,” said de Supinski.

The 4U Rabbit box plugs into the back of the Cray XE racks, where the switch modules are housed. “Instead of populating all the switch blades, which would have given us far more interconnect bandwidth than we feel our workloads need, we’re going to populate some of those slots with Rabbit modules,” said de Supinski.

The HPE Rabbit software is of course where much of the magic happens and allows the SSDs to be accessed both as direct attached storage and network attached storage. “You can actually access the SSDs directly from the compute blade, or you can treat it as just a block storage device allocated to a specific compute blade,” said de Supinski.

Rabbit can be implemented as a “transient Lustre file system” across the compute blades. The software facilitates moving files from these block devices — these transient Lustre file systems — to the capability tier of the overall IO subsystem of El Capitan, said De Supinski. “It will automate movements between these different tiers, but they are effectively very different in how we intend to use them.”

Another important aspect of the Rabbit software is it allows customer code to be run in containers on the Rabbit processors. This ties to Livermore’s choice of resource manager for El Capitan, which is Flux, a departure from Slurm. Designed by Livermore, Flux has awareness of the local compute node relationship to the SSDs, which is not supported in Slurm, according to de Supinski.

De Supinski reviewed some of the different use cases that Rabbit will facilitate at Livermore that make sense for the lab’s IO patterns. “There’s productive output, which is really the reason we run the simulations,” he shared. “And there’s checkpoints [typically a defensive output]. Those checkpoints can be one per MPI process, or at least one per all of the processes running on a given compute node. Or they can be shared across multiple processes and frequently across all of the processes in a given job. And so that’s this N-to-N versus N-to-M checkpoint.

“What we actually see in our jobs is frequently a combination of these where they maybe have a shared file that’s accessed by all of the processes in the job and then they write the bulk of their data to a file per compute process. And then you can have things that are semi-defensive or are semi-productive. In fact, it turns out that usually our application sciences want to use a checkpoint for subsequent data analysis.”

Using Rabbit to run analysis processes is an attractive use case for Livermore, and both post facto and in transit data analysis can be done in containers running on the Rabbit processors.

Like other leading HPC centers, Livermore has experienced compute capability outpacing the ability to deploy large file systems. “The cost would make it so that basically we can’t save all those checkpoints,” said de Supinski. “We tend to do much less input than output. The input can be executables, things like the applications themselves, but also operating system files and other system software files. We have the simulation input, which may just be something as simple as the parameters of a job, or you can restart data, i.e. the checkpoints that were previously output.”

The lab anticipates using Rabbit for caching OS files in order to reduce boot time, and notes that Rabbit will be an efficient input mechanism for machine learning model training.

When Rabbit is used as network attached storage, it’s as if the NVRAM servers were directly attached to the interconnect. While that does a good job of alleviating the swamping of the storage area network, it doesn’t alleviate the problem of swamping the high performance interconnect, said de Supinski. To address this, Livermore is developing a file system called UnifyFS. “We will use the Rabbit processors to run that, use the PCIe connections to access those disks, but with a file system that gives us shared file access.”  

Livermore is planning to put one Rabbit module in every El Capitan compute chassis (each chassis houses 8 blades / 16 compute nodes). “We expect that using these Rabbit models is going to significantly reduce system interference from IO on the overall system,” said de Supinski.

Checkpointing is the lab’s top priority, but other aspects of IO are becoming more important. De Supinski believes the Rabbit modules will support these other storage workloads very well. 

“When we were evaluating the responses we got to the CORAL-2 RFP at Livermore, we found the Rabbit solution to be one of the key innovations that HPE was offering,” said de Supinski. “It was a significant factor in our choice of the Cray response for El Capitan.”

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

African Supercomputing Center Inaugurates ‘Toubkal,’ Most Powerful Supercomputer on the Continent

February 25, 2021

Historically, Africa hasn’t exactly been synonymous with supercomputing. There are only a handful of supercomputers on the continent, with few ranking on the global stage. Now, the Mohammed VI Polytechnic University (U Read more…

By Oliver Peckham

Supercomputer-Powered Machine Learning Supports Fusion Energy Reactor Design

February 25, 2021

Energy researchers have been reaching for the stars for decades in their attempt to artificially recreate a stable fusion energy reactor. If successful, such a reactor would revolutionize the world’s energy supply over Read more…

By Oliver Peckham

Japan to Debut Integrated Fujitsu HPC/AI Supercomputer This Spring

February 25, 2021

The integrated Fujitsu HPC/AI Supercomputer, Wisteria, is coming to Japan this spring. The University of Tokyo is preparing to deploy a heterogeneous computing system, called "Wisteria/BDEC-01," that will tackle simulati Read more…

By Tiffany Trader

President Biden Signs Executive Order to Review Chip, Other Supply Chains

February 24, 2021

U.S. President Biden signed an executive order late today calling for a 100-day review of key supply chains including semiconductors, large capacity batteries, pharmaceuticals, and rare-earth elements. The scarcity of ch Read more…

By John Russell

Xilinx Launches Alveo SN1000 SmartNIC

February 24, 2021

FPGA vendor Xilinx has debuted its latest SmartNIC model, the Alveo SN1000, with integrated “composability” features that allow enterprise users to add their own custom networking functions to supplement its built-in networking. By providing deep flexibility... Read more…

By Todd R. Weiss

AWS Solution Channel

Introducing AWS HPC Tech Shorts

Amazon Web Services (AWS) is excited to announce a new videos series focused on running HPC workloads on AWS. This new video series will cover HPC workloads from genomics, computational chemistry, to computational fluid dynamics (CFD) and more. Read more…

ASF Keynotes Showcase How HPC and Big Data Have Pervaded the Pandemic

February 24, 2021

Last Thursday, a range of experts joined the Advanced Scale Forum (ASF) in a rapid-fire roundtable to discuss how advanced technologies have transformed the way humanity responded to the COVID-19 pandemic in indelible ways. The roundtable, held near the one-year mark of the first... Read more…

By Oliver Peckham

Japan to Debut Integrated Fujitsu HPC/AI Supercomputer This Spring

February 25, 2021

The integrated Fujitsu HPC/AI Supercomputer, Wisteria, is coming to Japan this spring. The University of Tokyo is preparing to deploy a heterogeneous computing Read more…

By Tiffany Trader

Xilinx Launches Alveo SN1000 SmartNIC

February 24, 2021

FPGA vendor Xilinx has debuted its latest SmartNIC model, the Alveo SN1000, with integrated “composability” features that allow enterprise users to add their own custom networking functions to supplement its built-in networking. By providing deep flexibility... Read more…

By Todd R. Weiss

ASF Keynotes Showcase How HPC and Big Data Have Pervaded the Pandemic

February 24, 2021

Last Thursday, a range of experts joined the Advanced Scale Forum (ASF) in a rapid-fire roundtable to discuss how advanced technologies have transformed the way humanity responded to the COVID-19 pandemic in indelible ways. The roundtable, held near the one-year mark of the first... Read more…

By Oliver Peckham

IBM’s Prototype Low-Power 7nm AI Chip Offers ‘Precision Scaling’

February 23, 2021

IBM has released details of a prototype AI chip geared toward low-precision training and inference across different AI model types while retaining model quality within AI applications. In a paper delivered during this year’s International Solid-State Circuits Virtual Conference, IBM... Read more…

By George Leopold

IBM Continues Mainstreaming Power Systems and Integrating Red Hat in Pivot to Cloud

February 23, 2021

As IBM continues its massive pivot to the cloud, its Power-microprocessor-based products are being mainstreamed and realigned with the corporate-wide strategy. Read more…

By John Russell

Livermore’s El Capitan Supercomputer to Debut HPE ‘Rabbit’ Near Node Local Storage

February 18, 2021

A near node local storage innovation called Rabbit factored heavily into Lawrence Livermore National Laboratory’s decision to select Cray’s proposal for its CORAL-2 machine, the lab’s first exascale-class supercomputer, El Capitan. Details of this new storage technology were revealed... Read more…

By Tiffany Trader

ENIAC at 75: Celebrating the World’s First Supercomputer

February 15, 2021

With little fanfare, today’s computer revolution was arguably born and announced through a small, innocuous, two-column story at the bottom of the front page of The New York Times on Feb. 15, 1946. In that story and others, the previously classified project, ENIAC... Read more…

By Todd R. Weiss

Microsoft, HPE Bringing AI, Edge, Cloud to Earth Orbit in Preparation for Mars Missions

February 12, 2021

The International Space Station will soon get a delivery of powerful AI, edge and cloud computing tools from HPE and Microsoft Azure to expand technology experi Read more…

By Todd R. Weiss

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from Julialang.org. I Read more…

By John Russell

Esperanto Unveils ML Chip with Nearly 1,100 RISC-V Cores

December 8, 2020

At the RISC-V Summit today, Art Swift, CEO of Esperanto Technologies, announced a new, RISC-V based chip aimed at machine learning and containing nearly 1,100 low-power cores based on the open-source RISC-V architecture. Esperanto Technologies, headquartered in... Read more…

By Oliver Peckham

Azure Scaled to Record 86,400 Cores for Molecular Dynamics

November 20, 2020

A new record for HPC scaling on the public cloud has been achieved on Microsoft Azure. Led by Dr. Jer-Ming Chia, the cloud provider partnered with the Beckman I Read more…

By Oliver Peckham

NICS Unleashes ‘Kraken’ Supercomputer

April 4, 2008

A Cray XT4 supercomputer, dubbed Kraken, is scheduled to come online in mid-summer at the National Institute for Computational Sciences (NICS). The soon-to-be petascale system, and the resulting NICS organization, are the result of an NSF Track II award of $65 million to the University of Tennessee and its partners to provide next-generation supercomputing for the nation's science community. Read more…

Programming the Soon-to-Be World’s Fastest Supercomputer, Frontier

January 5, 2021

What’s it like designing an app for the world’s fastest supercomputer, set to come online in the United States in 2021? The University of Delaware’s Sunita Chandrasekaran is leading an elite international team in just that task. Chandrasekaran, assistant professor of computer and information sciences, recently was named... Read more…

By Tracey Bryant

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

By Doug Black

Top500: Fugaku Keeps Crown, Nvidia’s Selene Climbs to #5

November 16, 2020

With the publication of the 56th Top500 list today from SC20's virtual proceedings, Japan's Fugaku supercomputer – now fully deployed – notches another win, Read more…

By Tiffany Trader

Gordon Bell Special Prize Goes to Massive SARS-CoV-2 Simulations

November 19, 2020

2020 has proven a harrowing year – but it has produced remarkable heroes. To that end, this year, the Association for Computing Machinery (ACM) introduced the Read more…

By Oliver Peckham

Leading Solution Providers

Contributors

Texas A&M Announces Flagship ‘Grace’ Supercomputer

November 9, 2020

Texas A&M University has announced its next flagship system: Grace. The new supercomputer, named for legendary programming pioneer Grace Hopper, is replacing the Ada system (itself named for mathematician Ada Lovelace) as the primary workhorse for Texas A&M’s High Performance Research Computing (HPRC). Read more…

By Oliver Peckham

At Oak Ridge, ‘End of Life’ Sometimes Isn’t

October 31, 2020

Sometimes, the old dog actually does go live on a farm. HPC systems are often cursed with short lifespans, as they are continually supplanted by the latest and Read more…

By Oliver Peckham

Saudi Aramco Unveils Dammam 7, Its New Top Ten Supercomputer

January 21, 2021

By revenue, oil and gas giant Saudi Aramco is one of the largest companies in the world, and it has historically employed commensurate amounts of supercomputing Read more…

By Oliver Peckham

Intel Xe-HP GPU Deployed for Aurora Exascale Development

November 17, 2020

At SC20, Intel announced that it is making its Xe-HP high performance discrete GPUs available to early access developers. Notably, the new chips have been deplo Read more…

By Tiffany Trader

Intel Teases Ice Lake-SP, Shows Competitive Benchmarking

November 17, 2020

At SC20 this week, Intel teased its forthcoming third-generation Xeon "Ice Lake-SP" server processor, claiming competitive benchmarking results against AMD's second-generation Epyc "Rome" processor. Ice Lake-SP, Intel's first server processor with 10nm technology... Read more…

By Tiffany Trader

New Deep Learning Algorithm Solves Rubik’s Cube

July 25, 2018

Solving (and attempting to solve) Rubik’s Cube has delighted millions of puzzle lovers since 1974 when the cube was invented by Hungarian sculptor and archite Read more…

By John Russell

It’s Fugaku vs. COVID-19: How the World’s Top Supercomputer Is Shaping Our New Normal

November 9, 2020

Fugaku is currently the most powerful publicly ranked supercomputer in the world – but we weren’t supposed to have it yet. The supercomputer, situated at Japan’s Riken scientific research institute, was scheduled to come online in 2021. When the pandemic struck... Read more…

By Oliver Peckham

MIT Makes a Big Breakthrough in Nonsilicon Transistors

December 10, 2020

What if Silicon Valley moved beyond silicon? In the 80’s, Seymour Cray was asking the same question, delivering at Supercomputing 1988 a talk titled “What’s All This About Gallium Arsenide?” The supercomputing legend intended to make gallium arsenide (GaA) the material of the future... Read more…

By Oliver Peckham

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire