Facebook Parent Meta’s New AI Supercomputer Will Be ‘World’s Fastest’

By Oliver Peckham

January 24, 2022

Fresh off its rebrand last October, Meta (née Facebook) is putting muscle behind its vision of a metaversal future with a massive new AI supercomputer called the AI Research SuperCluster (RSC). Meta says that RSC will be used to help build new AI models, develop augmented reality tools, seamlessly analyze multimedia data and more. The supercomputer’s first phase is already operational, and it is scheduled for full build-out by mid-year. HPCwire is estimating that the final system will weigh in at over 220 Linpack petaflops.

RSC as currently built. Image courtesy of Meta.

About the system

RSC’s first phase, already built-out and operational, consists of 760 Nvidia DGX A100 compute nodes, totaling some 6,080 Nvidia A100 GPUs, all networked with Nvidia’s Quantum 200Gb/s InfiniBand. For storage, the system is equipped with 175PB of Pure Storage FlashArray, 10PB of Pure Storage FlashBlade and 46PB of cache storage housed in Penguin Computing Altus servers. Meta says that, with just this first phase, they “believe [RSC] is among the fastest AI supercomputers running today[.]”

With the completion of the second phase around July, Meta says, RSC will contain a total of 16,000 GPUs (presumably through an additional 1,240 DGX A100 nodes, which Nvidia believes will make it the largest customer installation of DGX A100 systems) and a full exabyte of storage with the capacity to accommodate 16TB/s of training data. Meta indicated 16,000 GPUs will be the maximum configuration of the system. “This is due to the network configuration to reduce the number of hops, to ensure we provide a 1:1 oversubscription,” a Meta spokesperson told us.

Meta says this second phase will increase RSC’s AI training performance by more than 2.5× (tracking with the 2.63× increase in GPUs), cementing it as the single fastest AI supercomputer in the world.

Unlike preceding systems, RSC is intended for use with not just open-source/public datasets, but with real-world, internal production data from Meta. To that end, Meta says, they designed the system to be isolated from the internet, with all connections passing through Meta’s own datacenters. User-generated data—checked for anonymization—is encrypted from the storage systems to the GPUs and only decrypted in-memory immediately prior to its use in model training.

Meta also developed a storage service (called AI Research Store, or AIRStore) to handle the growing bandwidth and capacity requirements of RSC. AIRStore preprocesses training data for AI models and is designed to optimize transfer speeds.

In its announcement of RSC, Meta also quietly detailed the first generation of its AI research supercomputing hardware, launched in 2017. The unnamed cluster, Meta says, has 22,000 Nvidia V100 GPUs and performs 35,000 training jobs per day. Meta says that compared to this previous system, RSC’s early benchmarks show a 20× improvement on computer vision workflows and a 3× improvement in large-scale NLP model training (which, Meta says, translates to weeks of saved time).

So far, Meta has worked with a consistent roster of partners across these systems: Penguin Computing for architecture and managed services; Nvidia for systems, GPUs, networking, and software stack components; and Pure Storage for most of the storage functionality.

Image courtesy of Meta.

The fastest AI supercomputer(s)

In terms of flops, Meta estimates that RSC will deliver nearly five exaflops of mixed-precision AI compute power. Using Nvidia’s Selene supercomputer (also comprised of eight-GPU Nvidia DGX A100 nodes) as a benchmark, HPCwire estimates that (were Meta to run the HPL benchmark) the full iteration of RSC might deliver around 227 Linpack petaflops of compute power (up from perhaps 86 petaflops right now), though further optimizations made by Nvidia in the interim may make those numbers underestimates.

That is certainly a powerful system—the first phase of RSC would likely place fourth on November’s Top500 list, and its full form would likely place second—but the race for “fastest AI supercomputer” is crowded. While RSC will almost certainly best current comparable competitors like Selene (63.4 Linpack petaflops) and the similarly A100-based Perlmutter system at NERSC (70.9 Linpack petaflops), the near future boasts much stronger challengers.

The most like-to-like comparison might be EuroHPC’s forthcoming Leonardo system, a pre-exascale Atos-built supercomputer that will also be powered by Nvidia A100s (around 14,000 of them, compared to RSC’s planned 16,000). CINECA, which is slated to launch Leonardo’s GPU-powered booster module this month, expects that module alone to deliver 240.5 Linpack petaflops, and Nvidia has billed the forthcoming system as—you guessed it—the “world’s fastest AI supercomputer” (with an estimated ten exaflops of FP16 AI performance).

Tesla, too, is publicly building an enormous AI supercomputer called Dojo, targeting that system at model training for autonomous vehicle development. Currently, it has an A100-based precursor system that HPCwire previously estimated at around 82 Linpack petaflops, but Dojo itself will be powered by Tesla’s proprietary “D1” chip. Owing to the nontraditional hardware and other uncertainties, it is harder to estimate Dojo’s future Linpack performance, but Tesla says that when Dojo launches (as-yet unspecified) it will be “the fastest AI training computer.”

Two notes: first, HPCwire also estimates that RSC’s V100-based precursor system likely delivers around 135 Linpack petaflops and would probably place third on the current Top500, well above the competition from AI systems like Selene and Perlmutter. This would—at least in terms of the Top500—make it the world’s fastest AI supercomputer. Second: Meta (under the name Facebook) previously submitted a 3.3-Linpack petaflops system to the Top500 in early 2017 (it currently ranks 139th). While that system uses Penguin servers, the specs mention Nvidia Tesla P100s and Quadro GP100s rather than V100s, so it may not be part of the precursor system.

Only time (and benchmarks) will tell who comes out on top.

Image courtesy of Meta.

Into the metaverse

The first phase of RSC is already being used for applications like large-model training for natural language processing (NLP) and computer vision. But the long-term target is the metaverse, the nebulously defined virtual world that Meta (named for the metaverse) clearly believes will constitute a new digital revolution.

Meta has an ambitious vision of RSC for the metaverse, highlighting, as an example, how RSC could train models for real-time voice translation among large groups of people, enabling individuals speaking different languages to collaborate on work or gameplay without a language barrier.

“The experiences we’re building for the metaverse require enormous compute power (quintillions of operations/second!) and RSC will enable new AI models that can learn from trillions of examples, understand hundreds of languages, and more,” said Mark Zuckerberg, CEO of Meta.

Building RSC during a pandemic

Meta ties the ideas behind RSC all the way back to the founding of the Facebook AI Research lab in 2013, but says the real inception of the project dated back to early 2020, when they decided a new system was necessary to take advantage of advances in GPU and network fabric technologies. The headline goal: a system capable of training models with more than a trillion parameters on datasets as large as an exabyte.

Rack delivery for RSC. Image courtesy of Meta.

Covid, of course, impeded the development of such a system. Meta says that RSC started as a completely remote project, and the supply chain challenges that emerged later in the pandemic threw even more roadblocks into the path. Meta explained that supply chain disruptions made it difficult to obtain components from chips to GPUs.

“One does not simply buy and power on a supercomputer,” said George Niznik, sourcing manager for Meta. “RSC was designed and executed under extremely compressed timelines without the benefit of a traditional product release cycle. Additionally, the pandemic and a major industry chip supply shortage hit at precisely the wrong moment in time. We had to fully utilize of all our collective skills and experiences to solve these difficult constraints.”

Nevertheless, a year and a half later, the team had delivered a functioning cluster. Meta told HPCwire that the team had been able to mitigate supply chain issues for phase one and that the phased build is continuing according to plan.

“I think what I’m most proud of is doing this with the team completely remotely,” said Shubho Sengupta, an AI researcher at Meta. “I mean, it is insane that you can do this without ever meeting anybody.”

An image from Meta’s video announcing the system, perhaps showing the otherwise-undisclosed location of RSC. (We think the location shown is outside of Richmond, Virginia.)
Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industry updates delivered to you every week!

ARM, Fujitsu Targeting Open-source Software for Power Efficiency in 2-nm Chip

July 19, 2024

Fujitsu and ARM are relying on open-source software to bring power efficiency to an air-cooled supercomputing chip that will ship in 2027. Monaka chip, which will be made using the 2-nanometer process, is based on the Read more…

SCALEing the CUDA Castle

July 18, 2024

In a previous article, HPCwire has reported on a way in which AMD can get across the CUDA moat that protects the Nvidia CUDA castle (at least for PyTorch AI projects.). Other tools have joined the CUDA castle siege. AMD Read more…

Quantum Watchers – Terrific Interview with Caltech’s John Preskill by CERN

July 17, 2024

In case you missed it, there's a fascinating interview with John Preskill, the prominent Caltech physicist and pioneering quantum computing researcher that was recently posted by CERN’s department of experimental physi Read more…

Aurora AI-Driven Atmosphere Model is 5,000x Faster Than Traditional Systems

July 16, 2024

While the onset of human-driven climate change brings with it many horrors, the increase in the frequency and strength of storms poses an enormous threat to communities across the globe. As climate change is warming ocea Read more…

Researchers Say Memory Bandwidth and NVLink Speeds in Hopper Not So Simple

July 15, 2024

Researchers measured the real-world bandwidth of Nvidia's Grace Hopper superchip, with the chip-to-chip interconnect results falling well short of theoretical claims. A paper published on July 10 by researchers in the U. Read more…

Belt-Tightening in Store for Most Federal FY25 Science Budets

July 15, 2024

If it’s summer, it’s federal budgeting time, not to mention an election year as well. There’s an excellent summary of the curent state of FY25 efforts reported in AIP’s policy FYI: Science Policy News. Belt-tight Read more…

SCALEing the CUDA Castle

July 18, 2024

In a previous article, HPCwire has reported on a way in which AMD can get across the CUDA moat that protects the Nvidia CUDA castle (at least for PyTorch AI pro Read more…

Aurora AI-Driven Atmosphere Model is 5,000x Faster Than Traditional Systems

July 16, 2024

While the onset of human-driven climate change brings with it many horrors, the increase in the frequency and strength of storms poses an enormous threat to com Read more…

Shutterstock 1886124835

Researchers Say Memory Bandwidth and NVLink Speeds in Hopper Not So Simple

July 15, 2024

Researchers measured the real-world bandwidth of Nvidia's Grace Hopper superchip, with the chip-to-chip interconnect results falling well short of theoretical c Read more…

Shutterstock 2203611339

NSF Issues Next Solicitation and More Detail on National Quantum Virtual Laboratory

July 10, 2024

After percolating for roughly a year, NSF has issued the next solicitation for the National Quantum Virtual Lab program — this one focused on design and imple Read more…

NCSA’s SEAS Team Keeps APACE of AlphaFold2

July 9, 2024

High-performance computing (HPC) can often be challenging for researchers to use because it requires expertise in working with large datasets, scaling the softw Read more…

Anders Jensen on Europe’s Plan for AI-optimized Supercomputers, Welcoming the UK, and More

July 8, 2024

The recent ISC24 conference in Hamburg showcased LUMI and other leadership-class supercomputers co-funded by the EuroHPC Joint Undertaking (JU), including three Read more…

Generative AI to Account for 1.5% of World’s Power Consumption by 2029

July 8, 2024

Generative AI will take on a larger chunk of the world's power consumption to keep up with the hefty hardware requirements to run applications. "AI chips repres Read more…

US Senators Propose $32 Billion in Annual AI Spending, but Critics Remain Unconvinced

July 5, 2024

Senate leader, Chuck Schumer, and three colleagues want the US government to spend at least $32 billion annually by 2026 for non-defense related AI systems.  T Read more…

Atos Outlines Plans to Get Acquired, and a Path Forward

May 21, 2024

Atos – via its subsidiary Eviden – is the second major supercomputer maker outside of HPE, while others have largely dropped out. The lack of integrators and Atos' financial turmoil have the HPC market worried. If Atos goes under, HPE will be the only major option for building large-scale systems. Read more…

Everyone Except Nvidia Forms Ultra Accelerator Link (UALink) Consortium

May 30, 2024

Consider the GPU. An island of SIMD greatness that makes light work of matrix math. Originally designed to rapidly paint dots on a computer monitor, it was then Read more…

Comparing NVIDIA A100 and NVIDIA L40S: Which GPU is Ideal for AI and Graphics-Intensive Workloads?

October 30, 2023

With long lead times for the NVIDIA H100 and A100 GPUs, many organizations are looking at the new NVIDIA L40S GPU, which it’s a new GPU optimized for AI and g Read more…

Shutterstock_1687123447

Nvidia Economics: Make $5-$7 for Every $1 Spent on GPUs

June 30, 2024

Nvidia is saying that companies could make $5 to $7 for every $1 invested in GPUs over a four-year period. Customers are investing billions in new Nvidia hardwa Read more…

Nvidia Shipped 3.76 Million Data-center GPUs in 2023, According to Study

June 10, 2024

Nvidia had an explosive 2023 in data-center GPU shipments, which totaled roughly 3.76 million units, according to a study conducted by semiconductor analyst fir Read more…

AMD Clears Up Messy GPU Roadmap, Upgrades Chips Annually

June 3, 2024

In the world of AI, there's a desperate search for an alternative to Nvidia's GPUs, and AMD is stepping up to the plate. AMD detailed its updated GPU roadmap, w Read more…

Some Reasons Why Aurora Didn’t Take First Place in the Top500 List

May 15, 2024

The makers of the Aurora supercomputer, which is housed at the Argonne National Laboratory, gave some reasons why the system didn't make the top spot on the Top Read more…

Intel’s Next-gen Falcon Shores Coming Out in Late 2025 

April 30, 2024

It's a long wait for customers hanging on for Intel's next-generation GPU, Falcon Shores, which will be released in late 2025.  "Then we have a rich, a very Read more…

Leading Solution Providers

Contributors

Google Announces Sixth-generation AI Chip, a TPU Called Trillium

May 17, 2024

On Tuesday May 14th, Google announced its sixth-generation TPU (tensor processing unit) called Trillium.  The chip, essentially a TPU v6, is the company's l Read more…

Nvidia H100: Are 550,000 GPUs Enough for This Year?

August 17, 2023

The GPU Squeeze continues to place a premium on Nvidia H100 GPUs. In a recent Financial Times article, Nvidia reports that it expects to ship 550,000 of its lat Read more…

IonQ Plots Path to Commercial (Quantum) Advantage

July 2, 2024

IonQ, the trapped ion quantum computing specialist, delivered a progress report last week firming up 2024/25 product goals and reviewing its technology roadmap. Read more…

Choosing the Right GPU for LLM Inference and Training

December 11, 2023

Accelerating the training and inference processes of deep learning models is crucial for unleashing their true potential and NVIDIA GPUs have emerged as a game- Read more…

The NASA Black Hole Plunge

May 7, 2024

We have all thought about it. No one has done it, but now, thanks to HPC, we see what it looks like. Hold on to your feet because NASA has released videos of wh Read more…

Nvidia’s New Blackwell GPU Can Train AI Models with Trillions of Parameters

March 18, 2024

Nvidia's latest and fastest GPU, codenamed Blackwell, is here and will underpin the company's AI plans this year. The chip offers performance improvements from Read more…

Q&A with Nvidia’s Chief of DGX Systems on the DGX-GB200 Rack-scale System

March 27, 2024

Pictures of Nvidia's new flagship mega-server, the DGX GB200, on the GTC show floor got favorable reactions on social media for the sheer amount of computing po Read more…

MLPerf Inference 4.0 Results Showcase GenAI; Nvidia Still Dominates

March 28, 2024

There were no startling surprises in the latest MLPerf Inference benchmark (4.0) results released yesterday. Two new workloads — Llama 2 and Stable Diffusion Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire