Aurora the Survivor: Exascale Supercomputer Arrives After Eight Years of Doom

By Agam Shah

November 13, 2023

Many products were sacrificed in the eight years it took to bring the Aurora supercomputer to life. Nonetheless, anticipation for the second U.S.Exascale system reached a fever pitch over the last two years.

The supercomputer, built on Intel technology, finally crossed the finish line and made it to the Top500 list, but it has not reached the promised two exaflops of performance.

The supercomputer at Argonne National Labs placed second in the Top500 November list released this week, behind Frontier, which retained the top spot. The system delivered a peak performance of 585.34 PFlop, ranking it second on the list but still not into Exascale territory.

Aurora has more than 60,000 GPUs, making it the largest GPU installation in the world. It has over 10,000 computing nodes, over 166 racks, and over 80,000 networking nodes.

Argonne submitted HPL runs for a portion of Aurora, so the benchmarking is incomplete.

The system could ultimately pass two exaflops when all the testing and finetuning is complete, Top500 wrote in a statement.

“Aurora is currently being commissioned and will reportedly exceed Frontier with a peak performance of 2 EFlop/s when finished,” Top500 wrote.

Intel does not have a major GPU until 2025, so this may be the company’s last major Top 10 entry for a few years. Intel has canceled its next-generation Rialto Bridge GPU and has scheduled its next major GPU upgrade for release in 2025.

Meanwhile, Nvidia has three new GPUs coming in the next three years, and AMD’s Epyc CPUs and MI300A will be in a two-exaflop system called El Capitan, which is being installed at the Lawrence Livermore National Lab.

Aurora was first announced in 2015 as a 200-petaflop system and has survived eight years of configuration changes, hardware cancellations, and budget delays.

The initial Aurora system was scheduled to come online in 2018. At the time, it was due to have Intel’s now-canned Xeon Phi code-named Knights Hill, Xeon CPUs, and silicon photonics. At the time, Xeon Phi was Intel’s response to supercomputing GPUs.

Aurora’s plans changed after Intel axed the Xeon Phi chips in 2017 and replaced it with “a new platform and new microarchitecture specifically designed for exascale,” the company said.

Xeon Phi mixed vector processors with low-power CPUs, and it wasn’t a complete failure. It was in four top 10 systems, including China’s Tianhe-2A, in the November 2017 Top500 list.

But Nvidia’s GPUs broke Phi in the June 2018 list, when it took the first and third spot, in the Top500 with Summit and Sierra, which were based on IBM’s Power9 chips.

There was a lot of action when Intel discontinued Phi in 2017. In September of that year, the Advanced Scientific Computing Advisory Committee announced a change of plans that made Aurora US’s first exascale system. Intel and Cray were retained as vendors, and the server’s delivery date was moved to 2021 from 2018.

In late 2017, Intel hired graphics guru Raja Koduri away from AMD, which signaled the chipmaker’s intent to create a GPU. Intel wanted to replicate the success of Nvidia’s GPUs in supercomputers.

In March 2019, the U.S. Department of Energy announced Intel and Cray would deliver Aurora by 2021. Intel announced Ponte Vecchio on the sidelines of Supercomputing 2019 and said it would use the GPU in Aurora.

Intel Aurora Blade

But then more problems beset Intel. In July 2020, Intel delayed its move to the 7-nm process technology on which Ponte Vecchio would be made.

Ultimately, Intel had to turn to manufacturing rival TSMC for some Ponte Vecchio parts. The GPU has a chiplet design and more than 100 billion transistors and has sixteen compute tiles made on TSMC’s 5-nm process and eight tiles made on Intel’s 7-nm process.

The supercomputer’s main CPU, the 4th Gen Xeon chip code-named Sapphire Rapids, was also delayed by over a year, which delayed the installation.

Over the years, the U.S. Department of Energy has remained patient with Aurora. In a 2024 budget request, the DoE mentioned that Covid-related supply chain issues had delayed Aurora and that shortages had slowed the technical implementation.

In June 2023, Intel finally announced that it had completed the supercomputer installation. But the system still has not reached its peak of 2 exaflops, and further software finetuning will push the system speed even further.

The Aurora is being used for many A.I. and scientific computing applications, said Ogi Brkic, vice president and general manager for data center and HPC solutions, in a press briefing.

Aurora will be used to train a 1-trillion-parameter large-language model for scientific research. The supercomputer is also being used to reconstruct the mouse brain, which could take three years.

“This gives you a complexity of the problems being solved here. If you want to map the human brain, that’s not even close.” Brkic said.

The mouse brain reconstruction project called Connectome was running on 512 Aurora nodes and was showing better performance than Polaris, a top-20 supercomputer deployed recently by Argonne.

“Applications today are continuously being optimized, and they are not just functional, but they’re scalable, which is very important when trying to get to the solution fast,” Brkic said.

The A.I. capabilities on Aurora are also being used to understand the interaction between particles. Some data sets are polluted by other particles in the cosmos and noise, which is where A.I. fits in.

“Understanding these interactions requires a training algorithm that allows you to train and understand and reference these interactions most effectively,” Brkic said.

Intel also shared additional details about its next-generation GPU and A.I. chips. Next year, the company will start shipping the Gaudi 3 chip, competing with Nvidia’s GPUs.

The 2025 enterprise GPU, Falcon Shores, will mix Gaudi3 AI accelerator technology and general-purpose GPU cores. It will also have HBM memory and standard Ethernet switching and will support a wide range of large-language models, Brkic said.

Brkic also talked about the next Xeon chip, Emerald Rapids, which will be officially announced on December 14 and is considered an incremental upgrade to Sapphire Rapids. The successor to Emerald Rapids, called Granite Rapids, has many excited. It can do both A.I. and conventional and has breakthrough memory and bandwidth technology.

Intel is also trying to get Nvidia customers off the proprietary CUDA parallel programming framework and onto its OneAPI through tools such as SYCLomatic, which can strip proprietary code so A.I. models can work on industry-standard hardware.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industry updates delivered to you every week!

Natcast/NSTC Issues Roadmap to Implement CHIPS and Science Act

May 29, 2024

Yesterday, CHIPS for America and Natcast, the operator of the National Semiconductor Technology Center (NSTC), released a roadmap of early steps for implementing portions of the ambitious $5 billion program. Natcast is t Read more…

Scientists Use GenAI to Uncover New Insights in Materials Science

May 29, 2024

With the help of generative AI, researchers from MIT and the University of Basel in Switzerland have developed a new machine-learning framework that can help uncover new insights about materials science. The findings of Read more…

Microsoft’s ARM-based CPU Cobalt will Support Windows 11 in the Cloud

May 29, 2024

Microsoft's ARM-based CPU, called Cobalt, is now available in the cloud for public consumption. Cobalt is Microsoft's first homegrown CPU, which was first announced six months ago. The cloud-based Cobalt VMs will support Read more…

2024 Winter Classic Finale! Gala Awards Ceremony

May 28, 2024

We wrapped up the competition with our traditional Gala Awards Ceremony. This was an exciting show, given that only 40 points or so separated first place from fifth place after the Google GROMACS Challenge and heading in Read more…

IBM Makes a Push Towards Open-Source Services, Announces New watsonx Updates

May 28, 2024

Today, IBM declared that it is releasing a number of noteworthy changes to its watsonx platform, with the goal of increasing the openness, affordability, and flexibility of the platform’s AI capabilities. Announced Read more…

ISC 2024 Takeaways: Love for Top500, Extending HPC Systems, and Media Bashing

May 23, 2024

The ISC High Performance show is typically about time-to-science, but breakout sessions also focused on Europe's tech sovereignty, server infrastructure, storage, throughput, and new computing technologies. This round Read more…

Scientists Use GenAI to Uncover New Insights in Materials Science

May 29, 2024

With the help of generative AI, researchers from MIT and the University of Basel in Switzerland have developed a new machine-learning framework that can help un Read more…

watsonx

IBM Makes a Push Towards Open-Source Services, Announces New watsonx Updates

May 28, 2024

Today, IBM declared that it is releasing a number of noteworthy changes to its watsonx platform, with the goal of increasing the openness, affordability, and fl Read more…

ISC 2024 Takeaways: Love for Top500, Extending HPC Systems, and Media Bashing

May 23, 2024

The ISC High Performance show is typically about time-to-science, but breakout sessions also focused on Europe's tech sovereignty, server infrastructure, storag Read more…

ISC 2024 — A Few Quantum Gems and Slides from a Packed QC Agenda

May 22, 2024

If you were looking for quantum computing content, ISC 2024 was a good place to be last week — there were around 20 quantum computing related sessions. QC eve Read more…

Atos Outlines Plans to Get Acquired, and a Path Forward

May 21, 2024

Atos – via its subsidiary Eviden – is the second major supercomputer maker outside of HPE, while others have largely dropped out. The lack of integrators and Atos' financial turmoil have the HPC market worried. If Atos goes under, HPE will be the only major option for building large-scale systems. Read more…

Google Announces Sixth-generation AI Chip, a TPU Called Trillium

May 17, 2024

On Tuesday May 14th, Google announced its sixth-generation TPU (tensor processing unit) called Trillium.  The chip, essentially a TPU v6, is the company's l Read more…

Europe’s Race towards Quantum-HPC Integration and Quantum Advantage

May 16, 2024

What an interesting panel, Quantum Advantage — Where are We and What is Needed? While the panelists looked slightly weary — their’s was, after all, one of Read more…

The Future of AI in Science

May 15, 2024

AI is one of the most transformative and valuable scientific tools ever developed. By harnessing vast amounts of data and computational power, AI systems can un Read more…

Synopsys Eats Ansys: Does HPC Get Indigestion?

February 8, 2024

Recently, it was announced that Synopsys is buying HPC tool developer Ansys. Started in Pittsburgh, Pa., in 1970 as Swanson Analysis Systems, Inc. (SASI) by John Swanson (and eventually renamed), Ansys serves the CAE (Computer Aided Engineering)/multiphysics engineering simulation market. Read more…

Nvidia H100: Are 550,000 GPUs Enough for This Year?

August 17, 2023

The GPU Squeeze continues to place a premium on Nvidia H100 GPUs. In a recent Financial Times article, Nvidia reports that it expects to ship 550,000 of its lat Read more…

Comparing NVIDIA A100 and NVIDIA L40S: Which GPU is Ideal for AI and Graphics-Intensive Workloads?

October 30, 2023

With long lead times for the NVIDIA H100 and A100 GPUs, many organizations are looking at the new NVIDIA L40S GPU, which it’s a new GPU optimized for AI and g Read more…

Atos Outlines Plans to Get Acquired, and a Path Forward

May 21, 2024

Atos – via its subsidiary Eviden – is the second major supercomputer maker outside of HPE, while others have largely dropped out. The lack of integrators and Atos' financial turmoil have the HPC market worried. If Atos goes under, HPE will be the only major option for building large-scale systems. Read more…

Choosing the Right GPU for LLM Inference and Training

December 11, 2023

Accelerating the training and inference processes of deep learning models is crucial for unleashing their true potential and NVIDIA GPUs have emerged as a game- Read more…

Nvidia’s New Blackwell GPU Can Train AI Models with Trillions of Parameters

March 18, 2024

Nvidia's latest and fastest GPU, codenamed Blackwell, is here and will underpin the company's AI plans this year. The chip offers performance improvements from Read more…

Some Reasons Why Aurora Didn’t Take First Place in the Top500 List

May 15, 2024

The makers of the Aurora supercomputer, which is housed at the Argonne National Laboratory, gave some reasons why the system didn't make the top spot on the Top Read more…

AMD MI3000A

How AMD May Get Across the CUDA Moat

October 5, 2023

When discussing GenAI, the term "GPU" almost always enters the conversation and the topic often moves toward performance and access. Interestingly, the word "GPU" is assumed to mean "Nvidia" products. (As an aside, the popular Nvidia hardware used in GenAI are not technically... Read more…

Leading Solution Providers

Contributors

The GenAI Datacenter Squeeze Is Here

February 1, 2024

The immediate effect of the GenAI GPU Squeeze was to reduce availability, either direct purchase or cloud access, increase cost, and push demand through the roof. A secondary issue has been developing over the last several years. Even though your organization secured several racks... Read more…

The NASA Black Hole Plunge

May 7, 2024

We have all thought about it. No one has done it, but now, thanks to HPC, we see what it looks like. Hold on to your feet because NASA has released videos of wh Read more…

Shutterstock 1285747942

AMD’s Horsepower-packed MI300X GPU Beats Nvidia’s Upcoming H200

December 7, 2023

AMD and Nvidia are locked in an AI performance battle – much like the gaming GPU performance clash the companies have waged for decades. AMD has claimed it Read more…

Intel Plans Falcon Shores 2 GPU Supercomputing Chip for 2026  

August 8, 2023

Intel is planning to onboard a new version of the Falcon Shores chip in 2026, which is code-named Falcon Shores 2. The new product was announced by CEO Pat Gel Read more…

GenAI Having Major Impact on Data Culture, Survey Says

February 21, 2024

While 2023 was the year of GenAI, the adoption rates for GenAI did not match expectations. Most organizations are continuing to invest in GenAI but are yet to Read more…

Eyes on the Quantum Prize – D-Wave Says its Time is Now

January 30, 2024

Early quantum computing pioneer D-Wave again asserted – that at least for D-Wave – the commercial quantum era has begun. Speaking at its first in-person Ana Read more…

Q&A with Nvidia’s Chief of DGX Systems on the DGX-GB200 Rack-scale System

March 27, 2024

Pictures of Nvidia's new flagship mega-server, the DGX GB200, on the GTC show floor got favorable reactions on social media for the sheer amount of computing po Read more…

How the Chip Industry is Helping a Battery Company

May 8, 2024

Chip companies, once seen as engineering pure plays, are now at the center of geopolitical intrigue. Chip manufacturing firms, especially TSMC and Intel, have b Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire