TACC’s Texascale Days: Empowering Researchers to Tackle Grand Challenges with Frontera

April 3, 2024

April 3, 2024 — One of the biggest thrills for scientists who code is scaling up their simulations to push the limit of the most powerful supercomputers. Texascale Days at the Texas Advanced Computing Center (TACC) gives scientists that rare opportunity.

The quarterly event awards a handful of research groups full use of the National Science Foundation-funded Frontera supercomputer, the fastest supercomputer at any U.S. university and the leading capability system in the national cyberinfrastructure intended for large applications that require thousands of compute nodes.

Texascale Days on TACC’s Frontera supercomputer gives scientists full access to the most powerful supercomputer at any U.S. university. Shown is a volume rendered view of a thin equatorial slice of the model 25 Mʘ star. Credit: Paul Woodward, University of Minnesota.

“Texascale Days gives the researcher an opportunity to run code on problems at a scale that is not available during regular production on any NSF system,” said John Cazes, director of High Performance Computing at TACC.

Normally, at any given time, dozens of scientists share Frontera and run scientific computational jobs that need less than a quarter of its 8,300 Intel Cascade Lake Xeon nodes, supplemented by 90 graphics processing unit (GPU) nodes of NVIDIA Quadro RTX 5000. Allocations are requested through the Frontera user portal and the National Artificial Intelligence Research Resource.

“Texascale Days is different in that the simulations have demonstrated through smaller jobs that they can scale up to at least half the nodes on Frontera. It takes quite a bit of expertise and work to optimize the researcher’s code to hit that scale,” Cazes added.

Following are highlights of production and benchmarking runs from the latest Texascale Days in February 2024.

Wish Upon a Star

The magnitude of the horizontal velocity component in stellar convection simulations is shown volume rendered in a thin slice through the center of a 25 Mʘ star at intervals of 8.202 days, beginning at 8.202 days. The internal gravity waves excited by the core convection zone grow and work their way outward in time to influence the wave oscillations in the entire stably stratified envelope between the two convection zones. Credit: Paul Woodward, University of Minnesota.

The team of astronomer Paul Woodward at the University of Minnesota, in collaboration with Falk Herwig’s team at the University of Victoria, has been studying convection and its effects on the deep interiors of massive stars for several years. The gravity wave oscillations that can be seen using instruments like the Kepler space telescope and the Transiting Exoplanet Survey Satellite can provide a unique window into the interior structure of massive stars.

“These internal gravity waves (IGWs) can provide a connection between simulations and observations,” Woodward said.

Stellar hydrodynamic simulations have demonstrated that IGWs are excited by convection in the stellar core, and Woodward’s team has shown that features in the spectrum of excited waves and their stochastic time dependence bear a resemblance to the low-frequency excess that is observed.

However, open questions about the origins of IGWs prevent the scientific community from fully exploiting asteroseismic observations of massive stars. To resolve this question, the teams need simulations that reveal how the low-frequency waves are excited by core convection in the inner regions of the stable layer.

“The fine simulation grids and the high computational performance made possible with our PPMstar code running on Frontera enables us to resolve both the core convection, the near-surface convection, and the proper excitation and damping of IGWs in the stably stratified envelope between these convection zones,” Woodward said. “Our team exploited the most recent Texascale Days opportunity to perform some first experiments at scale to maximum of 3,510 nodes, in which we include nearly the entire star in our computational domain.”

“These are our first simulations at scale of full star models,” he added. “We learn from these numerical experiments how much gravity wave signals can tell us about the structure of a massive star’s deep interior.”

Cosmic History

The ASTRID cosmological simulation models large volumes of the cosmos spanning hundreds of millions of light years yet can zoom in to very high resolution. Credit: ASTRID team.

ASTRID, one of the largest-ever cosmological simulations, was developed on Frontera, and it too had its day during Texascale Days. It maxed out Frontera at 8,192 nodes during the peak of the simulation. The goal is to study galaxy formation, supermassive black hole coalescence, and re-ionization over the cosmic history.

“The Texascale Days run was very successful,” said Nianyi Chen of Carnegie Mellon University, “and utilized an optimized version of our cosmological hydrodynamics code MP-Gadget. “We evolved the ASTRID simulation by about 100 million years while efficiently processing galaxy and black hole catalogs on the fly.”, The science team includes Tiziana Di Matteo (CMU); Simeon Bird (UC Riverside); Yueying Ni (Harvard); and graduate students Yihao Zhou (CMU) and Yanhui Yang (UC Riverside).

“We finished massively parallelized I/O for a total of a few hundred terabytes of data during the 24-hour run. The adaptation of our code to the Frontera cores produced a speed-up of about 10 percent on our problem,” she added.

“The Texascale Day resources are crucial for this part of the ASTRID production run: our simulation is at the peak of cosmic star formation, and we need a larger memory to accommodate the information from the newly formed stars and galaxies. It provides a precious opportunity to test the scalability and reliability of our simulation code in a massively parallel context, allowing us to make further improvements to our simulation code for robust performance on large machines like Frontera and continue to push the simulation to the present day universe,” Chen said.

That’s A Moiré

The average computed density of the electron comprising one of the strongly bound excitonic states in a 55 atom silver nanoparticle (overlaid). Credit: The Jornada Group.

When two layers of atomically thin materials overlap, they can produce a moiré pattern that creates intriguing electronic phenomena such as superconductivity and ferromagnetism. What’s more, bouncing light off overlapping sheets of exotic materials can produce excitons, which are quasiparticles being studied for applications in new optical sensors and communication technology such as optical fibers and lasers.

“Using TACC’s Frontera supercomputer, we performed first-principles density functional theory calculations of the electronic ground state energies and wave functions for a plasmonic nanoparticle of experimentally relevant size,” said Felipe Jornada, an assistant professor in the Department of Materials Science and Engineering at Stanford University and a principal investigator at the SLAC National Accelerator Laboratory.

The Jornada Group needed over 4,000 nodes of Frontera to capture the atomistic details in these nanoparticles and the complex way that their electrons interact with light, using computationally demanding quantum-mechanical theories.

“This is the first calculation of its kind that addresses the intricate nature of the atomic structure of such nanoparticles, and the resultant correlations left behind in the electronic system after the photoexcitation,” Jornada said.

Plasmonic nanoparticles can be used to drive chemical reactions such as the production of ammonia fertilizer and hydrogen fuel, as well as plastic decomposition, using light instead of costly high temperature and pressure conditions created by burning fossil fuels.

“We think this is an exciting time where our theories, codes, and computational resources finally let us make practical predictions for new, light-driven chemical reactions,” added PH.D. student Akash Ramdas in the Jornada Group.

Going Nuclear

Density and the shape of the 24Mg ground state from the first-principle nuclear theory calculations. The apparent deformation of 24Mg nucleus is visible from the simulation obtained during the February 2024 Texascale Days on Frontera. Credit: Kristina Launey, LSU.

The isotope magnesium-24 (24Mg) is a heavy hitter in the universe. It’s one of the 10 most common elements in our galaxy and is vital in the synthesis of nuclei that form stars. During the Texascale Days event, a team led by Kristina Launey at Louisiana State University and Grigor Sargsyan at Michigan State University performed several large-scale simulations for the atomic nucleus of 24Mg across the entire 8,000+ nodes of Frontera.

Launey’s team uses a many-body method based on first principle approaches, which takes into account the underlying interactions of protons and neutrons. (use live link) Descriptions of alpha-conjugate nucleus — nuclei with multiples of alpha particles, i.e., two protons and two neutrons, such as 24Mg — are challenging to derive from first principle approaches.

“Texascale Days allowed us to utilize the full power of one of the largest supercomputers in the world to expand the first-principle simulations to heavier and more challenging nuclei,” Launey said.

“Almost all chemical elements on Earth have been created in the stars thanks to complex chains of nuclear processes,” said Grigor Sargsyan, Michigan State University, who is a co-PI on the Frontera allocation and a member of Launey’s team that does these first-principle calculations.

“To understand how these chains proceed, a reliable description of nuclear properties is needed. Thanks to the modern-day supercomputers and the advances in nuclear modeling, we are greatly expanding our knowledge of nuclear properties and complement the measurements at the state-of-the-art nuclear physics laboratories,” Sargsyan said.

A New Vista

The large-scale experiences gained from Texascale Days on Frontera apply to new systems on the horizon for TACC, such as Vista, slated for production in Summer of 2024 with an artificial intelligence focus.

“Texascale Days have been a great success for TACC in helping stress-test our flagship system, and for researchers in optimizing their codes to run at scales of the largest supercomputers in the world,” Cazes said. “We look forward to more years of Texascale Days on Frontera and on new, exciting systems to come.”

Source: Jorge Salazar, TACC

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industry updates delivered to you every week!

Empowering High-Performance Computing for Artificial Intelligence

April 19, 2024

Artificial intelligence (AI) presents some of the most challenging demands in information technology, especially concerning computing power and data movement. As a result of these challenges, high-performance computing Read more…

Kathy Yelick on Post-Exascale Challenges

April 18, 2024

With the exascale era underway, the HPC community is already turning its attention to zettascale computing, the next of the 1,000-fold performance leaps that have occurred about once a decade. With this in mind, the ISC Read more…

2024 Winter Classic: Texas Two Step

April 18, 2024

Texas Tech University. Their middle name is ‘tech’, so it’s no surprise that they’ve been fielding not one, but two teams in the last three Winter Classic cluster competitions. Their teams, dubbed Matador and Red Read more…

2024 Winter Classic: The Return of Team Fayetteville

April 18, 2024

Hailing from Fayetteville, NC, Fayetteville State University stayed under the radar in their first Winter Classic competition in 2022. Solid students for sure, but not a lot of HPC experience. All good. They didn’t Read more…

Software Specialist Horizon Quantum to Build First-of-a-Kind Hardware Testbed

April 18, 2024

Horizon Quantum Computing, a Singapore-based quantum software start-up, announced today it would build its own testbed of quantum computers, starting with use of Rigetti’s Novera 9-qubit QPU. The approach by a quantum Read more…

2024 Winter Classic: Meet Team Morehouse

April 17, 2024

Morehouse College? The university is well-known for their long list of illustrious graduates, the rigor of their academics, and the quality of the instruction. They were one of the first schools to sign up for the Winter Read more…

Kathy Yelick on Post-Exascale Challenges

April 18, 2024

With the exascale era underway, the HPC community is already turning its attention to zettascale computing, the next of the 1,000-fold performance leaps that ha Read more…

Software Specialist Horizon Quantum to Build First-of-a-Kind Hardware Testbed

April 18, 2024

Horizon Quantum Computing, a Singapore-based quantum software start-up, announced today it would build its own testbed of quantum computers, starting with use o Read more…

MLCommons Launches New AI Safety Benchmark Initiative

April 16, 2024

MLCommons, organizer of the popular MLPerf benchmarking exercises (training and inference), is starting a new effort to benchmark AI Safety, one of the most pre Read more…

Exciting Updates From Stanford HAI’s Seventh Annual AI Index Report

April 15, 2024

As the AI revolution marches on, it is vital to continually reassess how this technology is reshaping our world. To that end, researchers at Stanford’s Instit Read more…

Intel’s Vision Advantage: Chips Are Available Off-the-Shelf

April 11, 2024

The chip market is facing a crisis: chip development is now concentrated in the hands of the few. A confluence of events this week reminded us how few chips Read more…

The VC View: Quantonation’s Deep Dive into Funding Quantum Start-ups

April 11, 2024

Yesterday Quantonation — which promotes itself as a one-of-a-kind venture capital (VC) company specializing in quantum science and deep physics  — announce Read more…

Nvidia’s GTC Is the New Intel IDF

April 9, 2024

After many years, Nvidia's GPU Technology Conference (GTC) was back in person and has become the conference for those who care about semiconductors and AI. I Read more…

Google Announces Homegrown ARM-based CPUs 

April 9, 2024

Google sprang a surprise at the ongoing Google Next Cloud conference by introducing its own ARM-based CPU called Axion, which will be offered to customers in it Read more…

Nvidia H100: Are 550,000 GPUs Enough for This Year?

August 17, 2023

The GPU Squeeze continues to place a premium on Nvidia H100 GPUs. In a recent Financial Times article, Nvidia reports that it expects to ship 550,000 of its lat Read more…

Synopsys Eats Ansys: Does HPC Get Indigestion?

February 8, 2024

Recently, it was announced that Synopsys is buying HPC tool developer Ansys. Started in Pittsburgh, Pa., in 1970 as Swanson Analysis Systems, Inc. (SASI) by John Swanson (and eventually renamed), Ansys serves the CAE (Computer Aided Engineering)/multiphysics engineering simulation market. Read more…

Intel’s Server and PC Chip Development Will Blur After 2025

January 15, 2024

Intel's dealing with much more than chip rivals breathing down its neck; it is simultaneously integrating a bevy of new technologies such as chiplets, artificia Read more…

Choosing the Right GPU for LLM Inference and Training

December 11, 2023

Accelerating the training and inference processes of deep learning models is crucial for unleashing their true potential and NVIDIA GPUs have emerged as a game- Read more…

Baidu Exits Quantum, Closely Following Alibaba’s Earlier Move

January 5, 2024

Reuters reported this week that Baidu, China’s giant e-commerce and services provider, is exiting the quantum computing development arena. Reuters reported � Read more…

Comparing NVIDIA A100 and NVIDIA L40S: Which GPU is Ideal for AI and Graphics-Intensive Workloads?

October 30, 2023

With long lead times for the NVIDIA H100 and A100 GPUs, many organizations are looking at the new NVIDIA L40S GPU, which it’s a new GPU optimized for AI and g Read more…

Shutterstock 1179408610

Google Addresses the Mysteries of Its Hypercomputer 

December 28, 2023

When Google launched its Hypercomputer earlier this month (December 2023), the first reaction was, "Say what?" It turns out that the Hypercomputer is Google's t Read more…


How AMD May Get Across the CUDA Moat

October 5, 2023

When discussing GenAI, the term "GPU" almost always enters the conversation and the topic often moves toward performance and access. Interestingly, the word "GPU" is assumed to mean "Nvidia" products. (As an aside, the popular Nvidia hardware used in GenAI are not technically... Read more…

Leading Solution Providers


Shutterstock 1606064203

Meta’s Zuckerberg Puts Its AI Future in the Hands of 600,000 GPUs

January 25, 2024

In under two minutes, Meta's CEO, Mark Zuckerberg, laid out the company's AI plans, which included a plan to build an artificial intelligence system with the eq Read more…

China Is All In on a RISC-V Future

January 8, 2024

The state of RISC-V in China was discussed in a recent report released by the Jamestown Foundation, a Washington, D.C.-based think tank. The report, entitled "E Read more…

Shutterstock 1285747942

AMD’s Horsepower-packed MI300X GPU Beats Nvidia’s Upcoming H200

December 7, 2023

AMD and Nvidia are locked in an AI performance battle – much like the gaming GPU performance clash the companies have waged for decades. AMD has claimed it Read more…

Nvidia’s New Blackwell GPU Can Train AI Models with Trillions of Parameters

March 18, 2024

Nvidia's latest and fastest GPU, codenamed Blackwell, is here and will underpin the company's AI plans this year. The chip offers performance improvements from Read more…

Eyes on the Quantum Prize – D-Wave Says its Time is Now

January 30, 2024

Early quantum computing pioneer D-Wave again asserted – that at least for D-Wave – the commercial quantum era has begun. Speaking at its first in-person Ana Read more…

GenAI Having Major Impact on Data Culture, Survey Says

February 21, 2024

While 2023 was the year of GenAI, the adoption rates for GenAI did not match expectations. Most organizations are continuing to invest in GenAI but are yet to Read more…

The GenAI Datacenter Squeeze Is Here

February 1, 2024

The immediate effect of the GenAI GPU Squeeze was to reduce availability, either direct purchase or cloud access, increase cost, and push demand through the roof. A secondary issue has been developing over the last several years. Even though your organization secured several racks... Read more…

Intel’s Xeon General Manager Talks about Server Chips 

January 2, 2024

Intel is talking data-center growth and is done digging graves for its dead enterprise products, including GPUs, storage, and networking products, which fell to Read more…

  • arrow
  • Click Here for More Headlines
  • arrow