SC13 Research Highlight: There Goes the Performance Neighborhood…

By Drs. Abhinav Bhatele, Kathryn Mohror, Steven Langer & Katherine Isaacs

November 16, 2013

Message passing can take up a significant fraction of the run time for massively parallel science simulation codes. Consistently high message passing rates are required for these codes to deliver good performance. At Supercomputing 2013 (SC13), our research team from Lawrence Livermore (LLNL) will present the results of our study that show that run-to-run variability in message passing rates can reduce throughput by 30% or more due to contention with other jobs for the network links.

Performance variability can result in individual jobs running slower, which in turn can lead to a longer wait for the science results and increase the waiting time in the queue for other jobs.  Reducing such variability could improve overall throughput at a computer center and save energy costs.  Performance variability also impacts the development cycle for high-performance computing (HPC) applications. It can complicate tasks such as: debugging performance issues in an application code, quantifying the effects of code changes on performance, measuring the effects of compiler or system software changes, and determining how much time to request for a batch job. Thus, we set out to investigate the possible sources of such performance variability on supercomputer systems.

In our study, we focused on pF3D, a code that simulates laser-plasma interactions in experiments at the National Ignition Facility at LLNL.  In 2011, we began doing production runs of pF3D on Cielo, a 1.37 Petaflop/s Cray XE6 system installed at Los Alamos. Concurrently, we ran pF3D on Dawn, an IBM Blue Gene/P system at LLNL. The run times of identical jobs on Cielo varied by 20% while there was very little variability on Dawn. The differences in run time were due to varying message passing rates. These early results prompted us to do a systematic study of message rate variability on three U.S.  Department of Energy (DOE) supercomputers: Intrepid, an IBM Blue Gene/P at Argonne (ANL), Mira, an IBM Blue Gene/Q at Argonne, and Hopper, a Cray XE6 at Lawrence Berkeley (LBNL).

Over the course of forty-five days, we submitted a short benchmarking run of pF3D every day to record the performance behavior of the application and some information about the system state, including the shape of the job partition allocated to the job, and other jobs running on the system and their node allocations. The “shape” of the job partition refers to the physical locations of the allocated nodes in the interconnect topology of the system.  Mira has a five-dimensional torus interconnect while Hopper and Intrepid have a three-dimensional torus interconnect. Below, we show a plot of the average messaging rate for each job as a function of when it was run on the three systems.  We compute the average messaging rate by dividing the total volume of communication in bytes by the total time spent in sending the messages over the network.

AbRevise

Click to enlarge

We see that on the IBM systems, Intrepid and Mira, there is negligible variation in the messaging performance. However, on the Cray system (Hopper), the slowest job on any given day may run at half the speed of the fastest job. Application users choose the amount of work to assign to each batch job to ensure that there is sufficient time to save results even on a day with poor performance.  This results in less average work completed per batch job than on a system with repeatable performance and the need for more batch job slots (and more calendar days) to complete a simulation.

In this SC13 paper (presentation schedule below), we attempt to narrow down the root causes of this performance variability on Hopper.  Several factors can make the performance of an application variable within and across batch jobs. These factors include noise from operating system (OS) daemons, communication variability arising from the shape of the allocated partition, and interference from other jobs sharing the same network links.  Below, we present observational evidence that indicates which factor leads to the highest performance variability. We show the placement of pF3D (blue) and conflicting jobs (other colors) on Hopper for two separate short runs in the figure below. The job from April 11 (left) yielded a messaging rate nearly 25% below that of the job run on April 16 (right). The two jobs had the same node placement, but the slower April 11 job was surrounded by several other jobs, including a large communication-heavy job (green).  More detailed analysis that provides stronger evidence for the effect of inter-job interference on performance can be found in the paper.

same-shape-different-rates

Acknowledgement: This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344. This work was funded by the Laboratory Directed Research and Development Program at LLNL under project tracking codes 13-ERD-055 and 13-FS-002 (LLNL-MI-645823).

Read more about this research at https://computation-rnd.llnl.gov/extreme-computing/interconnection-networks.php

Presentation Schedule:

Authors: Abhinav Bhatele, Kathryn Mohror, Steven H. Langer, Katherine E. Isaacs

Day: Tuesday (November 19, 2013)

Time: 4:00 – 4:30 PM

Location: 401/402/403

About the Authors

bhateleDr. Abhinav Bhatele is a computer scientist in the Center for Applied Scientific Computing at Lawrence Livermore National Laboratory. His interests lie in performance optimizations through analysis, visualization and tuning and developing algorithms for high-end parallel systems.

 

mahKathryn Mohror is a computer scientist at Lawrence Livermore National Laboratory (LLNL). She is a member of the Scalability Team at LLNL and her research on high-end computing systems is currently focused on scalable fault tolerant computing and performance measurement and analysis.
Steven Langer is currently a computational physicist at Lawrence Livermore National Laboratory. His research interests include inertial confinement fusion, performance analysis of HPC applications, scalability of message passing and I/O in applications, and understanding how to modify multi-physics simulation codes to run efficiently on upcoming exascale hardware.
Katherine Isaacs is a graduate student at the University of California, Davis. Her research focuses on information visualization techniques for performance analysis.
Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Hyperion: HPC Server Market Ekes 1 Percent Gain in 2020, Storage Poised for ‘Tipping Point’

May 12, 2021

The HPC User Forum meeting taking place virtually this week (May 11-13) kicked off with Hyperion Research’s market update, covering the 2020 period. Although the HPC server market had been facing a 6.7 percent COVID-re Read more…

Finland’s CSC Chronicles the COVID Research Performed on Its ‘Puhti’ Supercomputer

May 11, 2021

CSC, Finland’s IT Center for Science, is home to a variety of computing resources, including the 1.7 petaflops Puhti supercomputer. The 682-node, Intel Cascade Lake-powered system, which places about halfway down the T Read more…

IBM Debuts Qiskit Runtime for Quantum Computing; Reports Dramatic Speed-up

May 11, 2021

In conjunction with its virtual Think event, IBM today introduced an enhanced Qiskit Runtime Software for quantum computing, which it says demonstrated 120x speedup in simulating molecules. Qiskit is IBM’s quantum soft Read more…

AMD Chipmaker TSMC to Use AMD Chips for Chipmaking

May 8, 2021

TSMC has tapped AMD to support its major manufacturing and R&D workloads. AMD will provide its Epyc Rome 7702P CPUs – with 64 cores operating at a base clock of 2.0GHz – implemented in HPE's single-socket ProLian Read more…

Supercomputer Research Tracks the Loss of the World’s Glaciers

May 7, 2021

British Columbia – which is over twice the size of California – contains around 17,000 glaciers that cover three percent of its landmass. These glaciers are crucial for the Canadian province, which relies on its many Read more…

AWS Solution Channel

FLYING WHALES runs CFD workloads 15 times faster on AWS

FLYING WHALES is a French startup that is developing a 60-ton payload cargo airship for the heavy lift and outsize cargo market. The project was born out of France’s ambition to provide efficient, environmentally friendly transportation for collecting wood in remote areas. Read more…

Meet Dell’s Pete Manca, an HPCwire Person to Watch in 2021

May 7, 2021

Pete Manca heads up Dell's newly formed HPC and AI leadership group. As senior vice president of the integrated solutions engineering team, he is focused on custom design, technology alliances, high-performance computing Read more…

Hyperion: HPC Server Market Ekes 1 Percent Gain in 2020, Storage Poised for ‘Tipping Point’

May 12, 2021

The HPC User Forum meeting taking place virtually this week (May 11-13) kicked off with Hyperion Research’s market update, covering the 2020 period. Although Read more…

IBM Debuts Qiskit Runtime for Quantum Computing; Reports Dramatic Speed-up

May 11, 2021

In conjunction with its virtual Think event, IBM today introduced an enhanced Qiskit Runtime Software for quantum computing, which it says demonstrated 120x spe Read more…

AMD Chipmaker TSMC to Use AMD Chips for Chipmaking

May 8, 2021

TSMC has tapped AMD to support its major manufacturing and R&D workloads. AMD will provide its Epyc Rome 7702P CPUs – with 64 cores operating at a base cl Read more…

Fast Pass Through (Some of) the Quantum Landscape with ORNL’s Raphael Pooser

May 7, 2021

In a rather remarkable way, and despite the frequent hype, the behind-the-scenes work of developing quantum computing has dramatically accelerated in the past f Read more…

IBM Research Debuts 2nm Test Chip with 50 Billion Transistors

May 6, 2021

IBM Research today announced the successful prototyping of the world's first 2 nanometer chip, fabricated with silicon nanosheet technology on a standard 300mm Read more…

LRZ Announces New Phase of SuperMUC-NG Supercomputer with Intel’s ‘Ponte Vecchio’ GPU

May 5, 2021

At the Leibniz Supercomputing Centre (LRZ) in München, Germany – one of the constituent centers of the Gauss Centre for Supercomputing (GCS) – the SuperMUC Read more…

Crystal Ball Gazing at Nvidia: R&D Chief Bill Dally Talks Targets and Approach

May 4, 2021

There’s no quibbling with Nvidia’s success. Entrenched atop the GPU market, Nvidia has ridden its own inventiveness and growing demand for accelerated computing to meet the needs of HPC and AI. Recently it embarked on an ambitious expansion by acquiring Mellanox (interconnect)... Read more…

Intel Invests $3.5 Billion in New Mexico Fab to Focus on Foveros Packaging Technology

May 3, 2021

Intel announced it is investing $3.5 billion in its Rio Rancho, New Mexico, facility to support its advanced 3D manufacturing and packaging technology, Foveros. Read more…

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from Julialang.org. I Read more…

AMD Chipmaker TSMC to Use AMD Chips for Chipmaking

May 8, 2021

TSMC has tapped AMD to support its major manufacturing and R&D workloads. AMD will provide its Epyc Rome 7702P CPUs – with 64 cores operating at a base cl Read more…

Intel Launches 10nm ‘Ice Lake’ Datacenter CPU with Up to 40 Cores

April 6, 2021

The wait is over. Today Intel officially launched its 10nm datacenter CPU, the third-generation Intel Xeon Scalable processor, codenamed Ice Lake. With up to 40 Read more…

CERN Is Betting Big on Exascale

April 1, 2021

The European Organization for Nuclear Research (CERN) involves 23 countries, 15,000 researchers, billions of dollars a year, and the biggest machine in the worl Read more…

HPE Launches Storage Line Loaded with IBM’s Spectrum Scale File System

April 6, 2021

HPE today launched a new family of storage solutions bundled with IBM’s Spectrum Scale Erasure Code Edition parallel file system (description below) and featu Read more…

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

Saudi Aramco Unveils Dammam 7, Its New Top Ten Supercomputer

January 21, 2021

By revenue, oil and gas giant Saudi Aramco is one of the largest companies in the world, and it has historically employed commensurate amounts of supercomputing Read more…

Quantum Computer Start-up IonQ Plans IPO via SPAC

March 8, 2021

IonQ, a Maryland-based quantum computing start-up working with ion trap technology, plans to go public via a Special Purpose Acquisition Company (SPAC) merger a Read more…

Leading Solution Providers

Contributors

Can Deep Learning Replace Numerical Weather Prediction?

March 3, 2021

Numerical weather prediction (NWP) is a mainstay of supercomputing. Some of the first applications of the first supercomputers dealt with climate modeling, and Read more…

AMD Launches Epyc ‘Milan’ with 19 SKUs for HPC, Enterprise and Hyperscale

March 15, 2021

At a virtual launch event held today (Monday), AMD revealed its third-generation Epyc “Milan” CPU lineup: a set of 19 SKUs -- including the flagship 64-core, 280-watt 7763 part --  aimed at HPC, enterprise and cloud workloads. Notably, the third-gen Epyc Milan chips achieve 19 percent... Read more…

Livermore’s El Capitan Supercomputer to Debut HPE ‘Rabbit’ Near Node Local Storage

February 18, 2021

A near node local storage innovation called Rabbit factored heavily into Lawrence Livermore National Laboratory’s decision to select Cray’s proposal for its CORAL-2 machine, the lab’s first exascale-class supercomputer, El Capitan. Details of this new storage technology were revealed... Read more…

African Supercomputing Center Inaugurates ‘Toubkal,’ Most Powerful Supercomputer on the Continent

February 25, 2021

Historically, Africa hasn’t exactly been synonymous with supercomputing. There are only a handful of supercomputers on the continent, with few ranking on the Read more…

GTC21: Nvidia Launches cuQuantum; Dips a Toe in Quantum Computing

April 13, 2021

Yesterday Nvidia officially dipped a toe into quantum computing with the launch of cuQuantum SDK, a development platform for simulating quantum circuits on GPU-accelerated systems. As Nvidia CEO Jensen Huang emphasized in his keynote, Nvidia doesn’t plan to build... Read more…

New Deep Learning Algorithm Solves Rubik’s Cube

July 25, 2018

Solving (and attempting to solve) Rubik’s Cube has delighted millions of puzzle lovers since 1974 when the cube was invented by Hungarian sculptor and archite Read more…

The History of Supercomputing vs. COVID-19

March 9, 2021

The COVID-19 pandemic poses a greater challenge to the high-performance computing community than any before. HPCwire's coverage of the supercomputing response t Read more…

HPE Names Justin Hotard New HPC Chief as Pete Ungaro Departs

March 2, 2021

HPE CEO Antonio Neri announced today (March 2, 2021) the appointment of Justin Hotard as general manager of HPC, mission critical solutions and labs, effective Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire