Message passing can take up a significant fraction of the run time for massively parallel science simulation codes. Consistently high message passing rates are required for these codes to deliver good performance. At Supercomputing 2013 (SC13), our research team from Lawrence Livermore (LLNL) will present the results of our study that show that run-to-run variability in message passing rates can reduce throughput by 30% or more due to contention with other jobs for the network links.

Performance variability can result in individual jobs running slower, which in turn can lead to a longer wait for the science results and increase the waiting time in the queue for other jobs.  Reducing such variability could improve overall throughput at a computer center and save energy costs.  Performance variability also impacts the development cycle for high-performance computing (HPC) applications. It can complicate tasks such as: debugging performance issues in an application code, quantifying the effects of code changes on performance, measuring the effects of compiler or system software changes, and determining how much time to request for a batch job. Thus, we set out to investigate the possible sources of such performance variability on supercomputer systems.

In our study, we focused on pF3D, a code that simulates laser-plasma interactions in experiments at the National Ignition Facility at LLNL.  In 2011, we began doing production runs of pF3D on Cielo, a 1.37 Petaflop/s Cray XE6 system installed at Los Alamos. Concurrently, we ran pF3D on Dawn, an IBM Blue Gene/P system at LLNL. The run times of identical jobs on Cielo varied by 20% while there was very little variability on Dawn. The differences in run time were due to varying message passing rates. These early results prompted us to do a systematic study of message rate variability on three U.S.  Department of Energy (DOE) supercomputers: Intrepid, an IBM Blue Gene/P at Argonne (ANL), Mira, an IBM Blue Gene/Q at Argonne, and Hopper, a Cray XE6 at Lawrence Berkeley (LBNL).

Over the course of forty-five days, we submitted a short benchmarking run of pF3D every day to record the performance behavior of the application and some information about the system state, including the shape of the job partition allocated to the job, and other jobs running on the system and their node allocations. The “shape” of the job partition refers to the physical locations of the allocated nodes in the interconnect topology of the system.  Mira has a five-dimensional torus interconnect while Hopper and Intrepid have a three-dimensional torus interconnect. Below, we show a plot of the average messaging rate for each job as a function of when it was run on the three systems.  We compute the average messaging rate by dividing the total volume of communication in bytes by the total time spent in sending the messages over the network.

Click to enlarge

We see that on the IBM systems, Intrepid and Mira, there is negligible variation in the messaging performance. However, on the Cray system (Hopper), the slowest job on any given day may run at half the speed of the fastest job. Application users choose the amount of work to assign to each batch job to ensure that there is sufficient time to save results even on a day with poor performance.  This results in less average work completed per batch job than on a system with repeatable performance and the need for more batch job slots (and more calendar days) to complete a simulation.

In this SC13 paper (presentation schedule below), we attempt to narrow down the root causes of this performance variability on Hopper.  Several factors can make the performance of an application variable within and across batch jobs. These factors include noise from operating system (OS) daemons, communication variability arising from the shape of the allocated partition, and interference from other jobs sharing the same network links.  Below, we present observational evidence that indicates which factor leads to the highest performance variability. We show the placement of pF3D (blue) and conflicting jobs (other colors) on Hopper for two separate short runs in the figure below. The job from April 11 (left) yielded a messaging rate nearly 25% below that of the job run on April 16 (right). The two jobs had the same node placement, but the slower April 11 job was surrounded by several other jobs, including a large communication-heavy job (green).  More detailed analysis that provides stronger evidence for the effect of inter-job interference on performance can be found in the paper.

Acknowledgement: This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344. This work was funded by the Laboratory Directed Research and Development Program at LLNL under project tracking codes 13-ERD-055 and 13-FS-002 (LLNL-MI-645823).

Read more about this research at https://computation-rnd.llnl.gov/extreme-computing/interconnection-networks.php

Presentation Schedule:

Authors: Abhinav Bhatele, Kathryn Mohror, Steven H. Langer, Katherine E. Isaacs

Day: Tuesday (November 19, 2013)

Time: 4:00 – 4:30 PM

Location: 401/402/403

About the Authors

Dr. Abhinav Bhatele is a computer scientist in the Center for Applied Scientific Computing at Lawrence Livermore National Laboratory. His interests lie in performance optimizations through analysis, visualization and tuning and developing algorithms for high-end parallel systems.

Kathryn Mohror is a computer scientist at Lawrence Livermore National Laboratory (LLNL). She is a member of the Scalability Team at LLNL and her research on high-end computing systems is currently focused on scalable fault tolerant computing and performance measurement and analysis.