NERSC Scales Scientific Deep Learning to 15 Petaflops

By Rob Farber

August 28, 2017

A collaborative effort between Intel, NERSC and Stanford has delivered the first 15-petaflops deep learning software running on HPC platforms and is, according to the authors of the paper (and to the best of their knowledge), currently the most scalable deep-learning implementation in the world. The work described in the paper, Deep Learning at 15PF: Supervised and Semi-Supervised Classification for Scientific Data1, reported that a Cray XC40 system with a configuration of 9,600 self-hosted 1.4GHz Intel Xeon Phi Processor 7250 based nodes achieved a peak rate between 11.73 and 15.07 petaflops (single-precision) and an average sustained performance of 11.41 to 13.47 petaflops when training on physics and climate based data sets using Lawrence Berkeley National Laboratory’s (Berkeley Lab) NERSC (National Energy Research Scientific Computing Center) Cori Phase-II supercomputer. The group utilized an amalgamation of Intel Caffe, Intel Math Kernel Library (Intel MKL), and Intel Machine Learning Scaling Library (Intel MLSL) software to achieve this scalability and performance.2

Along with scalability, Joe Curley, Intel’s senior director of HPC platform and ecosystem enabling, highlights the scientific accomplishments this level of performance brings to deep learning researchers, and data-intensive scientific communities such as climate and High Energy Physics (HEP). He also pointed out how the results further establish the deep learning performance capability of the Intel Xeon Phi processor based computational nodes in the Cori supercomputer. “These were not just a set of heroic runs, they have solved real problems at the scale of a top five supercomputer using new methods,” Curley said.

Advancing Deep Learning at Scale

Prabhat, Data and Analytics Group Lead at NERSC, Berkeley Lab, emphasized that this performance and scalability result was very much a collaborative effort that: (A) utilized a neural network update scheme by Christopher Ré’s group (Department of Computer Science at Stanford University), (B) created a software infrastructure by the Parallel Computing Lab, Intel MKL and Intel MLSL product teams at Intel, and (C) leveraged the world-class people and hardware resources at NERSC.

Overall, Curley observes that the collaboration reported “reasonably good scaling performance” as the 9,600 node cluster delivered an approximate 7,205x speedup. (Perfect scaling would have delivered a 9,600x speedup.) Curly is excited by the potential of this early work stating, “Opportunities exist to improve performance and scaling in either future runs, or in the course of solving new problems. This really was an amazing early result on a fairly new machine.”

Algorithmic advances for deep learning scalability

The update scheme by Ré allows both the synchronous and asynchronous updates of the ANN (Artificial Neural Network) parameters.

Conceptually asynchronous updates (and asynchronous architectures in general) provide the ability to scale to large numbers of nodes by removing synchronization barriers. This lock-free approach allows for faster model updates, but can require more updates to yield an equally good final model.  Thus asynchronous updates can make the training process run longer – meaning it can take longer to converge. The thought process behind the use of asynchronous updates is that the extra computational nodes add enough parallelism (and hence can deliver greater performance) to overcome the potentially slower convergence behavior and thus deliver an overall faster time-to-model. Failure to converge to a good solution is also a possibility although Ioannis Mitliagkas, former Postdoctoral scholar at Stanford and currently Assistant Professor at the University of Montreal, observes from both classical and modern results on asynchrony that, “on well-behaved objectives, failure to converge implies a mis-tuned system.” Thus tuning is critical. However, it is worth noting that on deep learning objectives no system, synchronous or asynchronous is guaranteed to converge to a good solution.

The asynchronous deep learning architecture used in the paper is illustrated below.

Each node works on its own iteration (mini-batch) and produces independent updates to the model. Those updates are sent to a central parameter store called the parameter server (noted as PS in the figure), which applies the updates to the model in the order they are received. After each setup, the PS sends the new model, back to the worker where the update originated.

Asynchronous systems do not suffer from straggler effects and are not limited by the total batch size in the same way that synchronous systems are, an important property at scale.

Figure 1: Example synchronous and asynchronous architectures

Staleness is the reason that asynchronous systems may need more iterations to converge to a solution. Said another way, they have worse statistical efficiency.

In contrast, the reduction operation used by synchronous training introduces an O(log(#Nodes)) runtime growth. Further, it make the training susceptible to jitter (e.g. the “straggler effect”) as the computation can be rate limited by the slowest node in the system during each iteration of the training procedure.3 The straggler effect occurs when any delay on any node exceeds the ability of the implementation of the reduction operation to hide latency. Achieving low latency is an important goal for developers of products such as Intel Omni-Path Architecture (Intel OPA) MPI libraries. In addition, using too many nodes during training can reduce the number of examples per node (e.g. the mini-batch size) to the point of reduced node efficiency. Thus the authors note in their paper that synchronous training can potentially deliver worse hardware efficiency.

The trade-off between statistical efficiency vs. hardware efficiency suggested a third kind of architecture to the paper authors, which they call a hybrid system.

Mitliagkas points out, “Synchronous systems are the classic, straightforward approach, and it has some good and bad attributes. The bad attributes (straggler effect and susceptibility to slow nodes, and huge effective mini-batches at scale) motivate asynchronous systems. Those have different strengths and weaknesses motivating a tradeoff. Hybrid systems give you control of the tradeoff.”

In the hybrid approach, worker nodes coalesce into separate, synchronous compute groups where the workers split a mini-batch quantity of work among themselves to produce a single update to the model. There is no synchronization across compute groups so they are able to run asynchronously. The hybrid architecture used in the paper is illustrated below.

Figure 2: Hybrid architecture example

The authors report they observed better scaling for their hybrid asynchronous updates over synchronous configurations due to reduced straggler effects. “This has been a big engineering effort,” Mitliagkas notes, “On the Stanford side, this would not have been possible without the engineering skills and hard work of Jian Zhang.” He also points out that in large-scale HPC runs, say 10,000 nodes, the likelihood that there will be some ’slow’ nodes is significant. For this reason, synchronous system’s performance can be really unpredictable. On the other hand, asynchronous systems degrade more gracefully: a slow node only affects a single synchronous group. Results in their paper show the hybrid method performing 1.66x better than the best synchronous run, and about 10x better than the worst synchronous run.

Figure 3: Training losses vs wall clock time for HEP on 1K nodes. Comparing synchronous configuration to 2, 4, and 8 groups

The weak scaling plots below, where the amount of work per node is kept constant, show that the scalability of the system can vary with the task.

Mitliagkas emphatically states, “People typically report weak scaling, because strong scaling is hard.” He continues, “For machine learning systems, strong scaling (keeping the total amount of work constant) is more representative of actual performance.”4 He reinforces his point by stating that, “synchronous approaches can only scale up to the size of the mini-batch.”

Figure 5: Strong scaling results for synchronous and hybrid approaches (batch size = 2048 per synchronous group).

Finding a good configuration is not an easy task. Recognizing this complexity, Thorsten Kurth, HPC consultant at NERSC, notes: “it is unreasonable to expect scientists to be conversant in the art of hyper-parameter tuning. Hybrid schemes, like the one presented in this paper, add an extra parameter to be tuned, which stresses the need for principled momentum tuning approaches, an active area of research (eg. YellowFin). With hyper-parameter tuning taken care of, higher-level libraries such as Spearmint can be used for automating the search for network architectures.”

Reduced precision

The results presented in the paper were based on 32-bit, single-precision arithmetic because there are open questions regarding the use of reduced precision for training. Specifically, Thorsten observes, “more aggressive optimizations involving computing in low-precision and communicating high order bits of weight updates are poorly understood with regards to their implications for classification and regression accuracy for scientific datasets” [Italic emphasis by the authors]. He concludes, “The field of Deep Learning is evolving rapidly, and we look forward to adopting advances in the near future.”

Asynchronous momentum and tuning the convergence rate

Mitliagkas notes that in the past industry groups have reported very good performance from asynchronous systems in commercial applications. In building on that work, his post-doc showed that asynchronous behavior effectively introduces a momentum term into the optimization. However, this momentum needs to be tuned dynamically to take into account past history as it can have significant impact on the convergence rate. Mitliagkas makes the point that the hybrid architecture means the user does not have to choose to run entirely in synchronous or asynchronous mode but can tune the hybrid method to best fit the machine and problem. In his opinion, this flexibility makes hybrid systems much more useful for general users and accounts for issues general users have had in the past with purely asynchronous systems. More information can be found in the Omnivore system. Mitliagkas and Zhang’s newer work, Yellowfin, automates the process of automatic tuning even more as described in this Stanford blog post.

Intel MLSL

Nadathur Satish, Research Scientist at Intel, noted, “The Intel team performed a significant amount of work to extend the Intel MLSL library to support the hybrid asynchronous code for this paper.”  Specifically the Intel MLSL team added the ability to instantiate multiple synchronous groups and interface it with a parameter server. The Intel MLSL was initially designed to provide scalable behavior for deep learning synchronous codes. Satish noted that scalability is key to advancing the field of deep learning.

Deep Learning for Science

Prabhat notes, “For this paper, it was critical for us to demonstrate the viability of scaling Deep Learning for real scientific applications, in contrast to ImageNet. We have numerous scientific workloads at NERSC that are currently using Deep Learning, this work sets a high bar for HPC systems. Thorsten Kurth and Wahid Bhimiji chose to demonstrate the efficacy of training on simulated HEP data at scale to learn how to separate the rare signals of new particles from background events – without human intervention. Improvements in identifying these new particles could aid discoveries that might redefine our understanding of the fundamental nature of our universe. Similarly, Evan Racah and I took on the problem of identifying features in climate data. Automatically extracting such patterns will enable us to better characterize changes in frequency and intensity of extreme weather under climate change.”

High-Energy Physics

For the paper, the team used data from an LHC simulator to identify massive supersymmetric particles in multi-jet final states as they should appear in real-life at the LHC. This required training on 10 million events contained in roughly 10 terabytes of data. The team verified the trained ANN had similar baseline performance as reported by the ATLAS collaboration. Kurth summarize the results by saying, “The capability to achieve high sensitivities to new-physics signals from classification on low-level detector quantities, without the need to design, reconstruct, or tune, high-level features offers considerable potential for enabling new-physics discoveries in future HEP analyses.”


For the task of detecting extreme weather patterns, the team developed a novel semi-supervised architecture that uses an auto-encoder to capture various patterns in the dataset and simultaneously asks the network to predict bounding boxes for known patterns (such as hurricanes , extra-tropical cyclones and atmospheric rivers). Figure 6 highlights predictions from the network: Black bounding boxes show ground truth and Red boxes are predictions by the network. Note the strong overlap in all but one example.

Figure 6: Results from plotting the network’s most confident (>95%) box predictions on an image for integrated water vapor (TMQ) from the test set for the climate problem.

Single-node Cori Intel Xeon Phi processor performance

The core computation in deep learning is dense linear algebra; specifically matrix multiply and convolution operations. The authors observe that the hardware efficiency of these kernels heavily depends on input data sizes and model parameters (weight matrix dimensions, number of convolutions, convolution strides, padding, and etcetera).

As a reference, they note that DeepBench from Baidu captures the best known performance of deep learning kernels with varied input sizes and model parameters on NVIDIA GPUs and Intel Xeon Phi processors. DeepBench results show that performance on all architectures can be as high as 75-80% of peak flops for some kernels,  and as low as 20-30% as the minibatch size decreases (determined by dimension ’N’ for matrix multiply and convolutions).

With those caveats in place, the team reports that an Intel Xeon Phi processor 7250 can deliver a 2.09 teraflops overall flop rate for the climate network and 1.90 teraflops for the HEP network. For both networks, most of the runtime is spent in convolutional layers, which can obtain between 3.5 teraflops for layers with many channels, and around 1.25 teraflops on the initial layers with very few channels.1


Work by a number of researchers around the world is demonstrating both the usefulness, performance, and scalability of deep learning on very large, data intensive workloads. High per node performance is simply not enough as people apply deep learning technology to increasingly complex problems. Succinctly, the more complex the problem, the larger the data set that is required to adequately represent the problem space during training. With scalable algorithms, researchers can train on orders of magnitude larger data sets and achieve thousands of times faster time-to-model performance which, in turn, means they can address more complex (and potentially more valuable) problems. Further, this collaborative effort by Intel, NERSC, and Stanford shows that deep learning is a candidate exascale workload that can help realize the tremendous potential of computing at the exascale.

About the Author

Rob Farber is a global technology consultant and author with an extensive background in HPC and in developing machine learning technology that he applies at national labs and commercial organizations. Rob can be reached at [email protected].


2 NERSC is a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231.

3 I recommend the paper “The case of the missing supercomputer performance” to better understand the impact of jitter at scale in HPC systems.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Democratization of HPC Part 3: Ninth Graders Tap HPC in the Cloud to Design Flying Boats

October 18, 2018

This is the third in a series of articles demonstrating the growing acceptance of high-performance computing (HPC) in new user communities and application areas. In this article we present UberCloud use case #208 on how Read more…

By Wolfgang Gentzsch and Håkon Bull Hove

Penguin Computing Launches Consultancy for Piecing AI Strategies Together

October 18, 2018

AI stands before the HPC industry as a beacon of great expectations, yet market research repeatedly shows that AI adoption is commonly stuck in the talking phase, on the near side of a difficult chasm to cross. In respon Read more…

By Tiffany Trader

When Water Quality—Not Quantity—Hinders HPC Cooling

October 18, 2018

Attention has been paid to the sheer quantity of water consumed by supercomputers’ cooling towers – and rightly so, as they can require thousands of gallons per minute to cool. But in the background, another factor can emerge, bottlenecking efficiency and raising costs: water quality. Read more…

By Oliver Peckham

HPE Extreme Performance Solutions

One Small Step Toward Mars: One Giant Leap for Supercomputing

Since the days of the Space Race between the U.S. and the former Soviet Union, we have continually sought ways to perform experiments in space. Read more…

IBM Accelerated Insights

Paper Offers ‘Proof’ of Quantum Advantage on Some Problems

October 18, 2018

Is quantum computing worth all the effort being poured into it or should we just wait for classical computing to catch up? An IBM blog today posed those questions and, you won’t be surprised, offers a firm “it’s wo Read more…

By John Russell

Penguin Computing Launches Consultancy for Piecing AI Strategies Together

October 18, 2018

AI stands before the HPC industry as a beacon of great expectations, yet market research repeatedly shows that AI adoption is commonly stuck in the talking phas Read more…

By Tiffany Trader

When Water Quality—Not Quantity—Hinders HPC Cooling

October 18, 2018

Attention has been paid to the sheer quantity of water consumed by supercomputers’ cooling towers – and rightly so, as they can require thousands of gallons per minute to cool. But in the background, another factor can emerge, bottlenecking efficiency and raising costs: water quality. Read more…

By Oliver Peckham

Paper Offers ‘Proof’ of Quantum Advantage on Some Problems

October 18, 2018

Is quantum computing worth all the effort being poured into it or should we just wait for classical computing to catch up? An IBM blog today posed those questio Read more…

By John Russell

Dell EMC to Supply U Michigan’s Great Lakes Cluster

October 16, 2018

The University of Michigan (U-M) today announced Dell EMC is the lead vendor for U-M’s $4.8 million Great Lakes HPC cluster scheduled for deployment in first Read more…

By John Russell

Houston to Field Massive, ‘Geophysically Configured’ Cloud Supercomputer

October 11, 2018

Based on some news stories out today, one might get the impression that the next system to crack number one on the Top500 would be an industrial oil and gas mon Read more…

By Tiffany Trader

Nvidia Platform Pushes GPUs into Machine Learning, High Performance Data Analytics

October 10, 2018

GPU leader Nvidia, generally associated with deep learning, autonomous vehicles and other higher-end enterprise and scientific workloads (and gaming, of course) Read more…

By Doug Black

Federal Investment in Exascale – What It Really Means

October 10, 2018

Earlier this month, the EuroHPC JU (Joint Undertaking) reached critical mass, and it seems all EU and affiliated member states, bar the UK (unsurprisingly), have or will sign on. The EuroHPC JU was born from a recognition that individual EU member states, and the EU as a whole, were significantly underinvesting in HPC compared to the US, China and Japan, who all have their own exascale investment and delivery strategies (NSCI, 13th 5 Year Plan, Post-K, etc). Read more…

By Dairsie Latimer

NERSC-9 Clues Found in NERSC 2017 Annual Report

October 8, 2018

If you’re eager to find out who’ll supply NERSC’s next-gen supercomputer, codenamed NERSC-9, here’s a project update to tide you over until the winning bid and system details are revealed. The upcoming system is referenced several times in the recently published 2017 NERSC annual report. Read more…

By Tiffany Trader

TACC Wins Next NSF-funded Major Supercomputer

July 30, 2018

The Texas Advanced Computing Center (TACC) has won the next NSF-funded big supercomputer beating out rivals including the National Center for Supercomputing Ap Read more…

By John Russell

IBM at Hot Chips: What’s Next for Power

August 23, 2018

With processor, memory and networking technologies all racing to fill in for an ailing Moore’s law, the era of the heterogeneous datacenter is well underway, Read more…

By Tiffany Trader

Requiem for a Phi: Knights Landing Discontinued

July 25, 2018

On Monday, Intel made public its end of life strategy for the Knights Landing "KNL" Phi product set. The announcement makes official what has already been wide Read more…

By Tiffany Trader

CERN Project Sees Orders-of-Magnitude Speedup with AI Approach

August 14, 2018

An award-winning effort at CERN has demonstrated potential to significantly change how the physics based modeling and simulation communities view machine learni Read more…

By Rob Farber

House Passes $1.275B National Quantum Initiative

September 17, 2018

Last Thursday the U.S. House of Representatives passed the National Quantum Initiative Act (NQIA) intended to accelerate quantum computing research and developm Read more…

By John Russell

Summit Supercomputer is Already Making its Mark on Science

September 20, 2018

Summit, now the fastest supercomputer in the world, is quickly making its mark in science – five of the six finalists just announced for the prestigious 2018 Read more…

By John Russell

New Deep Learning Algorithm Solves Rubik’s Cube

July 25, 2018

Solving (and attempting to solve) Rubik’s Cube has delighted millions of puzzle lovers since 1974 when the cube was invented by Hungarian sculptor and archite Read more…

By John Russell

D-Wave Breaks New Ground in Quantum Simulation

July 16, 2018

Last Friday D-Wave scientists and colleagues published work in Science which they say represents the first fulfillment of Richard Feynman’s 1982 notion that Read more…

By John Russell

Leading Solution Providers

HPC on Wall Street 2018 Booth Video Tours Playlist


Dell EMC





TACC’s ‘Frontera’ Supercomputer Expands Horizon for Extreme-Scale Science

August 29, 2018

The National Science Foundation and the Texas Advanced Computing Center announced today that a new system, called Frontera, will overtake Stampede 2 as the fast Read more…

By Tiffany Trader

HPE No. 1, IBM Surges, in ‘Bucking Bronco’ High Performance Server Market

September 27, 2018

Riding healthy U.S. and global economies, strong demand for AI-capable hardware and other tailwind trends, the high performance computing server market jumped 28 percent in the second quarter 2018 to $3.7 billion, up from $2.9 billion for the same period last year, according to industry analyst firm Hyperion Research. Read more…

By Doug Black

Intel Announces Cooper Lake, Advances AI Strategy

August 9, 2018

Intel's chief datacenter exec Navin Shenoy kicked off the company's Data-Centric Innovation Summit Wednesday, the day-long program devoted to Intel's datacenter Read more…

By Tiffany Trader

GPUs Power Five of World’s Top Seven Supercomputers

June 25, 2018

The top 10 echelon of the newly minted Top500 list boasts three powerful new systems with one common engine: the Nvidia Volta V100 general-purpose graphics proc Read more…

By Tiffany Trader

Germany Celebrates Launch of Two Fastest Supercomputers

September 26, 2018

The new high-performance computer SuperMUC-NG at the Leibniz Supercomputing Center (LRZ) in Garching is the fastest computer in Germany and one of the fastest i Read more…

By Tiffany Trader

MLPerf – Will New Machine Learning Benchmark Help Propel AI Forward?

May 2, 2018

Let the AI benchmarking wars begin. Today, a diverse group from academia and industry – Google, Baidu, Intel, AMD, Harvard, and Stanford among them – releas Read more…

By John Russell

Aerodynamic Simulation Reveals Best Position in a Peloton of Cyclists

July 5, 2018

Eindhoven University of Technology (TU/e) and KU Leuven research group conducts the largest numerical simulation ever done in the sport industry and cycling discipline. The goal was to understand the aerodynamic interactions in the peloton, i.e., the main pack of cyclists in a race. Read more…

Houston to Field Massive, ‘Geophysically Configured’ Cloud Supercomputer

October 11, 2018

Based on some news stories out today, one might get the impression that the next system to crack number one on the Top500 would be an industrial oil and gas mon Read more…

By Tiffany Trader

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This