NERSC Scales Scientific Deep Learning to 15 Petaflops

By Rob Farber

August 28, 2017

A collaborative effort between Intel, NERSC and Stanford has delivered the first 15-petaflops deep learning software running on HPC platforms and is, according to the authors of the paper (and to the best of their knowledge), currently the most scalable deep-learning implementation in the world. The work described in the paper, Deep Learning at 15PF: Supervised and Semi-Supervised Classification for Scientific Data1, reported that a Cray XC40 system with a configuration of 9,600 self-hosted 1.4GHz Intel Xeon Phi Processor 7250 based nodes achieved a peak rate between 11.73 and 15.07 petaflops (single-precision) and an average sustained performance of 11.41 to 13.47 petaflops when training on physics and climate based data sets using Lawrence Berkeley National Laboratory’s (Berkeley Lab) NERSC (National Energy Research Scientific Computing Center) Cori Phase-II supercomputer. The group utilized an amalgamation of Intel Caffe, Intel Math Kernel Library (Intel MKL), and Intel Machine Learning Scaling Library (Intel MLSL) software to achieve this scalability and performance.2

Along with scalability, Joe Curley, Intel’s senior director of HPC platform and ecosystem enabling, highlights the scientific accomplishments this level of performance brings to deep learning researchers, and data-intensive scientific communities such as climate and High Energy Physics (HEP). He also pointed out how the results further establish the deep learning performance capability of the Intel Xeon Phi processor based computational nodes in the Cori supercomputer. “These were not just a set of heroic runs, they have solved real problems at the scale of a top five supercomputer using new methods,” Curley said.

Advancing Deep Learning at Scale

Prabhat, Data and Analytics Group Lead at NERSC, Berkeley Lab, emphasized that this performance and scalability result was very much a collaborative effort that: (A) utilized a neural network update scheme by Christopher Ré’s group (Department of Computer Science at Stanford University), (B) created a software infrastructure by the Parallel Computing Lab, Intel MKL and Intel MLSL product teams at Intel, and (C) leveraged the world-class people and hardware resources at NERSC.

Overall, Curley observes that the collaboration reported “reasonably good scaling performance” as the 9,600 node cluster delivered an approximate 7,205x speedup. (Perfect scaling would have delivered a 9,600x speedup.) Curly is excited by the potential of this early work stating, “Opportunities exist to improve performance and scaling in either future runs, or in the course of solving new problems. This really was an amazing early result on a fairly new machine.”

Algorithmic advances for deep learning scalability

The update scheme by Ré allows both the synchronous and asynchronous updates of the ANN (Artificial Neural Network) parameters.

Conceptually asynchronous updates (and asynchronous architectures in general) provide the ability to scale to large numbers of nodes by removing synchronization barriers. This lock-free approach allows for faster model updates, but can require more updates to yield an equally good final model.  Thus asynchronous updates can make the training process run longer – meaning it can take longer to converge. The thought process behind the use of asynchronous updates is that the extra computational nodes add enough parallelism (and hence can deliver greater performance) to overcome the potentially slower convergence behavior and thus deliver an overall faster time-to-model. Failure to converge to a good solution is also a possibility although Ioannis Mitliagkas, former Postdoctoral scholar at Stanford and currently Assistant Professor at the University of Montreal, observes from both classical and modern results on asynchrony that, “on well-behaved objectives, failure to converge implies a mis-tuned system.” Thus tuning is critical. However, it is worth noting that on deep learning objectives no system, synchronous or asynchronous is guaranteed to converge to a good solution.

The asynchronous deep learning architecture used in the paper is illustrated below.

Each node works on its own iteration (mini-batch) and produces independent updates to the model. Those updates are sent to a central parameter store called the parameter server (noted as PS in the figure), which applies the updates to the model in the order they are received. After each setup, the PS sends the new model, back to the worker where the update originated.

Asynchronous systems do not suffer from straggler effects and are not limited by the total batch size in the same way that synchronous systems are, an important property at scale.

Figure 1: Example synchronous and asynchronous architectures

Staleness is the reason that asynchronous systems may need more iterations to converge to a solution. Said another way, they have worse statistical efficiency.

In contrast, the reduction operation used by synchronous training introduces an O(log(#Nodes)) runtime growth. Further, it make the training susceptible to jitter (e.g. the “straggler effect”) as the computation can be rate limited by the slowest node in the system during each iteration of the training procedure.3 The straggler effect occurs when any delay on any node exceeds the ability of the implementation of the reduction operation to hide latency. Achieving low latency is an important goal for developers of products such as Intel Omni-Path Architecture (Intel OPA) MPI libraries. In addition, using too many nodes during training can reduce the number of examples per node (e.g. the mini-batch size) to the point of reduced node efficiency. Thus the authors note in their paper that synchronous training can potentially deliver worse hardware efficiency.

The trade-off between statistical efficiency vs. hardware efficiency suggested a third kind of architecture to the paper authors, which they call a hybrid system.

Mitliagkas points out, “Synchronous systems are the classic, straightforward approach, and it has some good and bad attributes. The bad attributes (straggler effect and susceptibility to slow nodes, and huge effective mini-batches at scale) motivate asynchronous systems. Those have different strengths and weaknesses motivating a tradeoff. Hybrid systems give you control of the tradeoff.”

In the hybrid approach, worker nodes coalesce into separate, synchronous compute groups where the workers split a mini-batch quantity of work among themselves to produce a single update to the model. There is no synchronization across compute groups so they are able to run asynchronously. The hybrid architecture used in the paper is illustrated below.

Figure 2: Hybrid architecture example

The authors report they observed better scaling for their hybrid asynchronous updates over synchronous configurations due to reduced straggler effects. “This has been a big engineering effort,” Mitliagkas notes, “On the Stanford side, this would not have been possible without the engineering skills and hard work of Jian Zhang.” He also points out that in large-scale HPC runs, say 10,000 nodes, the likelihood that there will be some ’slow’ nodes is significant. For this reason, synchronous system’s performance can be really unpredictable. On the other hand, asynchronous systems degrade more gracefully: a slow node only affects a single synchronous group. Results in their paper show the hybrid method performing 1.66x better than the best synchronous run, and about 10x better than the worst synchronous run.

Figure 3: Training losses vs wall clock time for HEP on 1K nodes. Comparing synchronous configuration to 2, 4, and 8 groups

The weak scaling plots below, where the amount of work per node is kept constant, show that the scalability of the system can vary with the task.

Mitliagkas emphatically states, “People typically report weak scaling, because strong scaling is hard.” He continues, “For machine learning systems, strong scaling (keeping the total amount of work constant) is more representative of actual performance.”4 He reinforces his point by stating that, “synchronous approaches can only scale up to the size of the mini-batch.”

Figure 5: Strong scaling results for synchronous and hybrid approaches (batch size = 2048 per synchronous group).

Finding a good configuration is not an easy task. Recognizing this complexity, Thorsten Kurth, HPC consultant at NERSC, notes: “it is unreasonable to expect scientists to be conversant in the art of hyper-parameter tuning. Hybrid schemes, like the one presented in this paper, add an extra parameter to be tuned, which stresses the need for principled momentum tuning approaches, an active area of research (eg. YellowFin). With hyper-parameter tuning taken care of, higher-level libraries such as Spearmint can be used for automating the search for network architectures.”

Reduced precision

The results presented in the paper were based on 32-bit, single-precision arithmetic because there are open questions regarding the use of reduced precision for training. Specifically, Thorsten observes, “more aggressive optimizations involving computing in low-precision and communicating high order bits of weight updates are poorly understood with regards to their implications for classification and regression accuracy for scientific datasets” [Italic emphasis by the authors]. He concludes, “The field of Deep Learning is evolving rapidly, and we look forward to adopting advances in the near future.”

Asynchronous momentum and tuning the convergence rate

Mitliagkas notes that in the past industry groups have reported very good performance from asynchronous systems in commercial applications. In building on that work, his post-doc showed that asynchronous behavior effectively introduces a momentum term into the optimization. However, this momentum needs to be tuned dynamically to take into account past history as it can have significant impact on the convergence rate. Mitliagkas makes the point that the hybrid architecture means the user does not have to choose to run entirely in synchronous or asynchronous mode but can tune the hybrid method to best fit the machine and problem. In his opinion, this flexibility makes hybrid systems much more useful for general users and accounts for issues general users have had in the past with purely asynchronous systems. More information can be found in the Omnivore system. Mitliagkas and Zhang’s newer work, Yellowfin, automates the process of automatic tuning even more as described in this Stanford blog post.

Intel MLSL

Nadathur Satish, Research Scientist at Intel, noted, “The Intel team performed a significant amount of work to extend the Intel MLSL library to support the hybrid asynchronous code for this paper.”  Specifically the Intel MLSL team added the ability to instantiate multiple synchronous groups and interface it with a parameter server. The Intel MLSL was initially designed to provide scalable behavior for deep learning synchronous codes. Satish noted that scalability is key to advancing the field of deep learning.

Deep Learning for Science

Prabhat notes, “For this paper, it was critical for us to demonstrate the viability of scaling Deep Learning for real scientific applications, in contrast to ImageNet. We have numerous scientific workloads at NERSC that are currently using Deep Learning, this work sets a high bar for HPC systems. Thorsten Kurth and Wahid Bhimiji chose to demonstrate the efficacy of training on simulated HEP data at scale to learn how to separate the rare signals of new particles from background events – without human intervention. Improvements in identifying these new particles could aid discoveries that might redefine our understanding of the fundamental nature of our universe. Similarly, Evan Racah and I took on the problem of identifying features in climate data. Automatically extracting such patterns will enable us to better characterize changes in frequency and intensity of extreme weather under climate change.”

High-Energy Physics

For the paper, the team used data from an LHC simulator to identify massive supersymmetric particles in multi-jet final states as they should appear in real-life at the LHC. This required training on 10 million events contained in roughly 10 terabytes of data. The team verified the trained ANN had similar baseline performance as reported by the ATLAS collaboration. Kurth summarize the results by saying, “The capability to achieve high sensitivities to new-physics signals from classification on low-level detector quantities, without the need to design, reconstruct, or tune, high-level features offers considerable potential for enabling new-physics discoveries in future HEP analyses.”


For the task of detecting extreme weather patterns, the team developed a novel semi-supervised architecture that uses an auto-encoder to capture various patterns in the dataset and simultaneously asks the network to predict bounding boxes for known patterns (such as hurricanes , extra-tropical cyclones and atmospheric rivers). Figure 6 highlights predictions from the network: Black bounding boxes show ground truth and Red boxes are predictions by the network. Note the strong overlap in all but one example.

Figure 6: Results from plotting the network’s most confident (>95%) box predictions on an image for integrated water vapor (TMQ) from the test set for the climate problem.

Single-node Cori Intel Xeon Phi processor performance

The core computation in deep learning is dense linear algebra; specifically matrix multiply and convolution operations. The authors observe that the hardware efficiency of these kernels heavily depends on input data sizes and model parameters (weight matrix dimensions, number of convolutions, convolution strides, padding, and etcetera).

As a reference, they note that DeepBench from Baidu captures the best known performance of deep learning kernels with varied input sizes and model parameters on NVIDIA GPUs and Intel Xeon Phi processors. DeepBench results show that performance on all architectures can be as high as 75-80% of peak flops for some kernels,  and as low as 20-30% as the minibatch size decreases (determined by dimension ’N’ for matrix multiply and convolutions).

With those caveats in place, the team reports that an Intel Xeon Phi processor 7250 can deliver a 2.09 teraflops overall flop rate for the climate network and 1.90 teraflops for the HEP network. For both networks, most of the runtime is spent in convolutional layers, which can obtain between 3.5 teraflops for layers with many channels, and around 1.25 teraflops on the initial layers with very few channels.1


Work by a number of researchers around the world is demonstrating both the usefulness, performance, and scalability of deep learning on very large, data intensive workloads. High per node performance is simply not enough as people apply deep learning technology to increasingly complex problems. Succinctly, the more complex the problem, the larger the data set that is required to adequately represent the problem space during training. With scalable algorithms, researchers can train on orders of magnitude larger data sets and achieve thousands of times faster time-to-model performance which, in turn, means they can address more complex (and potentially more valuable) problems. Further, this collaborative effort by Intel, NERSC, and Stanford shows that deep learning is a candidate exascale workload that can help realize the tremendous potential of computing at the exascale.

About the Author

Rob Farber is a global technology consultant and author with an extensive background in HPC and in developing machine learning technology that he applies at national labs and commercial organizations. Rob can be reached at


2 NERSC is a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231.

3 I recommend the paper “The case of the missing supercomputer performance” to better understand the impact of jitter at scale in HPC systems.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

GlobalFoundries Puts Wind in AMD’s Sails with 12nm FinFET

September 24, 2017

From its annual tech conference last week (Sept. 13), where GlobalFoundries welcomed more than 600 semiconductor professionals (reaching the Santa Clara venue’s max capacity and doubling 2016 attendee numbers), the one Read more…

By Tiffany Trader

Machine Learning at HPC User Forum: Drilling into Specific Use Cases

September 22, 2017

The 66th HPC User Forum held September 5-7, in Milwaukee, Wisconsin, at the elegant and historic Pfister Hotel, highlighting the 1893 Victorian décor and art of “The Grand Hotel Of The West,” contrasted nicely with Read more…

By Arno Kolster

Google Cloud Makes Good on Promise to Add Nvidia P100 GPUs

September 21, 2017

Google has taken down the notice on its cloud platform website that says Nvidia Tesla P100s are “coming soon.” That's because the search giant has announced the beta launch of the high-end P100 Nvidia Tesla GPUs on t Read more…

By George Leopold

HPE Extreme Performance Solutions

HPE Prepares Customers for Success with the HPC Software Portfolio

High performance computing (HPC) software is key to harnessing the full power of HPC environments. Development and management tools enable IT departments to streamline installation and maintenance of their systems as well as create, optimize, and run their HPC applications. Read more…

Cray Wins $48M Supercomputer Contract from KISTI

September 21, 2017

It was a good day for Cray which won a $48 million contract from the Korea Institute of Science and Technology Information (KISTI) for a 128-rack CS500 cluster supercomputer. The new system, equipped with Intel Xeon Scal Read more…

By John Russell

GlobalFoundries Puts Wind in AMD’s Sails with 12nm FinFET

September 24, 2017

From its annual tech conference last week (Sept. 13), where GlobalFoundries welcomed more than 600 semiconductor professionals (reaching the Santa Clara venue Read more…

By Tiffany Trader

Machine Learning at HPC User Forum: Drilling into Specific Use Cases

September 22, 2017

The 66th HPC User Forum held September 5-7, in Milwaukee, Wisconsin, at the elegant and historic Pfister Hotel, highlighting the 1893 Victorian décor and art o Read more…

By Arno Kolster

Stanford University and UberCloud Achieve Breakthrough in Living Heart Simulations

September 21, 2017

Cardiac arrhythmia can be an undesirable and potentially lethal side effect of drugs. During this condition, the electrical activity of the heart turns chaotic, Read more…

By Wolfgang Gentzsch, UberCloud, and Francisco Sahli, Stanford University

PNNL’s Center for Advanced Tech Evaluation Seeks Wider HPC Community Ties

September 21, 2017

Two years ago the Department of Energy established the Center for Advanced Technology Evaluation (CENATE) at Pacific Northwest National Laboratory (PNNL). CENAT Read more…

By John Russell

Exascale Computing Project Names Doug Kothe as Director

September 20, 2017

The Department of Energy’s Exascale Computing Project (ECP) has named Doug Kothe as its new director effective October 1. He replaces Paul Messina, who is stepping down after two years to return to Argonne National Laboratory. Kothe is a 32-year veteran of DOE’s National Laboratory System. Read more…

Takeaways from the Milwaukee HPC User Forum

September 19, 2017

Milwaukee’s elegant Pfister Hotel hosted approximately 100 attendees for the 66th HPC User Forum (September 5-7, 2017). In the original home city of Pabst Blu Read more…

By Merle Giles

Kathy Yelick Charts the Promise and Progress of Exascale Science

September 15, 2017

On Friday, Sept. 8, Kathy Yelick of Lawrence Berkeley National Laboratory and the University of California, Berkeley, delivered the keynote address on “Breakthrough Science at the Exascale” at the ACM Europe Conference in Barcelona. In conjunction with her presentation, Yelick agreed to a short Q&A discussion with HPCwire. Read more…

By Tiffany Trader

DARPA Pledges Another $300 Million for Post-Moore’s Readiness

September 14, 2017

The Defense Advanced Research Projects Agency (DARPA) launched a giant funding effort to ensure the United States can sustain the pace of electronic innovation vital to both a flourishing economy and a secure military. Under the banner of the Electronics Resurgence Initiative (ERI), some $500-$800 million will be invested in post-Moore’s Law technologies. Read more…

By Tiffany Trader

How ‘Knights Mill’ Gets Its Deep Learning Flops

June 22, 2017

Intel, the subject of much speculation regarding the delayed, rewritten or potentially canceled “Aurora” contract (the Argonne Lab part of the CORAL “ Read more…

By Tiffany Trader

Reinders: “AVX-512 May Be a Hidden Gem” in Intel Xeon Scalable Processors

June 29, 2017

Imagine if we could use vector processing on something other than just floating point problems.  Today, GPUs and CPUs work tirelessly to accelerate algorithms Read more…

By James Reinders

NERSC Scales Scientific Deep Learning to 15 Petaflops

August 28, 2017

A collaborative effort between Intel, NERSC and Stanford has delivered the first 15-petaflops deep learning software running on HPC platforms and is, according Read more…

By Rob Farber

Oracle Layoffs Reportedly Hit SPARC and Solaris Hard

September 7, 2017

Oracle’s latest layoffs have many wondering if this is the end of the line for the SPARC processor and Solaris OS development. As reported by multiple sources Read more…

By John Russell

Six Exascale PathForward Vendors Selected; DoE Providing $258M

June 15, 2017

The much-anticipated PathForward awards for hardware R&D in support of the Exascale Computing Project were announced today with six vendors selected – AMD Read more…

By John Russell

Top500 Results: Latest List Trends and What’s in Store

June 19, 2017

Greetings from Frankfurt and the 2017 International Supercomputing Conference where the latest Top500 list has just been revealed. Although there were no major Read more…

By Tiffany Trader

IBM Clears Path to 5nm with Silicon Nanosheets

June 5, 2017

Two years since announcing the industry’s first 7nm node test chip, IBM and its research alliance partners GlobalFoundries and Samsung have developed a proces Read more…

By Tiffany Trader

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Leading Solution Providers

Graphcore Readies Launch of 16nm Colossus-IPU Chip

July 20, 2017

A second $30 million funding round for U.K. AI chip developer Graphcore sets up the company to go to market with its “intelligent processing unit” (IPU) in Read more…

By Tiffany Trader

Google Releases Deeplearn.js to Further Democratize Machine Learning

August 17, 2017

Spreading the use of machine learning tools is one of the goals of Google’s PAIR (People + AI Research) initiative, which was introduced in early July. Last w Read more…

By John Russell

Russian Researchers Claim First Quantum-Safe Blockchain

May 25, 2017

The Russian Quantum Center today announced it has overcome the threat of quantum cryptography by creating the first quantum-safe blockchain, securing cryptocurrencies like Bitcoin, along with classified government communications and other sensitive digital transfers. Read more…

By Doug Black

EU Funds 20 Million Euro ARM+FPGA Exascale Project

September 7, 2017

At the Barcelona Supercomputer Centre on Wednesday (Sept. 6), 16 partners gathered to launch the EuroEXA project, which invests €20 million over three-and-a-half years into exascale-focused research and development. Led by the Horizon 2020 program, EuroEXA picks up the banner of a triad of partner projects — ExaNeSt, EcoScale and ExaNoDe — building on their work... Read more…

By Tiffany Trader

Google Debuts TPU v2 and will Add to Google Cloud

May 25, 2017

Not long after stirring attention in the deep learning/AI community by revealing the details of its Tensor Processing Unit (TPU), Google last week announced the Read more…

By John Russell

Amazon Debuts New AMD-based GPU Instances for Graphics Acceleration

September 12, 2017

Last week Amazon Web Services (AWS) streaming service, AppStream 2.0, introduced a new GPU instance called Graphics Design intended to accelerate graphics. The Read more…

By John Russell

Cray Moves to Acquire the Seagate ClusterStor Line

July 28, 2017

This week Cray announced that it is picking up Seagate's ClusterStor HPC storage array business for an undisclosed sum. "In short we're effectively transitioning the bulk of the ClusterStor product line to Cray," said CEO Peter Ungaro. Read more…

By Tiffany Trader

GlobalFoundries: 7nm Chips Coming in 2018, EUV in 2019

June 13, 2017

GlobalFoundries has formally announced that its 7nm technology is ready for customer engagement with product tape outs expected for the first half of 2018. The Read more…

By Tiffany Trader

  • arrow
  • Click Here for More Headlines
  • arrow
Share This