Deep Learning at 15 PFlops Enables Training for Extreme Weather Identification at Scale

By Rob Farber

March 19, 2018

Petaflop per second deep learning training performance on the NERSC (National Energy Research Scientific Computing Center) Cori supercomputer has given climate scientists the ability to use machine learning to identify extreme weather events in huge climate simulation datasets. Predictive accuracies ranging from 89.4% to as high as 99.1% show that trained deep learning neural networks (DNNs) can identify weather fronts, tropical cyclones, and long narrow air flows that transport water vapor from the tropics called atmospheric rivers. As with image recognition, Michael Wehner (senior staff scientist, LBNL) noted they found the machine learning output outperforms humans. [i]

The strong relationship between ground truth and the neural network prediction can be seen in the classification plus regression results reported by Wehner at the recent Intel Developer Conference in Denver, Colorado.

Figure 1: Relation between ground truth (green boxes) and classification plus regression results (red boxes) of the DNN trained to recognize atmospheric phenomena (results courtesy NERSC)

When explaining the importance of this work, Wehner believes that the big impact lies in assessing the impact of climate change as exemplified by the recent painful experiences of hurricanes Harvey (tied with hurricane Katrina as the costliest tropical cyclone on record), Irma (the strongest storm on record to exist in the open Atlantic region), and Maria (regarded as the worst natural disaster on record in Dominica and Puerto Rico).

AI needed to evaluate faster and more accurate climate models

The performance of modern leadership class supercomputers like the CPU-based NERSC Cori system provides scientists an extraordinary tool to model climate change significantly faster and far more accurately than was possible on previous generation supercomputers. For example, Wehner believes that simulated storms can run 300x to 10,000x faster than real time[ii]. This meets the needs of climate scientists who need to run many-century long simulations to evaluate the impact of climate change far into the future.

Wehner pointed out that while humans can (and do) perform well in identifying and tracking extreme weather events in real time, they simply cannot keep up when climate models run two to five orders of magnitude faster. Thus machine learning has to be used to identify and track extreme weather events. Further, these machine learning based results can be used to validate the climate models so we have confidence in the future predictions of these models.

More powerful machines also means that scientists are able to run climate models with higher spatial resolution.

Utilizing higher spatial resolution models, Wehner points is necessary because only high resolution climate models (<25Km) can realistically reproduce extreme storms[iii]. The challenge for climate and computational scientists is that these higher spatial resolution models generate tens to hundreds of terabytes of data, thus increasing the challenges of evaluating the output of these models.

Visually, the benefits of higher spatial resolution models can be easily seen in the comparative images between a 200 km and 25 km spatial resolution simulation shown below.

Figure 2: Comparative results showing the additional detail that is modeled by a 25 Km spatial resolution model as opposed to a 200 Km model. (Image courtesy NERSC)

Valuable insight as well as confirmation of human intuition can be obtained from these long-term high resolution climate simulations. The following figure, for example, represents an assessment of what will happen to the number and intensity of hurricanes as the climate warms. The chart reflects the results of 75 million hours of climate simulation on the Cori supercomputer. The conclusion from this study is that a world that is 1.5oC and 2oC warmer that today will experience more frequent and intense hurricanes. [iv]

Figure 3: Projected increase in category 5 hurricanes and decrease in weaker tropical storms due to climate change. Number of storms is measured on the y-axis (Image courtesy NERSC)

Scaling deep learning from images to climate

Reflecting the thought behind the research, Prabhat, (Director Big Data Center, NERSC) observes that identifying phenomena in climate data is analogous to commercial vision applications as shown in the following slide from his Intel Developer Conference keynote presentation.

Figure 4: Intuition showing that conventional DNNs could potentially recognize atmospheric phenomena (Image courtesy NERSC)

Prabhat points out that initial supervised training results show that this analogy is correct in that machine learning was able to train and recognize each of three desired atmospheric phenomena with high accuracy.

Figure 5: Initial supervised learning results [v] (Courtesy NERSC).
Researchers from MILA, NERSC and Microsoft teamed up to create[vi] a novel semi-supervised convolutional DNN architecture that was able to do the work of all three individual supervised DNNs at the same time. Essentially, this novel neural network finds the bounding box size and location when it classifies the atmospheric phenomena. Further, the neural network also associates a probability with the classification. The high correspondence between prediction and ground truth are shown in Figure 1. Prabhat notes that refining the size of the bounding box is a work in progress.

Figure 6: Novel semi-supervised learning architecture used to classify atmospheric phenomena and demonstrate petascale training performance on the NERSC Intel Xeon Phi computational nodes (Image courtesy NERSC)

Training DNNs at 15 PF/s and with strong scaling

This climate problem was used in a collaborative effort between Intel, NERSC and Stanford to demonstrate the fastest and most scalable deep-learning training implementation in the world according to the authors of the paper Deep Learning at 15PF: Supervised and Semi-Supervised Classification for Scientific Data. More specifically, the authors report that a configuration of 9600 self-hosted 1.4GHz Intel Xeon Phi Processor 7250 based nodes achieved a peak rate between 11.73 and 15.07 PF/s and an average sustained performance of 11.41 to 13.47 PF/s.

The following strong scaling plots (below) show that the hybrid approach advocated by Kurth, et. al. scales well to run on the thousands of Cori nodes. Ioannis Mitliagkas (former Postdoctoral scholar at Stanford and currently Assistant Professor at the University of Montreal) emphatically states, “People typically report weak scaling, because strong scaling is hard.” He continues, “For machine learning systems, strong scaling (keeping the total amount of work constant) is more representative of actual performance.”[vii]

Figure 7: Strong scaling results for synchronous and hybrid approaches (batch size = 2048 per synchronous group).


Powerful leadership class supercomputers like the CPU-based Cori supercomputer have made fast, accurate global climate simulations possible. Innovations such as the petascale capable hybrid machine learning technique pioneered by Intel, NERSC and Stanford means those same machines can also train DNNs to evaluate the tens to hundreds of terabytes of data created by these faster and more accurate climate simulations.

About the Author

Rob Farber is a global technology consultant and author with an extensive background in HPC and in developing machine learning technology that he applies at national labs and commercial organizations. Rob can be reached at [email protected].


[ii] ibid

[iii] ibid

[iv] ibid



[vii] The authors point out that weak scaling is easy to achieve! However, it is not representative of true performance as it increases the total batch size as the scale increases. They note that most people who claim to be doing large-scale ML/DL report weak scaling. Strong scaling is more representative of true performance and it is hard to achieve. In strong scaling the batch size used per synchronous group is fixed: this means that no wasted computation happens at large scales. See Ameet Talwalkar’s paper and online tool (Paleo) for a model of DL system performance and good demonstrations on how strong scaling suffers for real systems. In terms of the hardware efficiency, hybrid systems can deliver superior strong scaling (this is what the top plots are showing).

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

NREL ‘Eagle’ Supercomputer to Advance Energy Tech R&D

August 14, 2018

The U.S. Department of Energy (DOE) National Renewable Energy Laboratory (NREL) has contracted with Hewlett Packard Enterprise (HPE) for a new 8-petaflops (peak) supercomputer that will be used to advance early-stage R&a Read more…

By Tiffany Trader

Training Time Slashed for Deep Learning

August 14, 2018, an organization offering free courses on deep learning, claimed a new speed record for training a popular image database using Nvidia GPUs running on public cloud infrastructure. A pair of researchers trained Read more…

By George Leopold

CERN Project Sees Orders-of-Magnitude Speedup with AI Approach

August 14, 2018

An award-winning effort at CERN has demonstrated potential to significantly change how the physics based modeling and simulation communities view machine learning. The CERN team demonstrated that AI-based models have the Read more…

By Rob Farber

HPE Extreme Performance Solutions

Introducing the First Integrated System Management Software for HPC Clusters from HPE

How do you manage your complex, growing cluster environments? Answer that big challenge with the new HPC cluster management solution: HPE Performance Cluster Manager. Read more…

IBM Accelerated Insights

Super Problem Solving

You might think that tackling the world’s toughest problems is a job only for superheroes, but at special places such as the Oak Ridge National Laboratory, supercomputers are the real heroes. Read more…

Rigetti Eyes Scaling with 128-Qubit Architecture

August 10, 2018

Rigetti Computing plans to build a 128-qubit quantum computer based on an equivalent quantum processor that leverages emerging hybrid computing algorithms used to test programs and potential applications. Founded in 2 Read more…

By George Leopold

NREL ‘Eagle’ Supercomputer to Advance Energy Tech R&D

August 14, 2018

The U.S. Department of Energy (DOE) National Renewable Energy Laboratory (NREL) has contracted with Hewlett Packard Enterprise (HPE) for a new 8-petaflops (peak Read more…

By Tiffany Trader

CERN Project Sees Orders-of-Magnitude Speedup with AI Approach

August 14, 2018

An award-winning effort at CERN has demonstrated potential to significantly change how the physics based modeling and simulation communities view machine learni Read more…

By Rob Farber

Intel Announces Cooper Lake, Advances AI Strategy

August 9, 2018

Intel's chief datacenter exec Navin Shenoy kicked off the company's Data-Centric Innovation Summit Wednesday, the day-long program devoted to Intel's datacenter Read more…

By Tiffany Trader

SLATE Update: Making Math Libraries Exascale-ready

August 9, 2018

Practically-speaking, achieving exascale computing requires enabling HPC software to effectively use accelerators – mostly GPUs at present – and that remain Read more…

By John Russell

Summertime in Washington: Some Unexpected Advanced Computing News

August 8, 2018

Summertime in Washington DC is known for its heat and humidity. That is why most people get away to either the mountains or the seashore and things slow down. H Read more…

By Alex R. Larzelere

NSF Invests $15 Million in Quantum STAQ

August 7, 2018

Quantum computing development is in full ascent as global backers aim to transcend the limitations of classical computing by leveraging the magical-seeming prop Read more…

By Tiffany Trader

By the Numbers: Cray Would Like Exascale to Be the Icing on the Cake

August 1, 2018

On its earnings call held for investors yesterday, Cray gave an accounting for its latest quarterly financials, offered future guidance and provided an update o Read more…

By Tiffany Trader

Google is First Partner in NIH’s STRIDES Effort to Speed Discovery in the Cloud

July 31, 2018

The National Institutes of Health, with the help of Google, last week launched STRIDES - Science and Technology Research Infrastructure for Discovery, Experimen Read more…

By John Russell

Leading Solution Providers

SC17 Booth Video Tours Playlist

Altair @ SC17


AMD @ SC17


ASRock Rack @ SC17

ASRock Rack



DDN Storage @ SC17

DDN Storage

Huawei @ SC17


IBM @ SC17


IBM Power Systems @ SC17

IBM Power Systems

Intel @ SC17


Lenovo @ SC17


Mellanox Technologies @ SC17

Mellanox Technologies

Microsoft @ SC17


Penguin Computing @ SC17

Penguin Computing

Pure Storage @ SC17

Pure Storage

Supericro @ SC17


Tyan @ SC17


Univa @ SC17


  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This