Deep Learning at 15 PFlops Enables Training for Extreme Weather Identification at Scale

By Rob Farber

March 19, 2018

Petaflop per second deep learning training performance on the NERSC (National Energy Research Scientific Computing Center) Cori supercomputer has given climate scientists the ability to use machine learning to identify extreme weather events in huge climate simulation datasets. Predictive accuracies ranging from 89.4% to as high as 99.1% show that trained deep learning neural networks (DNNs) can identify weather fronts, tropical cyclones, and long narrow air flows that transport water vapor from the tropics called atmospheric rivers. As with image recognition, Michael Wehner (senior staff scientist, LBNL) noted they found the machine learning output outperforms humans. [i]

The strong relationship between ground truth and the neural network prediction can be seen in the classification plus regression results reported by Wehner at the recent Intel Developer Conference in Denver, Colorado.

Figure 1: Relation between ground truth (green boxes) and classification plus regression results (red boxes) of the DNN trained to recognize atmospheric phenomena (results courtesy NERSC)

When explaining the importance of this work, Wehner believes that the big impact lies in assessing the impact of climate change as exemplified by the recent painful experiences of hurricanes Harvey (tied with hurricane Katrina as the costliest tropical cyclone on record), Irma (the strongest storm on record to exist in the open Atlantic region), and Maria (regarded as the worst natural disaster on record in Dominica and Puerto Rico).

AI needed to evaluate faster and more accurate climate models

The performance of modern leadership class supercomputers like the CPU-based NERSC Cori system provides scientists an extraordinary tool to model climate change significantly faster and far more accurately than was possible on previous generation supercomputers. For example, Wehner believes that simulated storms can run 300x to 10,000x faster than real time[ii]. This meets the needs of climate scientists who need to run many-century long simulations to evaluate the impact of climate change far into the future.

Wehner pointed out that while humans can (and do) perform well in identifying and tracking extreme weather events in real time, they simply cannot keep up when climate models run two to five orders of magnitude faster. Thus machine learning has to be used to identify and track extreme weather events. Further, these machine learning based results can be used to validate the climate models so we have confidence in the future predictions of these models.

More powerful machines also means that scientists are able to run climate models with higher spatial resolution.

Utilizing higher spatial resolution models, Wehner points is necessary because only high resolution climate models (<25Km) can realistically reproduce extreme storms[iii]. The challenge for climate and computational scientists is that these higher spatial resolution models generate tens to hundreds of terabytes of data, thus increasing the challenges of evaluating the output of these models.

Visually, the benefits of higher spatial resolution models can be easily seen in the comparative images between a 200 km and 25 km spatial resolution simulation shown below.

Figure 2: Comparative results showing the additional detail that is modeled by a 25 Km spatial resolution model as opposed to a 200 Km model. (Image courtesy NERSC)

Valuable insight as well as confirmation of human intuition can be obtained from these long-term high resolution climate simulations. The following figure, for example, represents an assessment of what will happen to the number and intensity of hurricanes as the climate warms. The chart reflects the results of 75 million hours of climate simulation on the Cori supercomputer. The conclusion from this study is that a world that is 1.5oC and 2oC warmer that today will experience more frequent and intense hurricanes. [iv]

Figure 3: Projected increase in category 5 hurricanes and decrease in weaker tropical storms due to climate change. Number of storms is measured on the y-axis (Image courtesy NERSC)

Scaling deep learning from images to climate

Reflecting the thought behind the research, Prabhat, (Director Big Data Center, NERSC) observes that identifying phenomena in climate data is analogous to commercial vision applications as shown in the following slide from his Intel Developer Conference keynote presentation.

Figure 4: Intuition showing that conventional DNNs could potentially recognize atmospheric phenomena (Image courtesy NERSC)

Prabhat points out that initial supervised training results show that this analogy is correct in that machine learning was able to train and recognize each of three desired atmospheric phenomena with high accuracy.

Figure 5: Initial supervised learning results [v] (Courtesy NERSC).
Researchers from MILA, NERSC and Microsoft teamed up to create[vi] a novel semi-supervised convolutional DNN architecture that was able to do the work of all three individual supervised DNNs at the same time. Essentially, this novel neural network finds the bounding box size and location when it classifies the atmospheric phenomena. Further, the neural network also associates a probability with the classification. The high correspondence between prediction and ground truth are shown in Figure 1. Prabhat notes that refining the size of the bounding box is a work in progress.

Figure 6: Novel semi-supervised learning architecture used to classify atmospheric phenomena and demonstrate petascale training performance on the NERSC Intel Xeon Phi computational nodes (Image courtesy NERSC)

Training DNNs at 15 PF/s and with strong scaling

This climate problem was used in a collaborative effort between Intel, NERSC and Stanford to demonstrate the fastest and most scalable deep-learning training implementation in the world according to the authors of the paper Deep Learning at 15PF: Supervised and Semi-Supervised Classification for Scientific Data. More specifically, the authors report that a configuration of 9600 self-hosted 1.4GHz Intel Xeon Phi Processor 7250 based nodes achieved a peak rate between 11.73 and 15.07 PF/s and an average sustained performance of 11.41 to 13.47 PF/s.

The following strong scaling plots (below) show that the hybrid approach advocated by Kurth, et. al. scales well to run on the thousands of Cori nodes. Ioannis Mitliagkas (former Postdoctoral scholar at Stanford and currently Assistant Professor at the University of Montreal) emphatically states, “People typically report weak scaling, because strong scaling is hard.” He continues, “For machine learning systems, strong scaling (keeping the total amount of work constant) is more representative of actual performance.”[vii]

Figure 7: Strong scaling results for synchronous and hybrid approaches (batch size = 2048 per synchronous group).

Summary

Powerful leadership class supercomputers like the CPU-based Cori supercomputer have made fast, accurate global climate simulations possible. Innovations such as the petascale capable hybrid machine learning technique pioneered by Intel, NERSC and Stanford means those same machines can also train DNNs to evaluate the tens to hundreds of terabytes of data created by these faster and more accurate climate simulations.

About the Author

Rob Farber is a global technology consultant and author with an extensive background in HPC and in developing machine learning technology that he applies at national labs and commercial organizations. Rob can be reached at [email protected].

[i] https://software.intel.com/en-us/events/hpc-devcon/2017/keynote?multiplayer=5646473936001

[ii] ibid

[iii] ibid

[iv] ibid

[v] https://arxiv.org/abs/1605.01156

[vi] https://papers.nips.cc/paper/6932-extremeweather-a-large-scale-climate-dataset-for-semi-supervised-detection-localization-and-understanding-of-extreme-weather-events.pdf

[vii] The authors point out that weak scaling is easy to achieve! However, it is not representative of true performance as it increases the total batch size as the scale increases. They note that most people who claim to be doing large-scale ML/DL report weak scaling. Strong scaling is more representative of true performance and it is hard to achieve. In strong scaling the batch size used per synchronous group is fixed: this means that no wasted computation happens at large scales. See Ameet Talwalkar’s paper and online tool (Paleo) for a model of DL system performance and good demonstrations on how strong scaling suffers for real systems. In terms of the hardware efficiency, hybrid systems can deliver superior strong scaling (this is what the top plots are showing).

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industry updates delivered to you every week!

MLPerf Inference 4.0 Results Showcase GenAI; Nvidia Still Dominates

March 28, 2024

There were no startling surprises in the latest MLPerf Inference benchmark (4.0) results released yesterday. Two new workloads — Llama 2 and Stable Diffusion XL — were added to the benchmark suite as MLPerf continues Read more…

Q&A with Nvidia’s Chief of DGX Systems on the DGX-GB200 Rack-scale System

March 27, 2024

Pictures of Nvidia's new flagship mega-server, the DGX GB200, on the GTC show floor got favorable reactions on social media for the sheer amount of computing power it brings to artificial intelligence.  Nvidia's DGX Read more…

Call for Participation in Workshop on Potential NSF CISE Quantum Initiative

March 26, 2024

Editor’s Note: Next month there will be a workshop to discuss what a quantum initiative led by NSF’s Computer, Information Science and Engineering (CISE) directorate could entail. The details are posted below in a Ca Read more…

Waseda U. Researchers Reports New Quantum Algorithm for Speeding Optimization

March 25, 2024

Optimization problems cover a wide range of applications and are often cited as good candidates for quantum computing. However, the execution time for constrained combinatorial optimization applications on quantum device Read more…

NVLink: Faster Interconnects and Switches to Help Relieve Data Bottlenecks

March 25, 2024

Nvidia’s new Blackwell architecture may have stolen the show this week at the GPU Technology Conference in San Jose, California. But an emerging bottleneck at the network layer threatens to make bigger and brawnier pro Read more…

Who is David Blackwell?

March 22, 2024

During GTC24, co-founder and president of NVIDIA Jensen Huang unveiled the Blackwell GPU. This GPU itself is heavily optimized for AI work, boasting 192GB of HBM3E memory as well as the the ability to train 1 trillion pa Read more…

MLPerf Inference 4.0 Results Showcase GenAI; Nvidia Still Dominates

March 28, 2024

There were no startling surprises in the latest MLPerf Inference benchmark (4.0) results released yesterday. Two new workloads — Llama 2 and Stable Diffusion Read more…

Q&A with Nvidia’s Chief of DGX Systems on the DGX-GB200 Rack-scale System

March 27, 2024

Pictures of Nvidia's new flagship mega-server, the DGX GB200, on the GTC show floor got favorable reactions on social media for the sheer amount of computing po Read more…

NVLink: Faster Interconnects and Switches to Help Relieve Data Bottlenecks

March 25, 2024

Nvidia’s new Blackwell architecture may have stolen the show this week at the GPU Technology Conference in San Jose, California. But an emerging bottleneck at Read more…

Who is David Blackwell?

March 22, 2024

During GTC24, co-founder and president of NVIDIA Jensen Huang unveiled the Blackwell GPU. This GPU itself is heavily optimized for AI work, boasting 192GB of HB Read more…

Nvidia Looks to Accelerate GenAI Adoption with NIM

March 19, 2024

Today at the GPU Technology Conference, Nvidia launched a new offering aimed at helping customers quickly deploy their generative AI applications in a secure, s Read more…

The Generative AI Future Is Now, Nvidia’s Huang Says

March 19, 2024

We are in the early days of a transformative shift in how business gets done thanks to the advent of generative AI, according to Nvidia CEO and cofounder Jensen Read more…

Nvidia’s New Blackwell GPU Can Train AI Models with Trillions of Parameters

March 18, 2024

Nvidia's latest and fastest GPU, codenamed Blackwell, is here and will underpin the company's AI plans this year. The chip offers performance improvements from Read more…

Nvidia Showcases Quantum Cloud, Expanding Quantum Portfolio at GTC24

March 18, 2024

Nvidia’s barrage of quantum news at GTC24 this week includes new products, signature collaborations, and a new Nvidia Quantum Cloud for quantum developers. Wh Read more…

Alibaba Shuts Down its Quantum Computing Effort

November 30, 2023

In case you missed it, China’s e-commerce giant Alibaba has shut down its quantum computing research effort. It’s not entirely clear what drove the change. Read more…

Nvidia H100: Are 550,000 GPUs Enough for This Year?

August 17, 2023

The GPU Squeeze continues to place a premium on Nvidia H100 GPUs. In a recent Financial Times article, Nvidia reports that it expects to ship 550,000 of its lat Read more…

Shutterstock 1285747942

AMD’s Horsepower-packed MI300X GPU Beats Nvidia’s Upcoming H200

December 7, 2023

AMD and Nvidia are locked in an AI performance battle – much like the gaming GPU performance clash the companies have waged for decades. AMD has claimed it Read more…

DoD Takes a Long View of Quantum Computing

December 19, 2023

Given the large sums tied to expensive weapon systems – think $100-million-plus per F-35 fighter – it’s easy to forget the U.S. Department of Defense is a Read more…

Synopsys Eats Ansys: Does HPC Get Indigestion?

February 8, 2024

Recently, it was announced that Synopsys is buying HPC tool developer Ansys. Started in Pittsburgh, Pa., in 1970 as Swanson Analysis Systems, Inc. (SASI) by John Swanson (and eventually renamed), Ansys serves the CAE (Computer Aided Engineering)/multiphysics engineering simulation market. Read more…

Choosing the Right GPU for LLM Inference and Training

December 11, 2023

Accelerating the training and inference processes of deep learning models is crucial for unleashing their true potential and NVIDIA GPUs have emerged as a game- Read more…

Intel’s Server and PC Chip Development Will Blur After 2025

January 15, 2024

Intel's dealing with much more than chip rivals breathing down its neck; it is simultaneously integrating a bevy of new technologies such as chiplets, artificia Read more…

Baidu Exits Quantum, Closely Following Alibaba’s Earlier Move

January 5, 2024

Reuters reported this week that Baidu, China’s giant e-commerce and services provider, is exiting the quantum computing development arena. Reuters reported � Read more…

Leading Solution Providers

Contributors

Comparing NVIDIA A100 and NVIDIA L40S: Which GPU is Ideal for AI and Graphics-Intensive Workloads?

October 30, 2023

With long lead times for the NVIDIA H100 and A100 GPUs, many organizations are looking at the new NVIDIA L40S GPU, which it’s a new GPU optimized for AI and g Read more…

Shutterstock 1179408610

Google Addresses the Mysteries of Its Hypercomputer 

December 28, 2023

When Google launched its Hypercomputer earlier this month (December 2023), the first reaction was, "Say what?" It turns out that the Hypercomputer is Google's t Read more…

AMD MI3000A

How AMD May Get Across the CUDA Moat

October 5, 2023

When discussing GenAI, the term "GPU" almost always enters the conversation and the topic often moves toward performance and access. Interestingly, the word "GPU" is assumed to mean "Nvidia" products. (As an aside, the popular Nvidia hardware used in GenAI are not technically... Read more…

Shutterstock 1606064203

Meta’s Zuckerberg Puts Its AI Future in the Hands of 600,000 GPUs

January 25, 2024

In under two minutes, Meta's CEO, Mark Zuckerberg, laid out the company's AI plans, which included a plan to build an artificial intelligence system with the eq Read more…

Google Introduces ‘Hypercomputer’ to Its AI Infrastructure

December 11, 2023

Google ran out of monikers to describe its new AI system released on December 7. Supercomputer perhaps wasn't an apt description, so it settled on Hypercomputer Read more…

China Is All In on a RISC-V Future

January 8, 2024

The state of RISC-V in China was discussed in a recent report released by the Jamestown Foundation, a Washington, D.C.-based think tank. The report, entitled "E Read more…

Intel Won’t Have a Xeon Max Chip with New Emerald Rapids CPU

December 14, 2023

As expected, Intel officially announced its 5th generation Xeon server chips codenamed Emerald Rapids at an event in New York City, where the focus was really o Read more…

IBM Quantum Summit: Two New QPUs, Upgraded Qiskit, 10-year Roadmap and More

December 4, 2023

IBM kicks off its annual Quantum Summit today and will announce a broad range of advances including its much-anticipated 1121-qubit Condor QPU, a smaller 133-qu Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire