Hurricane Force Supercomputing: Petascale Simulations of Sandy

By Peter Johnsen, Mark Straka, Melvyn Shapiro, Alan Norton & Tom Galerneau

November 14, 2013

The devastation incurred by the landfall of Hurricane Sandy on the northeast coast of the United States just over one year ago exemplifies the need for further advances in accuracy and reliability in numerical weather prediction.  High resolution numerical weather simulations carried out on hundreds of thousands of processors on the largest supercomputers are providing these very insights.

The National Center for Atmospheric Research (NCAR) Weather Research and Forecasting (WRF) model has been employed on the largest yet storm prediction model using real data of over 4 billion points to simulate the landfall of Hurricane Sandy. Using an unprecedented 13,680 nodes (437,760 cores) of the Cray XE6 Blue Waters supercomputer at the National Center for Supercomputing Applications  at the University of Illinois, the team of Peter Johnsen from Cray, Inc., Mark Straka from NCSA, and Mel Shapiro, Alan Norton, and Tom Galarneau from NCAR achieved an unprecedented level of performance for any weather model.  The model used approximately 4 billion grid points at an extremely fine resolution of 500 meters.  Forecast data was written and analyzed by the NCAR team members using the NCAR VAPOR visualization suite.

The landfall of Hurricane Sandy along the New Jersey shoreline late on October 30th, 2012 produced a catastrophic storm surge extending from New Jersey to Rhode Island. The research highlighted here demonstrates the capability of the NCSA/Cray Blue Waters supercomputer to conduct a cloud-resolving WRF-ARW simulation of an intense cyclone over a relatively large domain at a very-fine spatial resolution.

The Blue Waters system is a Cray XE/XK hybrid machine composed of 362,240 AMD 6276 “Interlagos” processors and 4224 NVIDIA GK110 Kepler accelerators all connected by the Cray Gemini 3-D (24^3) torus interconnect. It provides sustained performance of 1 petaflop on a range of real-world science and engineering applications. Our motivation was to reduce time to solution as much as was under our control without major source code restructuring. The WRF version 3.3.1 source code was modified from the public distribution chiefly with concerns for reducing the I/O burden per MPI task and limiting the necessary information to a single MPI rank.

Topology Considerations Are Vital

Domain configuration and process layout using MPI rank ordering features of the Cray XE6 job scheduler (ALPS) form a cornerstone in efficiently using the XE6 3D torus interconnect and allowing WRF to scale this successfully.  We used the Cray grid_order utility to generate improved placement of the ranks for the primary communication pattern in the WRF solver, which is nearest neighbor halo exchanges. Reducing the number of neighbors communicating off-node is the primary goal. Using an alternate placement allows us to get 3 communication partners for most MPI ranks on the same node, instead of only 2, as would be with the default placement.  At very high scales, this strategy improves overall WRF performance by 18% or more.

We found the most effective way to run WRF on the AMD Bulldozer core-modules was to exploit WRF’s “hybrid” MPI/OpenMP structure, utilizing 2 OpenMP threads per MPI rank.  This puts 16 MPI ranks on each XE6 node.

The optimized placement we’ve employed also has the benefit of sending smaller east-west direction exchanges off-node and keeping as many larger north-south messages on-node as possible, resulting in 75% fewer bytes being sent over the network. We verified empirically the long-known tactic of decomposing WRF’s domain with many fewer MPI ranks in the X direction than the Y, as this leads to longer vectors on the inner compute loops.

Our simulations yielded an average Tflop count of 32.454 Tflops per second, per simulation time step. Parallel efficiency was still above 60% even on 13,680 XE6 nodes. Over 12 million off-node halo exchange messages totaling 280 GB were processed every WRF time step.

I/O Considerations at Scale

On the Blue Waters system, the Lustre file system was used for all file activity.

Two techniques were used to handle the large I/O requirements for the Sandy simulation –

  1. Parallel NetCDF (PnetCDF), jointly developed by Northwestern University and Argonne National Laboratory, was used where practical.  The MPICH library from Cray has a tuned MPI-IO implementation that aligns parallel I/O with the Lustre file system.  This format is required when post-processing tools are used.
  2. WRF has a multi-file option where each subdomain, or MPI rank, reads and writes unique files.  This was used for very large restart files and some of the pre-processing steps.  The Blue Waters Lustre file system was able to open and read 145,920 restart files in 18 seconds for a 4560 node case.

Additionally, use of WRF’s auxiliary history output options to select only the output fields of greatest interest, thus reducing the volume of output considerably, was of great utility in our work.

graph_1

Scalability of Hurricane Sandy run.  Sustained performance in Tflops/second (y-axis, left) and parallel efficiency over base run on 8,192 cores (y-axis, right) are shown.

Forecast Analysis and Validation

The following figures show a comparison of the maximum radar reflectivity (a surrogate for precipitation) from the simulations at 3-km and 500-m horizontal resolution. In both simulations, a broad region of heavy precipitation is located on the west and southwest side of Sandy, and is organized in a region where warm moist northeasterly flow intersects a northwesterly surge of cold continental air (not shown).

graph_2

Comparison of (a) 3-km and (b) 500-m horizontal resolution ARW simulations of maximum radar reflectivity (shaded according to the color bar in dBZ) verifying at 1500 UTC 29 October 2012.

The 500-m simulation is superior to that at 3-km because it shows the fine-scale linear structure of the convective precipitation bands, consistent with the available observations (not shown). The next images show a zoomed-in view of maximum radar reflectivity and 300-m wind speed within the inner-core of Sandy at 1800 UTC 29 October 2012. This zoomed perspective allows for examination of the full detail of the simulation, noting that the resolution of the simulation (7000×7000 grid points) exceeds the resolution of standard computer monitors by a factor of seven. Here we note the utility of ultra-advanced computational capability to represent the full range of scales spanning the storm-scale circulations down to fine-scale turbulent motions and individual cloud and precipitation systems.

graph_3

500-m ARW simulation of (a) maximum radar reflectivity (shaded according to the color bar in dBZ) and (b) 300-m wind speed (shaded according to the color bar in m s−1) verifying at 1800 UTC 29 October 2012.

The model accuracy for predicting such key output fields as rainfall, pressures, wind speeds, and storm track was graphically validated against actual atmospheric measurements from the storm using NCAR’s VAPOR software suite. Given recent advances in accessing and displaying large volume geophysical datasets as exemplified by the VAPOR software, it is now possible to view the full temporal evolution of numerical simulations and predictions of atmospheric and other geophysical systems. Examples of the advanced visualizations of Hurricane Sandy with VAPOR can be found at:

https://www.vapor.ucar.edu/sites/default/files/movies/sandy_SC13_web_0.mp4

The results of this research will be presented at the Supercomputing conference this month in November.  See the conference agenda here:

http://sc13.supercomputing.org/schedule/event_detail.php?evid=pap255

Cutting Edge Forecasting

NOAA has initiated the ten-year Hurricane Forecast Improvement Project (HFIP), which is evaluating a variety of modeling approaches, exploring the feasibility of real-time fine-scale hurricane projections. Its enhanced Hurricane WRF model (HWRF) is already being run in real time at a somewhat smaller scale. In a collaborative effort involving NOAA’s Hurricane Research Division and Environmental Modeling Center, Cray , NCSA, and NCAR, this code is already being run on Blue Waters to conduct performance studies at scale with grid nesting never before possible. Results are already promising that coming years’ hurricane seasons will be able to incorporate much finer detailed real-time forecasts generated by these simulations. The team is also exploring high resolution simulations with the Office of Naval Research ONR using the COAMPS model.

Research Team:

Peter Johnsen is a performance engineer and meteorologist with Cray, Inc.  Peter’s expertise is optimizing environmental applications on HPC systems.

Mark Straka specializes in performance analysis of scientific applications on the Blue Waters system at the National Center for Supercomputing Applications.

Melvyn Shapiro, Alan Norton, and Thomas Galarneau are research meteorologists with the National Center for Atmospheric Research and are studying many weather phenomena, including Hurricane Sandy’s unique nature.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Fluid HPC: How Extreme-Scale Computing Should Respond to Meltdown and Spectre

February 15, 2018

The Meltdown and Spectre vulnerabilities are proving difficult to fix, and initial experiments suggest security patches will cause significant performance penalties to HPC applications. Even as these patches are rolled o Read more…

By Pete Beckman

Intel Touts Silicon Spin Qubits for Quantum Computing

February 14, 2018

Debate around what makes a good qubit and how best to manufacture them is a sprawling topic. There are many insistent voices favoring one or another approach. Referencing a paper published today in Nature, Intel has offe Read more…

By John Russell

Brookhaven Ramps Up Computing for National Security Effort

February 14, 2018

Last week, Dan Coats, the director of Director of National Intelligence for the U.S., warned the Senate Intelligence Committee that Russia was likely to meddle in the 2018 mid-term U.S. elections, much as it stands accused of doing in the 2016 Presidential election. Read more…

By John Russell

HPE Extreme Performance Solutions

Safeguard Your HPC Environment with the World’s Most Secure Industry Standard Servers

Today’s organizations operate in an environment with ever-evolving threats, and in order to protect themselves they must continuously bolster their security strategy. Hewlett Packard Enterprise (HPE) and Intel® are addressing modern security challenges with the world’s most secure industry standard servers powered by the latest generation of Intel® Xeon® Scalable processors. Read more…

AI Cloud Competition Heats Up: Google’s TPUs, Amazon Building AI Chip

February 12, 2018

Competition in the white hot AI (and public cloud) market pits Google against Amazon this week, with Google offering AI hardware on its cloud platform intended to make it easier, faster and cheaper to train and run machi Read more…

By Doug Black

Fluid HPC: How Extreme-Scale Computing Should Respond to Meltdown and Spectre

February 15, 2018

The Meltdown and Spectre vulnerabilities are proving difficult to fix, and initial experiments suggest security patches will cause significant performance penal Read more…

By Pete Beckman

Brookhaven Ramps Up Computing for National Security Effort

February 14, 2018

Last week, Dan Coats, the director of Director of National Intelligence for the U.S., warned the Senate Intelligence Committee that Russia was likely to meddle in the 2018 mid-term U.S. elections, much as it stands accused of doing in the 2016 Presidential election. Read more…

By John Russell

AI Cloud Competition Heats Up: Google’s TPUs, Amazon Building AI Chip

February 12, 2018

Competition in the white hot AI (and public cloud) market pits Google against Amazon this week, with Google offering AI hardware on its cloud platform intended Read more…

By Doug Black

Russian Nuclear Engineers Caught Cryptomining on Lab Supercomputer

February 12, 2018

Nuclear scientists working at the All-Russian Research Institute of Experimental Physics (RFNC-VNIIEF) have been arrested for using lab supercomputing resources to mine crypto-currency, according to a report in Russia’s Interfax News Agency. Read more…

By Tiffany Trader

The Food Industry’s Next Journey — from Mars to Exascale

February 12, 2018

Global food producer and one of the world's leading chocolate companies Mars Inc. has a unique perspective on the impact that exascale computing will have on the food industry. Read more…

By Scott Gibson, Oak Ridge National Laboratory

Singularity HPC Container Start-Up – Sylabs – Emerges from Stealth

February 8, 2018

The driving force behind Singularity, the popular HPC container technology, is bringing the open source platform to the enterprise with the launch of a new vent Read more…

By George Leopold

Dell EMC Debuts PowerEdge Servers with AMD EPYC Chips

February 6, 2018

AMD notched another EPYC processor win today with Dell EMC’s introduction of three PowerEdge servers (R6415, R7415, and R7425) based on the EPYC 7000-series p Read more…

By John Russell

‘Next Generation’ Universe Simulation Is Most Advanced Yet

February 5, 2018

The research group that gave us the most detailed time-lapse simulation of the universe’s evolution in 2014, spanning 13.8 billion years of cosmic evolution, is back in the spotlight with an even more advanced cosmological model that is providing new insights into how black holes influence the distribution of dark matter, how heavy elements are produced and distributed, and where magnetic fields originate. Read more…

By Tiffany Trader

Inventor Claims to Have Solved Floating Point Error Problem

January 17, 2018

"The decades-old floating point error problem has been solved," proclaims a press release from inventor Alan Jorgensen. The computer scientist has filed for and Read more…

By Tiffany Trader

Japan Unveils Quantum Neural Network

November 22, 2017

The U.S. and China are leading the race toward productive quantum computing, but it's early enough that ultimate leadership is still something of an open questi Read more…

By Tiffany Trader

AMD Showcases Growing Portfolio of EPYC and Radeon-based Systems at SC17

November 13, 2017

AMD’s charge back into HPC and the datacenter is on full display at SC17. Having launched the EPYC processor line in June along with its MI25 GPU the focus he Read more…

By John Russell

Researchers Measure Impact of ‘Meltdown’ and ‘Spectre’ Patches on HPC Workloads

January 17, 2018

Computer scientists from the Center for Computational Research, State University of New York (SUNY), University at Buffalo have examined the effect of Meltdown Read more…

By Tiffany Trader

IBM Begins Power9 Rollout with Backing from DOE, Google

December 6, 2017

After over a year of buildup, IBM is unveiling its first Power9 system based on the same architecture as the Department of Energy CORAL supercomputers, Summit a Read more…

By Tiffany Trader

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Fast Forward: Five HPC Predictions for 2018

December 21, 2017

What’s on your list of high (and low) lights for 2017? Volta 100’s arrival on the heels of the P100? Appearance, albeit late in the year, of IBM’s Power9? Read more…

By John Russell

Russian Nuclear Engineers Caught Cryptomining on Lab Supercomputer

February 12, 2018

Nuclear scientists working at the All-Russian Research Institute of Experimental Physics (RFNC-VNIIEF) have been arrested for using lab supercomputing resources to mine crypto-currency, according to a report in Russia’s Interfax News Agency. Read more…

By Tiffany Trader

Leading Solution Providers

Chip Flaws ‘Meltdown’ and ‘Spectre’ Loom Large

January 4, 2018

The HPC and wider tech community have been abuzz this week over the discovery of critical design flaws that impact virtually all contemporary microprocessors. T Read more…

By Tiffany Trader

Perspective: What Really Happened at SC17?

November 22, 2017

SC is over. Now comes the myriad of follow-ups. Inboxes are filled with templated emails from vendors and other exhibitors hoping to win a place in the post-SC thinking of booth visitors. Attendees of tutorials, workshops and other technical sessions will be inundated with requests for feedback. Read more…

By Andrew Jones

How Meltdown and Spectre Patches Will Affect HPC Workloads

January 10, 2018

There have been claims that the fixes for the Meltdown and Spectre security vulnerabilities, named the KPTI (aka KAISER) patches, are going to affect applicatio Read more…

By Rosemary Francis

GlobalFoundries, Ayar Labs Team Up to Commercialize Optical I/O

December 4, 2017

GlobalFoundries (GF) and Ayar Labs, a startup focused on using light, instead of electricity, to transfer data between chips, today announced they've entered in Read more…

By Tiffany Trader

Tensors Come of Age: Why the AI Revolution Will Help HPC

November 13, 2017

Thirty years ago, parallel computing was coming of age. A bitter battle began between stalwart vector computing supporters and advocates of various approaches to parallel computing. IBM skeptic Alan Karp, reacting to announcements of nCUBE’s 1024-microprocessor system and Thinking Machines’ 65,536-element array, made a public $100 wager that no one could get a parallel speedup of over 200 on real HPC workloads. Read more…

By John Gustafson & Lenore Mullin

Flipping the Flops and Reading the Top500 Tea Leaves

November 13, 2017

The 50th edition of the Top500 list, the biannual publication of the world’s fastest supercomputers based on public Linpack benchmarking results, was released Read more…

By Tiffany Trader

V100 Good but not Great on Select Deep Learning Aps, Says Xcelerit

November 27, 2017

Wringing optimum performance from hardware to accelerate deep learning applications is a challenge that often depends on the specific application in use. A benc Read more…

By John Russell

SC17: Singularity Preps Version 3.0, Nears 1M Containers Served Daily

November 1, 2017

Just a few months ago about half a million jobs were being run daily using Singularity containers, the LBNL-founded container platform intended for HPC. That wa Read more…

By John Russell

  • arrow
  • Click Here for More Headlines
  • arrow
Share This