Summit Supercomputer is Already Making its Mark on Science

By John Russell

September 20, 2018

Summit, now the fastest supercomputer in the world, is quickly making its mark in science – five of the six finalists just announced for the prestigious 2018 Gordon Bell Prize used Summit in their work. That’s impressive given that Summit only began full operation in early summer. Also noteworthy is Summit’s heterogeneous architecture which leverages IBM’s Power9 CPU, Nvidia V100 GPUs, and fast interconnect technology from Mellanox to accommodate traditional simulation workloads as well as mixed-precision workloads associated with AI and data analytics.

By now, Summit needs little introduction having topped the most recent Top500 list. Located at the Oak Ridge Leadership Computing Facility (OCLF), it cost an estimated $200 million to build as part of the DoE CORAL procurement program. It’s been heralded as the world’s most powerful supercomputer at 200 petaflops theoretical peak for high-performance computing workloads and 3.3 peak exaops for emerging AI workloads. (Sierra, the similarly architected but somewhat smaller, 125 petaflops theoretical peak machine based at Lawrence Livermore National Laboratory, was also used in some of the cited research.)

“These Gordon Bell finalists are an encouraging preview of the challenges users will be able to tackle on Summit when formal allocation programs begin in 2019,” said OLCF director of science Jack Wells. “Of particular note is the system’s ability to handle large volumes of data at scale, whether that be processing and analyzing experimental data or training artificial intelligence software to carry out specialized tasks.”

Nvidia and IBM are, understandably, ecstatic over Summit’s progress.

In a Nvidia blog, product manager Geetika Gupta wrote, “The revolutionary accelerators enable multi-precision computing that fuses the highly precise calculations to tackle the challenges of high performance computing with the efficient processing required for deep learning…[H]alf of the six projects included NVIDIA researchers who were heavily involved with the code development and performance tuning.”

Dave Turek, IBM Cognitive Systems VP, said “IBM designed Summit and Sierra to be data-centric, heterogeneous systems that maximized data flow for optimal application performance. The industry-leading IO features of IBM POWER9 processors allow for data to flow in and out of Summit’s GPUs to achieve the unprecedented level of performance demonstrated by these Gordon Bell finalists.”

They can, perhaps, be forgiven a little excess enthusiasm. These machines are difficult to design and build. Clearly, Summit’s early success is more evidence that heterogeneous architectures that leverage accelerators are likely to dominate high-end computing going forward.

Here is a lightly edited excerpt from an OCLF article describing the finalists who used Summit in their research:

  • “Genomics. An ORNL team led by computational systems biologist Dan Jacobson and OLCF computational scientist Wayne Joubert that developed a genomics algorithm capable of using mixed-precision arithmetic to attain exascale speeds. On Summit, the team’s Combinatorial Metrics application achieved a peak throughput of 2.36 exaops—or 2.36 billion billion calculations per second, the fastest science application ever reported. Jacobson’s work compares genetic variations within a population to uncover hidden networks of genes that contribute to complex traits. One condition Jacobson’s team is studying is opioid addiction, which has been linked to the deaths of more than 49,000 people in the United States in 2017.
  • Earthquake Simulation. A team from the University of Tokyo led by associate professor Tsuyoshi Ichimura that applied artificial intelligence (AI) and mixed-precision arithmetic to accelerate the simulation of earthquake physics in urban environments. As cities continue to grow, preparedness and improved understanding of ground-shaking’s effects on buildings and urban infrastructure become increasingly important. On Summit, the Tokyo team expanded on its 2014 algorithm, which was also a Gordon Bell Finalist, to achieve a fourfold speedup and to couple the shaking of ground and urban structures during large earthquakes into the same simulation.
  • Extreme Weather. A Lawrence Berkeley National Laboratory-led collaboration that trained a deep neural network to identify extreme weather patterns from high-resolution climate simulations.The team, led by Berkeley data scientist Prabhat, plans to use the AI software to predict how extreme weather is likely to change in the future. By tapping into the specialized tensor cores built into Summit’s NVIDIA GPUs at scale, the Berkeley team achieved a peak performance of 1.13 exaops, the fastest deep-learning algorithm yet reported. Though the team applied its work to climate science, many of its innovations can be adapted for other deep-learning applications.
  • Materials Science. An ORNL team led by data scientist Robert Patton that scaled a deep-learning technique on Summit to produce intelligent software that can automatically identify materials’ atomic-level information from electron microscopy With advanced microscopes capable of producing hundreds of images per day, real-time feedback supplied by AI could give scientists the ability to fabricate materials at the atomic level. Scaled across 4,200 nodes, the team’s MENNDL algorithm achieved a speed of 152.5 petaflops with an estimated performance rate of 167 petaflops across the whole machine.
  • Physics. A team from Lawrence Berkeley and Lawrence Livermore National Laboratories led by physicists André Walker-Loud and Pavlos Vranas that developed improved algorithms to help scientists predict the lifetime of neutrons and answer fundamental questions about the universe. The team built upon its previous work using lattice quantum chromodynamics—a numerical method for calculating the underlying physics of the subatomic particles that make up protons and neutrons. In addition to optimized GPU software, the team developed lightweight, application-agnostic management software capable of managing hundreds of thousands of tasks. Using GPU-accelerated systems Sierra at Lawrence Livermore and the OLCF’s Summit, the team was able to start 1,056 four-node jobs on 4,224 nodes in 5 minutes, achieving a machine-to-machine speedup of factors of 10 and 15, respectively, over the OLCF’s previous leadership-class system, Titan. The achievement supplies nuclear physicists with the necessary computational power to support the experimental search for new physics.”

We’d be remiss not to mention the sixth Gordon finalist; it’s from a group of researchers from China who developed a graph processing framework (ShenTu) adapted for use on HPC resources. Here is a description of that very impressive work (ShenTu: Processing Multi-Trillion Edge Graphs on Millions of Cores in Seconds) taken from the SC18 web site.

“DescriptionGraphs are an important abstraction used in many scientific fields. With the magnitude of graph-structured data constantly increasing, effective data analytics requires efficient and scalable graph processing systems. Although HPC systems have long been used for scientific computing, people have only recently started to assess their potential for graph processing, a workload with inherent load imbalance, lack of locality, and access irregularity. We propose ShenTu, the first general-purpose graph processing framework that can efficiently utilize an entire petascale system to process multi-trillion edge graphs in seconds. ShenTu embodies four key innovations: hardware specializing, supernode routing, on-chip sorting, and degree-aware messaging, which together enable its unprecedented performance and scalability. It can traverse an unprecedented 70-trillion-edge graph in seconds.”

Jack Wells, OLCF

But back to Summit. Wells shared with HPCwire some of the distinguishing advantages Summit provides generally and some of which were leveraged by the Gordon Bell finalists. He noted two of the teams “were highly targeting the system’s mixed precision capabilities. The finite-element application explored ways mixed precision can boost performance by minimizing communication.”

Wells singled out four areas where Summit stands out:

  • “Because of the NVLink the users can use more system memory than they could on Titan. Connecting the Volta GPUs to the Power9 CPU using NVLink provides much higher bandwidth than possible with PCIe Gen4. NVLink provides enough bandwidth so that the three GPUs can saturate the Power9’s memory bandwidth. This enables app to use system memory in addition to the GPU’s HBM which is not practical on systems like Titan with PCIe-attached GPUs.
  • “The burst buffers, a reliable, high-speed storage layer that sits between the machine’s computing and file systems, significantly benefitted some teams who used it as a read accelerator rather than a write accelerator. Machine learning applications are read-heavy, so duplicating and moving data to the node local scratch memory was much faster than having it in GPFS.”
  • “With this generation of InfiniBand, Mellanox has vastly improved its Adaptive Routing which greatly reduces congestion and allows applications to scale better. Additionally, one of the teams extensively took advantage of Mellanox’s switch-based collective operations, which shaved significant time off synchronization operations that typically limit an application’s scalability.
  • “The Volta’s high bandwidth memory is very important. Summit’s nodes have more HBM than any other comparable system, which will allow our users to solve Gordon-Bell sized problems.”

On the software side, Summit users benefit from OCLF’s past experience with accelerators. Wells, noted, “Summit, like Titan, is a GPU-based system. Previous efforts to port and optimize codes for Titan have been beneficial for helping get codes ready for Summit. However, the Summit node is more complex, for example having multiple GPUs per node and having new features such as burst buffers and the GPU tensor cores. Adapting to this new node architecture has required effort by the code teams.”

Another issue for the Gordon Bell users, said Wells, is “[They only] had access to our relatively small test-and-development file system, not the full production file system that is undergoing acceptance testing these days. So, they had to work around this limitation. Also, the system software was still undergoing testing and debugging, so these teams were helping us identify such shortcomings and fix them.”

Information about the Summit stack – which includes XL, GNU, LLVM, PGI and NVCC compilers, LMOD, Spectrum MPI, ESSL, CUDA, LSF and JSM – is available on the web. The operating system is Red Hat Linux.

Obviously, these are early day for Summit which is still under preparation for full acceptance testing said Wells: “Users currently do not have access to the system as we attempt to finish this task. The IBM system is planned to be made available to the research community through DOE’s user programs beginning with allocations made under the Innovative and Novel Computational Impact on Theory and Experiment (INCITE) user program that will start in January 2019.”

That doesn’t mean plans aren’t afoot. They are. “For the past three years, teams have been preparing their applications to run on Summit,” said Wells. “A selection of the principal investigators of these application readiness teams includes:

  • Salman Habib of Argonne National Laboratory, whose team is modeling the large-scale structure and distribution of matter over the 13-billion-year lifespan of the universe.
  • Dmytro Bykov of Oak Ridge National Laboratory, whose team aims to describe the electronic structure of large molecular systems using quantum chemistry techniques, with targeted applications that include pharmacology and nanotechnology.
  • Abhishek Singharoy of Arizona State University, whose team is investigating the mechanics of a biological motor called ATP synthase in all-atom detail, a study which may aid the design of bioinspired clean energy technology.
  • Gaute Hagen of Oak Ridge National Laboratory, whose team is calculating the forces within atomic nuclei to study phenomena such as neutrinoless double-beta decay, a hypothesized form of radioactive decay.
  • Joe Oefelein of Georgia Tech, whose team is carrying out combustion simulations that closely match engine operating conditions to inform the design of fuel-efficient, low-emission engines.”

The Gordon Bell Prize winner will be announced at the SC2018 in Dallas in November; as you may know it’s awarded each year by the Association of Computing Machinery (ACM) to recognize outstanding achievement in high-performance computing. “The purpose of the award is to track the progress over time of parallel computing, with particular emphasis on rewarding innovation in applying high-performance computing to applications in science, engineering, and large-scale data analytics…Financial support of the $10,000 award is provided by Gordon Bell, a pioneer in high-performance and parallel computing,” says ACM.

Link to article on OCLF web site: https://www.olcf.ornl.gov/2018/09/17/uncharted-territory/

Link to Nvidia blog: https://blogs.nvidia.com/blog/2018/09/17/nvidia-volta-tensor-core-gpus-gordon-bell-finalists/

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Pfizer HPC Engineer Aims to Automate Software Stack Testing

January 17, 2019

Seeking to reign in the tediousness of manual software testing, Pfizer HPC Engineer Shahzeb Siddiqui is developing an open source software tool called buildtest, aimed at automating software stack testing by providing the community with a central repository of tests for common HPC apps and the ability to automate execution of testing. Read more…

By Tiffany Trader

Senegal Prepares to Take Delivery of Atos Supercomputer

January 16, 2019

In just a few months time, Senegal will be operating the second largest HPC system in sub-Saharan Africa. The Minister of Higher Education, Research and Innovation Mary Teuw Niane made the announcement on Monday (Jan. 14 Read more…

By Tiffany Trader

Google Cloud Platform Extends GPU Instance Options

January 16, 2019

If it's Nvidia GPUs you're after to power your AI/HPC/visualization workload, Google Cloud has them, now claiming "broadest GPU availability." Each of the three big public cloud vendors has by turn touted the latest and Read more…

By Tiffany Trader

HPE Extreme Performance Solutions

HPE Systems With Intel Omni-Path: Architected for Value and Accessible High-Performance Computing

Today’s high-performance computing (HPC) and artificial intelligence (AI) users value high performing clusters. And the higher the performance that their system can deliver, the better. Read more…

IBM Accelerated Insights

Resource Management in the Age of Artificial Intelligence

New challenges demand fresh approaches

Fueled by GPUs, big data, and rapid advances in software, the AI revolution is upon us. Read more…

STAC Floats ML Benchmark for Financial Services Workloads

January 16, 2019

STAC (Securities Technology Analysis Center) recently released an ‘exploratory’ benchmark for machine learning which it hopes will evolve into a firm benchmark or suite of benchmarking tools to compare the performanc Read more…

By John Russell

Google Cloud Platform Extends GPU Instance Options

January 16, 2019

If it's Nvidia GPUs you're after to power your AI/HPC/visualization workload, Google Cloud has them, now claiming "broadest GPU availability." Each of the three Read more…

By Tiffany Trader

STAC Floats ML Benchmark for Financial Services Workloads

January 16, 2019

STAC (Securities Technology Analysis Center) recently released an ‘exploratory’ benchmark for machine learning which it hopes will evolve into a firm benchm Read more…

By John Russell

A Big Data Journey While Seeking to Catalog our Universe

January 16, 2019

It turns out, astronomers have lots of photos of the sky but seek knowledge about what the photos mean. Sound familiar? Big data problems are often characterize Read more…

By James Reinders

Intel Bets Big on 2-Track Quantum Strategy

January 15, 2019

Quantum computing has lived so long in the future it’s taken on a futuristic life of its own, with a Gartner-style hype cycle that includes triggers of innovation, inflated expectations and – though a useful quantum system is still years away – anticipatory troughs of disillusionment. Read more…

By Doug Black

IBM Quantum Update: Q System One Launch, New Collaborators, and QC Center Plans

January 10, 2019

IBM made three significant quantum computing announcements at CES this week. One was introduction of IBM Q System One; it’s really the integration of IBM’s Read more…

By John Russell

IBM’s New Global Weather Forecasting System Runs on GPUs

January 9, 2019

Anyone who has checked a forecast to decide whether or not to pack an umbrella knows that weather prediction can be a mercurial endeavor. It is a Herculean task: the constant modeling of incredibly complex systems to a high degree of accuracy at a local level within very short spans of time. Read more…

By Oliver Peckham

The Case Against ‘The Case Against Quantum Computing’

January 9, 2019

It’s not easy to be a physicist. Richard Feynman (basically the Jimi Hendrix of physicists) once said: “The first principle is that you must not fool yourse Read more…

By Ben Criger

The Deep500 – Researchers Tackle an HPC Benchmark for Deep Learning

January 7, 2019

How do you know if an HPC system, particularly a larger-scale system, is well-suited for deep learning workloads? Today, that’s not an easy question to answer Read more…

By John Russell

Quantum Computing Will Never Work

November 27, 2018

Amid the gush of money and enthusiastic predictions being thrown at quantum computing comes a proposed cold shower in the form of an essay by physicist Mikhail Read more…

By John Russell

Cray Unveils Shasta, Lands NERSC-9 Contract

October 30, 2018

Cray revealed today the details of its next-gen supercomputing architecture, Shasta, selected to be the next flagship system at NERSC. We've known of the code-name "Shasta" since the Argonne slice of the CORAL project was announced in 2015 and although the details of that plan have changed considerably, Cray didn't slow down its timeline for Shasta. Read more…

By Tiffany Trader

AMD Sets Up for Epyc Epoch

November 16, 2018

It’s been a good two weeks, AMD’s Gary Silcott and Andy Parma told me on the last day of SC18 in Dallas at the restaurant where we met to discuss their show news and recent successes. Heck, it’s been a good year. Read more…

By Tiffany Trader

The Case Against ‘The Case Against Quantum Computing’

January 9, 2019

It’s not easy to be a physicist. Richard Feynman (basically the Jimi Hendrix of physicists) once said: “The first principle is that you must not fool yourse Read more…

By Ben Criger

US Leads Supercomputing with #1, #2 Systems & Petascale Arm

November 12, 2018

The 31st Supercomputing Conference (SC) - commemorating 30 years since the first Supercomputing in 1988 - kicked off in Dallas yesterday, taking over the Kay Ba Read more…

By Tiffany Trader

Contract Signed for New Finnish Supercomputer

December 13, 2018

After the official contract signing yesterday, configuration details were made public for the new BullSequana system that the Finnish IT Center for Science (CSC Read more…

By Tiffany Trader

Summit Supercomputer is Already Making its Mark on Science

September 20, 2018

Summit, now the fastest supercomputer in the world, is quickly making its mark in science – five of the six finalists just announced for the prestigious 2018 Read more…

By John Russell

Nvidia’s Jensen Huang Delivers Vision for the New HPC

November 14, 2018

For nearly two hours on Monday at SC18, Jensen Huang, CEO of Nvidia, presented his expansive view of the future of HPC (and computing in general) as only he can do. Animated. Backstopped by a stream of data charts, product photos, and even a beautiful image of supernovae... Read more…

By John Russell

Leading Solution Providers

SC 18 Virtual Booth Video Tour

Advania @ SC18 AMD @ SC18
ASRock Rack @ SC18
DDN Storage @ SC18
HPE @ SC18
IBM @ SC18
Lenovo @ SC18 Mellanox Technologies @ SC18
NVIDIA @ SC18
One Stop Systems @ SC18
Oracle @ SC18 Panasas @ SC18
Supermicro @ SC18 SUSE @ SC18 TYAN @ SC18
Verne Global @ SC18

HPE No. 1, IBM Surges, in ‘Bucking Bronco’ High Performance Server Market

September 27, 2018

Riding healthy U.S. and global economies, strong demand for AI-capable hardware and other tailwind trends, the high performance computing server market jumped 28 percent in the second quarter 2018 to $3.7 billion, up from $2.9 billion for the same period last year, according to industry analyst firm Hyperion Research. Read more…

By Doug Black

HPC Reflections and (Mostly Hopeful) Predictions

December 19, 2018

So much ‘spaghetti’ gets tossed on walls by the technology community (vendors and researchers) to see what sticks that it is often difficult to peer through Read more…

By John Russell

Intel Confirms 48-Core Cascade Lake-AP for 2019

November 4, 2018

As part of the run-up to SC18, taking place in Dallas next week (Nov. 11-16), Intel is doling out info on its next-gen Cascade Lake family of Xeon processors, specifically the “Advanced Processor” version (Cascade Lake-AP), architected for high-performance computing, artificial intelligence and infrastructure-as-a-service workloads. Read more…

By Tiffany Trader

Germany Celebrates Launch of Two Fastest Supercomputers

September 26, 2018

The new high-performance computer SuperMUC-NG at the Leibniz Supercomputing Center (LRZ) in Garching is the fastest computer in Germany and one of the fastest i Read more…

By Tiffany Trader

House Passes $1.275B National Quantum Initiative

September 17, 2018

Last Thursday the U.S. House of Representatives passed the National Quantum Initiative Act (NQIA) intended to accelerate quantum computing research and developm Read more…

By John Russell

Houston to Field Massive, ‘Geophysically Configured’ Cloud Supercomputer

October 11, 2018

Based on some news stories out today, one might get the impression that the next system to crack number one on the Top500 would be an industrial oil and gas mon Read more…

By Tiffany Trader

Microsoft to Buy Mellanox?

December 20, 2018

Networking equipment powerhouse Mellanox could be an acquisition target by Microsoft, according to a published report in an Israeli financial publication. Microsoft has reportedly gone so far as to engage Goldman Sachs to handle negotiations with Mellanox. Read more…

By Doug Black

The Deep500 – Researchers Tackle an HPC Benchmark for Deep Learning

January 7, 2019

How do you know if an HPC system, particularly a larger-scale system, is well-suited for deep learning workloads? Today, that’s not an easy question to answer Read more…

By John Russell

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This