Summit Supercomputer is Already Making its Mark on Science

By John Russell

September 20, 2018

Summit, now the fastest supercomputer in the world, is quickly making its mark in science – five of the six finalists just announced for the prestigious 2018 Gordon Bell Prize used Summit in their work. That’s impressive given that Summit only began full operation in early summer. Also noteworthy is Summit’s heterogeneous architecture which leverages IBM’s Power9 CPU, Nvidia V100 GPUs, and fast interconnect technology from Mellanox to accommodate traditional simulation workloads as well as mixed-precision workloads associated with AI and data analytics.

By now, Summit needs little introduction having topped the most recent Top500 list. Located at the Oak Ridge Leadership Computing Facility (OCLF), it cost an estimated $200 million to build as part of the DoE CORAL procurement program. It’s been heralded as the world’s most powerful supercomputer at 200 petaflops theoretical peak for high-performance computing workloads and 3.3 peak exaops for emerging AI workloads. (Sierra, the similarly architected but somewhat smaller, 125 petaflops theoretical peak machine based at Lawrence Livermore National Laboratory, was also used in some of the cited research.)

“These Gordon Bell finalists are an encouraging preview of the challenges users will be able to tackle on Summit when formal allocation programs begin in 2019,” said OLCF director of science Jack Wells. “Of particular note is the system’s ability to handle large volumes of data at scale, whether that be processing and analyzing experimental data or training artificial intelligence software to carry out specialized tasks.”

Nvidia and IBM are, understandably, ecstatic over Summit’s progress.

In a Nvidia blog, product manager Geetika Gupta wrote, “The revolutionary accelerators enable multi-precision computing that fuses the highly precise calculations to tackle the challenges of high performance computing with the efficient processing required for deep learning…[H]alf of the six projects included NVIDIA researchers who were heavily involved with the code development and performance tuning.”

Dave Turek, IBM Cognitive Systems VP, said “IBM designed Summit and Sierra to be data-centric, heterogeneous systems that maximized data flow for optimal application performance. The industry-leading IO features of IBM POWER9 processors allow for data to flow in and out of Summit’s GPUs to achieve the unprecedented level of performance demonstrated by these Gordon Bell finalists.”

They can, perhaps, be forgiven a little excess enthusiasm. These machines are difficult to design and build. Clearly, Summit’s early success is more evidence that heterogeneous architectures that leverage accelerators are likely to dominate high-end computing going forward.

Here is a lightly edited excerpt from an OCLF article describing the finalists who used Summit in their research:

  • “Genomics. An ORNL team led by computational systems biologist Dan Jacobson and OLCF computational scientist Wayne Joubert that developed a genomics algorithm capable of using mixed-precision arithmetic to attain exascale speeds. On Summit, the team’s Combinatorial Metrics application achieved a peak throughput of 2.36 exaops—or 2.36 billion billion calculations per second, the fastest science application ever reported. Jacobson’s work compares genetic variations within a population to uncover hidden networks of genes that contribute to complex traits. One condition Jacobson’s team is studying is opioid addiction, which has been linked to the deaths of more than 49,000 people in the United States in 2017.
  • Earthquake Simulation. A team from the University of Tokyo led by associate professor Tsuyoshi Ichimura that applied artificial intelligence (AI) and mixed-precision arithmetic to accelerate the simulation of earthquake physics in urban environments. As cities continue to grow, preparedness and improved understanding of ground-shaking’s effects on buildings and urban infrastructure become increasingly important. On Summit, the Tokyo team expanded on its 2014 algorithm, which was also a Gordon Bell Finalist, to achieve a fourfold speedup and to couple the shaking of ground and urban structures during large earthquakes into the same simulation.
  • Extreme Weather. A Lawrence Berkeley National Laboratory-led collaboration that trained a deep neural network to identify extreme weather patterns from high-resolution climate simulations.The team, led by Berkeley data scientist Prabhat, plans to use the AI software to predict how extreme weather is likely to change in the future. By tapping into the specialized tensor cores built into Summit’s NVIDIA GPUs at scale, the Berkeley team achieved a peak performance of 1.13 exaops, the fastest deep-learning algorithm yet reported. Though the team applied its work to climate science, many of its innovations can be adapted for other deep-learning applications.
  • Materials Science. An ORNL team led by data scientist Robert Patton that scaled a deep-learning technique on Summit to produce intelligent software that can automatically identify materials’ atomic-level information from electron microscopy With advanced microscopes capable of producing hundreds of images per day, real-time feedback supplied by AI could give scientists the ability to fabricate materials at the atomic level. Scaled across 4,200 nodes, the team’s MENNDL algorithm achieved a speed of 152.5 petaflops with an estimated performance rate of 167 petaflops across the whole machine.
  • Physics. A team from Lawrence Berkeley and Lawrence Livermore National Laboratories led by physicists André Walker-Loud and Pavlos Vranas that developed improved algorithms to help scientists predict the lifetime of neutrons and answer fundamental questions about the universe. The team built upon its previous work using lattice quantum chromodynamics—a numerical method for calculating the underlying physics of the subatomic particles that make up protons and neutrons. In addition to optimized GPU software, the team developed lightweight, application-agnostic management software capable of managing hundreds of thousands of tasks. Using GPU-accelerated systems Sierra at Lawrence Livermore and the OLCF’s Summit, the team was able to start 1,056 four-node jobs on 4,224 nodes in 5 minutes, achieving a machine-to-machine speedup of factors of 10 and 15, respectively, over the OLCF’s previous leadership-class system, Titan. The achievement supplies nuclear physicists with the necessary computational power to support the experimental search for new physics.”

We’d be remiss not to mention the sixth Gordon finalist; it’s from a group of researchers from China who developed a graph processing framework (ShenTu) adapted for use on HPC resources. Here is a description of that very impressive work (ShenTu: Processing Multi-Trillion Edge Graphs on Millions of Cores in Seconds) taken from the SC18 web site.

“DescriptionGraphs are an important abstraction used in many scientific fields. With the magnitude of graph-structured data constantly increasing, effective data analytics requires efficient and scalable graph processing systems. Although HPC systems have long been used for scientific computing, people have only recently started to assess their potential for graph processing, a workload with inherent load imbalance, lack of locality, and access irregularity. We propose ShenTu, the first general-purpose graph processing framework that can efficiently utilize an entire petascale system to process multi-trillion edge graphs in seconds. ShenTu embodies four key innovations: hardware specializing, supernode routing, on-chip sorting, and degree-aware messaging, which together enable its unprecedented performance and scalability. It can traverse an unprecedented 70-trillion-edge graph in seconds.”

Jack Wells, OLCF

But back to Summit. Wells shared with HPCwire some of the distinguishing advantages Summit provides generally and some of which were leveraged by the Gordon Bell finalists. He noted two of the teams “were highly targeting the system’s mixed precision capabilities. The finite-element application explored ways mixed precision can boost performance by minimizing communication.”

Wells singled out four areas where Summit stands out:

  • “Because of the NVLink the users can use more system memory than they could on Titan. Connecting the Volta GPUs to the Power9 CPU using NVLink provides much higher bandwidth than possible with PCIe Gen4. NVLink provides enough bandwidth so that the three GPUs can saturate the Power9’s memory bandwidth. This enables app to use system memory in addition to the GPU’s HBM which is not practical on systems like Titan with PCIe-attached GPUs.
  • “The burst buffers, a reliable, high-speed storage layer that sits between the machine’s computing and file systems, significantly benefitted some teams who used it as a read accelerator rather than a write accelerator. Machine learning applications are read-heavy, so duplicating and moving data to the node local scratch memory was much faster than having it in GPFS.”
  • “With this generation of InfiniBand, Mellanox has vastly improved its Adaptive Routing which greatly reduces congestion and allows applications to scale better. Additionally, one of the teams extensively took advantage of Mellanox’s switch-based collective operations, which shaved significant time off synchronization operations that typically limit an application’s scalability.
  • “The Volta’s high bandwidth memory is very important. Summit’s nodes have more HBM than any other comparable system, which will allow our users to solve Gordon-Bell sized problems.”

On the software side, Summit users benefit from OCLF’s past experience with accelerators. Wells, noted, “Summit, like Titan, is a GPU-based system. Previous efforts to port and optimize codes for Titan have been beneficial for helping get codes ready for Summit. However, the Summit node is more complex, for example having multiple GPUs per node and having new features such as burst buffers and the GPU tensor cores. Adapting to this new node architecture has required effort by the code teams.”

Another issue for the Gordon Bell users, said Wells, is “[They only] had access to our relatively small test-and-development file system, not the full production file system that is undergoing acceptance testing these days. So, they had to work around this limitation. Also, the system software was still undergoing testing and debugging, so these teams were helping us identify such shortcomings and fix them.”

Information about the Summit stack – which includes XL, GNU, LLVM, PGI and NVCC compilers, LMOD, Spectrum MPI, ESSL, CUDA, LSF and JSM – is available on the web. The operating system is Red Hat Linux.

Obviously, these are early day for Summit which is still under preparation for full acceptance testing said Wells: “Users currently do not have access to the system as we attempt to finish this task. The IBM system is planned to be made available to the research community through DOE’s user programs beginning with allocations made under the Innovative and Novel Computational Impact on Theory and Experiment (INCITE) user program that will start in January 2019.”

That doesn’t mean plans aren’t afoot. They are. “For the past three years, teams have been preparing their applications to run on Summit,” said Wells. “A selection of the principal investigators of these application readiness teams includes:

  • Salman Habib of Argonne National Laboratory, whose team is modeling the large-scale structure and distribution of matter over the 13-billion-year lifespan of the universe.
  • Dmytro Bykov of Oak Ridge National Laboratory, whose team aims to describe the electronic structure of large molecular systems using quantum chemistry techniques, with targeted applications that include pharmacology and nanotechnology.
  • Abhishek Singharoy of Arizona State University, whose team is investigating the mechanics of a biological motor called ATP synthase in all-atom detail, a study which may aid the design of bioinspired clean energy technology.
  • Gaute Hagen of Oak Ridge National Laboratory, whose team is calculating the forces within atomic nuclei to study phenomena such as neutrinoless double-beta decay, a hypothesized form of radioactive decay.
  • Joe Oefelein of Georgia Tech, whose team is carrying out combustion simulations that closely match engine operating conditions to inform the design of fuel-efficient, low-emission engines.”

The Gordon Bell Prize winner will be announced at the SC2018 in Dallas in November; as you may know it’s awarded each year by the Association of Computing Machinery (ACM) to recognize outstanding achievement in high-performance computing. “The purpose of the award is to track the progress over time of parallel computing, with particular emphasis on rewarding innovation in applying high-performance computing to applications in science, engineering, and large-scale data analytics…Financial support of the $10,000 award is provided by Gordon Bell, a pioneer in high-performance and parallel computing,” says ACM.

Link to article on OCLF web site: https://www.olcf.ornl.gov/2018/09/17/uncharted-territory/

Link to Nvidia blog: https://blogs.nvidia.com/blog/2018/09/17/nvidia-volta-tensor-core-gpus-gordon-bell-finalists/

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

IBM Unveils Latest Achievements in AI Hardware

December 13, 2019

“The increased capabilities of contemporary AI models provide unprecedented recognition accuracy, but often at the expense of larger computational and energetic effort,” IBM Research wrote in a blog post. “Therefor Read more…

By Oliver Peckham

Focused on ‘Silicon TAM,’ Intel Puts Gary Patton, Former GlobalFoundries CTO, in Charge of Design Enablement

December 12, 2019

Change within Intel’s upper management – and to its company mission – has continued as a published report has disclosed that chip technology heavyweight Gary Patton, GlobalFoundries’ CTO and R&D SVP as well a Read more…

By Doug Black

Quantum Bits: Rigetti Debuts New Gates, D-Wave Cuts NEC Deal, AWS Jumps into the Quantum Pool

December 12, 2019

There’s been flurry of significant news in the quantum computing world. Yesterday, Rigetti introduced a new family of gates that reduces circuit depth required on some problems and D-Wave struck a deal with NEC to coll Read more…

By John Russell

How Formula 1 Used Cloud HPC to Build the Next Generation of Racing

December 12, 2019

Formula 1, Rob Smedley explained, is maybe the biggest racing spectacle in the world, with five hundred million fans tuning in for every race. Smedley, a chief engineer with Formula 1’s performance engineering and anal Read more…

By Oliver Peckham

RPI Powers Up ‘AiMOS’ AI Supercomputer

December 11, 2019

Designed to push the frontiers of computing chip and systems performance optimized for AI workloads, an 8 petaflops (Linpack) IBM Power9-based supercomputer has been unveiled in upstate New York that will be used by IBM Read more…

By Doug Black

AWS Solution Channel

Making High Performance Computing Affordable and Accessible for Small and Medium Businesses with HPC on AWS

High performance computing (HPC) brings a powerful set of tools to a broad range of industries, helping to drive innovation and boost revenue in finance, genomics, oil and gas extraction, and other fields. Read more…

IBM Accelerated Insights

GPU Scheduling and Resource Accounting: The Key to an Efficient AI Data Center

[Connect with LSF users and learn new skills in the IBM Spectrum LSF User Community!]

GPUs are the new CPUs

GPUs have become a staple technology in modern HPC and AI data centers. Read more…

At SC19: Developing a Digital Twin

December 11, 2019

In the not too distant future, we can expect to see our skies filled with unmanned aerial vehicles (UAVs) delivering packages, maybe even people, from location to location. In such a world, there will also be a digital twin for each UAV in the fleet: a virtual model that will follow the UAV through its existence, evolving with time. Read more…

By Aaron Dubrow

Focused on ‘Silicon TAM,’ Intel Puts Gary Patton, Former GlobalFoundries CTO, in Charge of Design Enablement

December 12, 2019

Change within Intel’s upper management – and to its company mission – has continued as a published report has disclosed that chip technology heavyweight G Read more…

By Doug Black

Quantum Bits: Rigetti Debuts New Gates, D-Wave Cuts NEC Deal, AWS Jumps into the Quantum Pool

December 12, 2019

There’s been flurry of significant news in the quantum computing world. Yesterday, Rigetti introduced a new family of gates that reduces circuit depth require Read more…

By John Russell

RPI Powers Up ‘AiMOS’ AI Supercomputer

December 11, 2019

Designed to push the frontiers of computing chip and systems performance optimized for AI workloads, an 8 petaflops (Linpack) IBM Power9-based supercomputer has Read more…

By Doug Black

At SC19: Developing a Digital Twin

December 11, 2019

In the not too distant future, we can expect to see our skies filled with unmanned aerial vehicles (UAVs) delivering packages, maybe even people, from location to location. In such a world, there will also be a digital twin for each UAV in the fleet: a virtual model that will follow the UAV through its existence, evolving with time. Read more…

By Aaron Dubrow

Intel’s Jim Clarke on its New Cryo-controller and why Intel isn’t Late to the Quantum Party

December 9, 2019

Intel today introduced the ‘first-of-its-kind’ cryo-controller chip for quantum computing and previewed a cryo-prober tool for characterizing quantum proces Read more…

By John Russell

On the Spack Track @SC19

December 5, 2019

At the annual supercomputing conference, SC19 in Denver, Colorado, there were Spack events each day of the conference. As a reflection of its grassroots heritage, nine sessions were planned by more than a dozen thought leaders from seven organizations, including three U.S. national Department of Energy (DOE) laboratories and Sylabs... Read more…

By Elizabeth Leake

Intel’s New Hyderabad Design Center Targets Exascale Era Technologies

December 3, 2019

Intel's Raja Koduri was in India this week to help launch a new 300,000 square foot design and engineering center in Hyderabad, which will focus on advanced com Read more…

By Tiffany Trader

AWS Debuts 7nm 2nd-Gen Graviton Arm Processor

December 3, 2019

The “x86 Big Bang,” in which market dominance of the venerable Intel CPU has exploded into fragments of processor options suited to varying workloads, has n Read more…

By Doug Black

Using AI to Solve One of the Most Prevailing Problems in CFD

October 17, 2019

How can artificial intelligence (AI) and high-performance computing (HPC) solve mesh generation, one of the most commonly referenced problems in computational engineering? A new study has set out to answer this question and create an industry-first AI-mesh application... Read more…

By James Sharpe

D-Wave’s Path to 5000 Qubits; Google’s Quantum Supremacy Claim

September 24, 2019

On the heels of IBM’s quantum news last week come two more quantum items. D-Wave Systems today announced the name of its forthcoming 5000-qubit system, Advantage (yes the name choice isn’t serendipity), at its user conference being held this week in Newport, RI. Read more…

By John Russell

DARPA Looks to Propel Parallelism

September 4, 2019

As Moore’s law runs out of steam, new programming approaches are being pursued with the goal of greater hardware performance with less coding. The Defense Advanced Projects Research Agency is launching a new programming effort aimed at leveraging the benefits of massive distributed parallelism with less sweat. Read more…

By George Leopold

Ayar Labs to Demo Photonics Chiplet in FPGA Package at Hot Chips

August 19, 2019

Silicon startup Ayar Labs continues to gain momentum with its DARPA-backed optical chiplet technology that puts advanced electronics and optics on the same chip Read more…

By Tiffany Trader

SC19: IBM Changes Its HPC-AI Game Plan

November 25, 2019

It’s probably fair to say IBM is known for big bets. Summit supercomputer – a big win. Red Hat acquisition – looking like a big win. OpenPOWER and Power processors – jury’s out? At SC19, long-time IBMer Dave Turek sketched out a different kind of bet for Big Blue – a small ball strategy, if you’ll forgive the baseball analogy... Read more…

By John Russell

Cray, Fujitsu Both Bringing Fujitsu A64FX-based Supercomputers to Market in 2020

November 12, 2019

The number of top-tier HPC systems makers has shrunk due to a steady march of M&A activity, but there is increased diversity and choice of processing compon Read more…

By Tiffany Trader

Crystal Ball Gazing: IBM’s Vision for the Future of Computing

October 14, 2019

Dario Gil, IBM’s relatively new director of research, painted a intriguing portrait of the future of computing along with a rough idea of how IBM thinks we’ Read more…

By John Russell

Kubernetes, Containers and HPC

September 19, 2019

Software containers and Kubernetes are important tools for building, deploying, running and managing modern enterprise applications at scale and delivering enterprise software faster and more reliably to the end user — while using resources more efficiently and reducing costs. Read more…

By Daniel Gruber, Burak Yenier and Wolfgang Gentzsch, UberCloud

Leading Solution Providers

SC 2019 Virtual Booth Video Tour

AMD
AMD
CEJN
CJEN
DDN
DDN
MELLANOX
MELLANOX
ONE STOP SYSTEMS
ONE STOP SYSTEMS
PANASAS
PANASAS
SIX NINES IT
SIX NINES IT
VERNE GLOBAL
VERNE GLOBAL
WEKAIO
WEKAIO

Intel Debuts New GPU – Ponte Vecchio – and Outlines Aspirations for oneAPI

November 17, 2019

Intel today revealed a few more details about its forthcoming Xe line of GPUs – the top SKU is named Ponte Vecchio and will be used in Aurora, the first plann Read more…

By John Russell

Dell Ramps Up HPC Testing of AMD Rome Processors

October 21, 2019

Dell Technologies is wading deeper into the AMD-based systems market with a growing evaluation program for the latest Epyc (Rome) microprocessors from AMD. In a Read more…

By John Russell

SC19: Welcome to Denver

November 17, 2019

A significant swath of the HPC community has come to Denver for SC19, which began today (Sunday) with a rich technical program. As is customary, the ribbon cutt Read more…

By Tiffany Trader

When Dense Matrix Representations Beat Sparse

September 9, 2019

In our world filled with unintended consequences, it turns out that saving memory space to help deal with GPU limitations, knowing it introduces performance pen Read more…

By James Reinders

Cerebras to Supply DOE with Wafer-Scale AI Supercomputing Technology

September 17, 2019

Cerebras Systems, which debuted its wafer-scale AI silicon at Hot Chips last month, has entered into a multi-year partnership with Argonne National Laboratory and Lawrence Livermore National Laboratory as part of a larger collaboration with the U.S. Department of Energy... Read more…

By Tiffany Trader

With the Help of HPC, Astronomers Prepare to Deflect a Real Asteroid

September 26, 2019

For years, NASA has been running simulations of asteroid impacts to understand the risks (and likelihoods) of asteroids colliding with Earth. Now, NASA and the European Space Agency (ESA) are preparing for the next, crucial step in planetary defense against asteroid impacts: physically deflecting a real asteroid. Read more…

By Oliver Peckham

Jensen Huang’s SC19 – Fast Cars, a Strong Arm, and Aiming for the Cloud(s)

November 20, 2019

We’ve come to expect Nvidia CEO Jensen Huang’s annual SC keynote to contain stunning graphics and lively bravado (with plenty of examples) in support of GPU Read more…

By John Russell

IBM Opens Quantum Computing Center; Announces 53-Qubit Machine

September 19, 2019

Gauging progress in quantum computing is a tricky thing. IBM yesterday announced the opening of the IBM Quantum Computing Center in New York, with five 20-qubit Read more…

By John Russell

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This