Summit Supercomputer is Already Making its Mark on Science

By John Russell

September 20, 2018

Summit, now the fastest supercomputer in the world, is quickly making its mark in science – five of the six finalists just announced for the prestigious 2018 Gordon Bell Prize used Summit in their work. That’s impressive given that Summit only began full operation in early summer. Also noteworthy is Summit’s heterogeneous architecture which leverages IBM’s Power9 CPU, Nvidia V100 GPUs, and fast interconnect technology from Mellanox to accommodate traditional simulation workloads as well as mixed-precision workloads associated with AI and data analytics.

By now, Summit needs little introduction having topped the most recent Top500 list. Located at the Oak Ridge Leadership Computing Facility (OCLF), it cost an estimated $200 million to build as part of the DoE CORAL procurement program. It’s been heralded as the world’s most powerful supercomputer at 200 petaflops theoretical peak for high-performance computing workloads and 3.3 peak exaops for emerging AI workloads. (Sierra, the similarly architected but somewhat smaller, 125 petaflops theoretical peak machine based at Lawrence Livermore National Laboratory, was also used in some of the cited research.)

“These Gordon Bell finalists are an encouraging preview of the challenges users will be able to tackle on Summit when formal allocation programs begin in 2019,” said OLCF director of science Jack Wells. “Of particular note is the system’s ability to handle large volumes of data at scale, whether that be processing and analyzing experimental data or training artificial intelligence software to carry out specialized tasks.”

Nvidia and IBM are, understandably, ecstatic over Summit’s progress.

In a Nvidia blog, product manager Geetika Gupta wrote, “The revolutionary accelerators enable multi-precision computing that fuses the highly precise calculations to tackle the challenges of high performance computing with the efficient processing required for deep learning…[H]alf of the six projects included NVIDIA researchers who were heavily involved with the code development and performance tuning.”

Dave Turek, IBM Cognitive Systems VP, said “IBM designed Summit and Sierra to be data-centric, heterogeneous systems that maximized data flow for optimal application performance. The industry-leading IO features of IBM POWER9 processors allow for data to flow in and out of Summit’s GPUs to achieve the unprecedented level of performance demonstrated by these Gordon Bell finalists.”

They can, perhaps, be forgiven a little excess enthusiasm. These machines are difficult to design and build. Clearly, Summit’s early success is more evidence that heterogeneous architectures that leverage accelerators are likely to dominate high-end computing going forward.

Here is a lightly edited excerpt from an OCLF article describing the finalists who used Summit in their research:

  • “Genomics. An ORNL team led by computational systems biologist Dan Jacobson and OLCF computational scientist Wayne Joubert that developed a genomics algorithm capable of using mixed-precision arithmetic to attain exascale speeds. On Summit, the team’s Combinatorial Metrics application achieved a peak throughput of 2.36 exaops—or 2.36 billion billion calculations per second, the fastest science application ever reported. Jacobson’s work compares genetic variations within a population to uncover hidden networks of genes that contribute to complex traits. One condition Jacobson’s team is studying is opioid addiction, which has been linked to the deaths of more than 49,000 people in the United States in 2017.
  • Earthquake Simulation. A team from the University of Tokyo led by associate professor Tsuyoshi Ichimura that applied artificial intelligence (AI) and mixed-precision arithmetic to accelerate the simulation of earthquake physics in urban environments. As cities continue to grow, preparedness and improved understanding of ground-shaking’s effects on buildings and urban infrastructure become increasingly important. On Summit, the Tokyo team expanded on its 2014 algorithm, which was also a Gordon Bell Finalist, to achieve a fourfold speedup and to couple the shaking of ground and urban structures during large earthquakes into the same simulation.
  • Extreme Weather. A Lawrence Berkeley National Laboratory-led collaboration that trained a deep neural network to identify extreme weather patterns from high-resolution climate simulations.The team, led by Berkeley data scientist Prabhat, plans to use the AI software to predict how extreme weather is likely to change in the future. By tapping into the specialized tensor cores built into Summit’s NVIDIA GPUs at scale, the Berkeley team achieved a peak performance of 1.13 exaops, the fastest deep-learning algorithm yet reported. Though the team applied its work to climate science, many of its innovations can be adapted for other deep-learning applications.
  • Materials Science. An ORNL team led by data scientist Robert Patton that scaled a deep-learning technique on Summit to produce intelligent software that can automatically identify materials’ atomic-level information from electron microscopy With advanced microscopes capable of producing hundreds of images per day, real-time feedback supplied by AI could give scientists the ability to fabricate materials at the atomic level. Scaled across 4,200 nodes, the team’s MENNDL algorithm achieved a speed of 152.5 petaflops with an estimated performance rate of 167 petaflops across the whole machine.
  • Physics. A team from Lawrence Berkeley and Lawrence Livermore National Laboratories led by physicists André Walker-Loud and Pavlos Vranas that developed improved algorithms to help scientists predict the lifetime of neutrons and answer fundamental questions about the universe. The team built upon its previous work using lattice quantum chromodynamics—a numerical method for calculating the underlying physics of the subatomic particles that make up protons and neutrons. In addition to optimized GPU software, the team developed lightweight, application-agnostic management software capable of managing hundreds of thousands of tasks. Using GPU-accelerated systems Sierra at Lawrence Livermore and the OLCF’s Summit, the team was able to start 1,056 four-node jobs on 4,224 nodes in 5 minutes, achieving a machine-to-machine speedup of factors of 10 and 15, respectively, over the OLCF’s previous leadership-class system, Titan. The achievement supplies nuclear physicists with the necessary computational power to support the experimental search for new physics.”

We’d be remiss not to mention the sixth Gordon finalist; it’s from a group of researchers from China who developed a graph processing framework (ShenTu) adapted for use on HPC resources. Here is a description of that very impressive work (ShenTu: Processing Multi-Trillion Edge Graphs on Millions of Cores in Seconds) taken from the SC18 web site.

“DescriptionGraphs are an important abstraction used in many scientific fields. With the magnitude of graph-structured data constantly increasing, effective data analytics requires efficient and scalable graph processing systems. Although HPC systems have long been used for scientific computing, people have only recently started to assess their potential for graph processing, a workload with inherent load imbalance, lack of locality, and access irregularity. We propose ShenTu, the first general-purpose graph processing framework that can efficiently utilize an entire petascale system to process multi-trillion edge graphs in seconds. ShenTu embodies four key innovations: hardware specializing, supernode routing, on-chip sorting, and degree-aware messaging, which together enable its unprecedented performance and scalability. It can traverse an unprecedented 70-trillion-edge graph in seconds.”

Jack Wells, OLCF

But back to Summit. Wells shared with HPCwire some of the distinguishing advantages Summit provides generally and some of which were leveraged by the Gordon Bell finalists. He noted two of the teams “were highly targeting the system’s mixed precision capabilities. The finite-element application explored ways mixed precision can boost performance by minimizing communication.”

Wells singled out four areas where Summit stands out:

  • “Because of the NVLink the users can use more system memory than they could on Titan. Connecting the Volta GPUs to the Power9 CPU using NVLink provides much higher bandwidth than possible with PCIe Gen4. NVLink provides enough bandwidth so that the three GPUs can saturate the Power9’s memory bandwidth. This enables app to use system memory in addition to the GPU’s HBM which is not practical on systems like Titan with PCIe-attached GPUs.
  • “The burst buffers, a reliable, high-speed storage layer that sits between the machine’s computing and file systems, significantly benefitted some teams who used it as a read accelerator rather than a write accelerator. Machine learning applications are read-heavy, so duplicating and moving data to the node local scratch memory was much faster than having it in GPFS.”
  • “With this generation of InfiniBand, Mellanox has vastly improved its Adaptive Routing which greatly reduces congestion and allows applications to scale better. Additionally, one of the teams extensively took advantage of Mellanox’s switch-based collective operations, which shaved significant time off synchronization operations that typically limit an application’s scalability.
  • “The Volta’s high bandwidth memory is very important. Summit’s nodes have more HBM than any other comparable system, which will allow our users to solve Gordon-Bell sized problems.”

On the software side, Summit users benefit from OCLF’s past experience with accelerators. Wells, noted, “Summit, like Titan, is a GPU-based system. Previous efforts to port and optimize codes for Titan have been beneficial for helping get codes ready for Summit. However, the Summit node is more complex, for example having multiple GPUs per node and having new features such as burst buffers and the GPU tensor cores. Adapting to this new node architecture has required effort by the code teams.”

Another issue for the Gordon Bell users, said Wells, is “[They only] had access to our relatively small test-and-development file system, not the full production file system that is undergoing acceptance testing these days. So, they had to work around this limitation. Also, the system software was still undergoing testing and debugging, so these teams were helping us identify such shortcomings and fix them.”

Information about the Summit stack – which includes XL, GNU, LLVM, PGI and NVCC compilers, LMOD, Spectrum MPI, ESSL, CUDA, LSF and JSM – is available on the web. The operating system is Red Hat Linux.

Obviously, these are early day for Summit which is still under preparation for full acceptance testing said Wells: “Users currently do not have access to the system as we attempt to finish this task. The IBM system is planned to be made available to the research community through DOE’s user programs beginning with allocations made under the Innovative and Novel Computational Impact on Theory and Experiment (INCITE) user program that will start in January 2019.”

That doesn’t mean plans aren’t afoot. They are. “For the past three years, teams have been preparing their applications to run on Summit,” said Wells. “A selection of the principal investigators of these application readiness teams includes:

  • Salman Habib of Argonne National Laboratory, whose team is modeling the large-scale structure and distribution of matter over the 13-billion-year lifespan of the universe.
  • Dmytro Bykov of Oak Ridge National Laboratory, whose team aims to describe the electronic structure of large molecular systems using quantum chemistry techniques, with targeted applications that include pharmacology and nanotechnology.
  • Abhishek Singharoy of Arizona State University, whose team is investigating the mechanics of a biological motor called ATP synthase in all-atom detail, a study which may aid the design of bioinspired clean energy technology.
  • Gaute Hagen of Oak Ridge National Laboratory, whose team is calculating the forces within atomic nuclei to study phenomena such as neutrinoless double-beta decay, a hypothesized form of radioactive decay.
  • Joe Oefelein of Georgia Tech, whose team is carrying out combustion simulations that closely match engine operating conditions to inform the design of fuel-efficient, low-emission engines.”

The Gordon Bell Prize winner will be announced at the SC2018 in Dallas in November; as you may know it’s awarded each year by the Association of Computing Machinery (ACM) to recognize outstanding achievement in high-performance computing. “The purpose of the award is to track the progress over time of parallel computing, with particular emphasis on rewarding innovation in applying high-performance computing to applications in science, engineering, and large-scale data analytics…Financial support of the $10,000 award is provided by Gordon Bell, a pioneer in high-performance and parallel computing,” says ACM.

Link to article on OCLF web site: https://www.olcf.ornl.gov/2018/09/17/uncharted-territory/

Link to Nvidia blog: https://blogs.nvidia.com/blog/2018/09/17/nvidia-volta-tensor-core-gpus-gordon-bell-finalists/

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

NOAA Announces Major Upgrade to Ensemble Forecast Model, Extends Range to 35 Days

September 23, 2020

A bit over a year ago, the United States’ Global Forecast System (GFS) received a major upgrade: a new dynamical core – its first in 40 years – called the finite-volume cubed-sphere, or FV3. Now, the National Oceanic and Atmospheric Administration (NOAA) is bringing the FV3 dynamical core to... Read more…

By Oliver Peckham

AI Silicon Startup Graphcore Launches Channel Partner Program

September 23, 2020

AI compute platform vendor Graphcore has launched its first formal global channel partner program to promote and boost the sales of its AI processors and blade computing products. The formalized, all-new Graphcore Elite Partner Program follows the company’s past history of working with several... Read more…

By Todd R. Weiss

Arm Targets HPC with New Neoverse Platforms

September 22, 2020

UK-based semiconductor design company Arm today teased details of its Neoverse roadmap, introducing V1 (codenamed Zeus) and N2 (codenamed Perseus), Arm’s second generation N-series platform. The chip IP vendor said the new platforms will deliver 50 percent and 40 percent more... Read more…

By Tiffany Trader

Microsoft’s Azure Quantum Platform Now Offers Toshiba’s ‘Simulated Bifurcation Machine’

September 22, 2020

While pure-play quantum computing (QC) gets most of the QC-related attention, there’s also been steady progress adapting quantum methods for select use on classical computers. Today, Microsoft announced that Toshiba’ Read more…

By John Russell

Oracle Cloud Deepens HPC Embrace with Launch of A100 Instances, Plans for Arm, More 

September 22, 2020

Oracle Cloud Infrastructure (OCI) continued its steady ramp-up of HPC capabilities today with a flurry of announcements. Topping the list is general availability of instances with Nvidia’s newest GPU, the A100. OCI als Read more…

By John Russell

AWS Solution Channel

The Water Institute of the Gulf runs compute-heavy storm surge and wave simulations on AWS

The Water Institute of the Gulf (Water Institute) runs its storm surge and wave analysis models on Amazon Web Services (AWS)—a task that sometimes requires large bursts of compute power. Read more…

Intel® HPC + AI Pavilion

Berlin Institute of Health: Putting HPC to Work for the World

Researchers from the Center for Digital Health at the Berlin Institute of Health (BIH) are using science to understand the pathophysiology of COVID-19, which can help to inform the development of targeted treatments. Read more…

IBM, CQC Enable Cloud-based Quantum Random Number Generation

September 21, 2020

IBM and Cambridge Quantum Computing (CQC) have partnered to achieve progress on one of the major business aspirations for quantum computing – the goal of generating verified, truly random numbers that can be used for a Read more…

By Todd R. Weiss

NOAA Announces Major Upgrade to Ensemble Forecast Model, Extends Range to 35 Days

September 23, 2020

A bit over a year ago, the United States’ Global Forecast System (GFS) received a major upgrade: a new dynamical core – its first in 40 years – called the finite-volume cubed-sphere, or FV3. Now, the National Oceanic and Atmospheric Administration (NOAA) is bringing the FV3 dynamical core to... Read more…

By Oliver Peckham

Arm Targets HPC with New Neoverse Platforms

September 22, 2020

UK-based semiconductor design company Arm today teased details of its Neoverse roadmap, introducing V1 (codenamed Zeus) and N2 (codenamed Perseus), Arm’s second generation N-series platform. The chip IP vendor said the new platforms will deliver 50 percent and 40 percent more... Read more…

By Tiffany Trader

Oracle Cloud Deepens HPC Embrace with Launch of A100 Instances, Plans for Arm, More 

September 22, 2020

Oracle Cloud Infrastructure (OCI) continued its steady ramp-up of HPC capabilities today with a flurry of announcements. Topping the list is general availabilit Read more…

By John Russell

European Commission Declares €8 Billion Investment in Supercomputing

September 18, 2020

Just under two years ago, the European Commission formalized the EuroHPC Joint Undertaking (JU): a concerted HPC effort (comprising 32 participating states at c Read more…

By Oliver Peckham

Google Hires Longtime Intel Exec Bill Magro to Lead HPC Strategy

September 18, 2020

In a sign of the times, another prominent HPCer has made a move to a hyperscaler. Longtime Intel executive Bill Magro joined Google as chief technologist for hi Read more…

By Tiffany Trader

Future of Fintech on Display at HPC + AI Wall Street

September 17, 2020

Those who tuned in for Tuesday's HPC + AI Wall Street event got a peak at the future of fintech and lively discussion of topics like blockchain, AI for risk man Read more…

By Alex Woodie, Tiffany Trader and Todd R. Weiss

IBM’s Quantum Race to One Million Qubits

September 15, 2020

IBM today outlined its ambitious quantum computing technology roadmap at its virtual Quantum Summit. The eye-popping million qubit number is still far out, agrees IBM, but perhaps not that far out. Just as eye-popping is IBM’s nearer-term plan for a 1,000-plus qubit system named Condor... Read more…

By John Russell

Nvidia Commits to Buy Arm for $40B

September 14, 2020

Nvidia is acquiring semiconductor design company Arm Ltd. for $40 billion from SoftBank in a blockbuster deal that catapults the GPU chipmaker to a dominant position in the datacenter while helping troubled SoftBank reverse its financial woes. The deal, which has been rumored for... Read more…

By Todd R. Weiss and George Leopold

Supercomputer-Powered Research Uncovers Signs of ‘Bradykinin Storm’ That May Explain COVID-19 Symptoms

July 28, 2020

Doctors and medical researchers have struggled to pinpoint – let alone explain – the deluge of symptoms induced by COVID-19 infections in patients, and what Read more…

By Oliver Peckham

Nvidia Said to Be Close on Arm Deal

August 3, 2020

GPU leader Nvidia Corp. is in talks to buy U.K. chip designer Arm from parent company Softbank, according to several reports over the weekend. If consummated Read more…

By George Leopold

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

By Doug Black

Intel’s 7nm Slip Raises Questions About Ponte Vecchio GPU, Aurora Supercomputer

July 30, 2020

During its second-quarter earnings call, Intel announced a one-year delay of its 7nm process technology, which it says it will create an approximate six-month shift for its CPU product timing relative to prior expectations. The primary issue is a defect mode in the 7nm process that resulted in yield degradation... Read more…

By Tiffany Trader

Google Hires Longtime Intel Exec Bill Magro to Lead HPC Strategy

September 18, 2020

In a sign of the times, another prominent HPCer has made a move to a hyperscaler. Longtime Intel executive Bill Magro joined Google as chief technologist for hi Read more…

By Tiffany Trader

HPE Keeps Cray Brand Promise, Reveals HPE Cray Supercomputing Line

August 4, 2020

The HPC community, ever-affectionate toward Cray and its eponymous founder, can breathe a (virtual) sigh of relief. The Cray brand will live on, encompassing th Read more…

By Tiffany Trader

Neocortex Will Be First-of-Its-Kind 800,000-Core AI Supercomputer

June 9, 2020

Pittsburgh Supercomputing Center (PSC - a joint research organization of Carnegie Mellon University and the University of Pittsburgh) has won a $5 million award Read more…

By Tiffany Trader

European Commission Declares €8 Billion Investment in Supercomputing

September 18, 2020

Just under two years ago, the European Commission formalized the EuroHPC Joint Undertaking (JU): a concerted HPC effort (comprising 32 participating states at c Read more…

By Oliver Peckham

Leading Solution Providers

Contributors

Supercomputer Modeling Tests How COVID-19 Spreads in Grocery Stores

April 8, 2020

In the COVID-19 era, many people are treating simple activities like getting gas or groceries with caution as they try to heed social distancing mandates and protect their own health. Still, significant uncertainty surrounds the relative risk of different activities, and conflicting information is prevalent. A team of Finnish researchers set out to address some of these uncertainties by... Read more…

By Oliver Peckham

Oracle Cloud Infrastructure Powers Fugaku’s Storage, Scores IO500 Win

August 28, 2020

In June, RIKEN shook the supercomputing world with its Arm-based, Fujitsu-built juggernaut: Fugaku. The system, which weighs in at 415.5 Linpack petaflops, topp Read more…

By Oliver Peckham

Google Cloud Debuts 16-GPU Ampere A100 Instances

July 7, 2020

On the heels of the Nvidia’s Ampere A100 GPU launch in May, Google Cloud is announcing alpha availability of the A100 “Accelerator Optimized” VM A2 instance family on Google Compute Engine. The instances are powered by the HGX A100 16-GPU platform, which combines two HGX A100 8-GPU baseboards using... Read more…

By Tiffany Trader

DOD Orders Two AI-Focused Supercomputers from Liqid

August 24, 2020

The U.S. Department of Defense is making a big investment in data analytics and AI computing with the procurement of two HPC systems that will provide the High Read more…

By Tiffany Trader

Microsoft Azure Adds A100 GPU Instances for ‘Supercomputer-Class AI’ in the Cloud

August 19, 2020

Microsoft Azure continues to infuse its cloud platform with HPC- and AI-directed technologies. Today the cloud services purveyor announced a new virtual machine Read more…

By Tiffany Trader

Japan’s Fugaku Tops Global Supercomputing Rankings

June 22, 2020

A new Top500 champ was unveiled today. Supercomputer Fugaku, the pride of Japan and the namesake of Mount Fuji, vaulted to the top of the 55th edition of the To Read more…

By Tiffany Trader

Joliot-Curie Supercomputer Used to Build First Full, High-Fidelity Aircraft Engine Simulation

July 14, 2020

When industrial designers plan the design of a new element of a vehicle’s propulsion or exterior, they typically use fluid dynamics to optimize airflow and in Read more…

By Oliver Peckham

Intel Speeds NAMD by 1.8x: Saves Xeon Processor Users Millions of Compute Hours

August 12, 2020

Potentially saving datacenters millions of CPU node hours, Intel and the University of Illinois at Urbana–Champaign (UIUC) have collaborated to develop AVX-512 optimizations for the NAMD scalable molecular dynamics code. These optimizations will be incorporated into release 2.15 with patches available for earlier versions. Read more…

By Rob Farber

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This