Summit Supercomputer is Already Making its Mark on Science

By John Russell

September 20, 2018

Summit, now the fastest supercomputer in the world, is quickly making its mark in science – five of the six finalists just announced for the prestigious 2018 Gordon Bell Prize used Summit in their work. That’s impressive given that Summit only began full operation in early summer. Also noteworthy is Summit’s heterogeneous architecture which leverages IBM’s Power9 CPU, Nvidia V100 GPUs, and fast interconnect technology from Mellanox to accommodate traditional simulation workloads as well as mixed-precision workloads associated with AI and data analytics.

By now, Summit needs little introduction having topped the most recent Top500 list. Located at the Oak Ridge Leadership Computing Facility (OCLF), it cost an estimated $200 million to build as part of the DoE CORAL procurement program. It’s been heralded as the world’s most powerful supercomputer at 200 petaflops theoretical peak for high-performance computing workloads and 3.3 peak exaops for emerging AI workloads. (Sierra, the similarly architected but somewhat smaller, 125 petaflops theoretical peak machine based at Lawrence Livermore National Laboratory, was also used in some of the cited research.)

“These Gordon Bell finalists are an encouraging preview of the challenges users will be able to tackle on Summit when formal allocation programs begin in 2019,” said OLCF director of science Jack Wells. “Of particular note is the system’s ability to handle large volumes of data at scale, whether that be processing and analyzing experimental data or training artificial intelligence software to carry out specialized tasks.”

Nvidia and IBM are, understandably, ecstatic over Summit’s progress.

In a Nvidia blog, product manager Geetika Gupta wrote, “The revolutionary accelerators enable multi-precision computing that fuses the highly precise calculations to tackle the challenges of high performance computing with the efficient processing required for deep learning…[H]alf of the six projects included NVIDIA researchers who were heavily involved with the code development and performance tuning.”

Dave Turek, IBM Cognitive Systems VP, said “IBM designed Summit and Sierra to be data-centric, heterogeneous systems that maximized data flow for optimal application performance. The industry-leading IO features of IBM POWER9 processors allow for data to flow in and out of Summit’s GPUs to achieve the unprecedented level of performance demonstrated by these Gordon Bell finalists.”

They can, perhaps, be forgiven a little excess enthusiasm. These machines are difficult to design and build. Clearly, Summit’s early success is more evidence that heterogeneous architectures that leverage accelerators are likely to dominate high-end computing going forward.

Here is a lightly edited excerpt from an OCLF article describing the finalists who used Summit in their research:

  • “Genomics. An ORNL team led by computational systems biologist Dan Jacobson and OLCF computational scientist Wayne Joubert that developed a genomics algorithm capable of using mixed-precision arithmetic to attain exascale speeds. On Summit, the team’s Combinatorial Metrics application achieved a peak throughput of 2.36 exaops—or 2.36 billion billion calculations per second, the fastest science application ever reported. Jacobson’s work compares genetic variations within a population to uncover hidden networks of genes that contribute to complex traits. One condition Jacobson’s team is studying is opioid addiction, which has been linked to the deaths of more than 49,000 people in the United States in 2017.
  • Earthquake Simulation. A team from the University of Tokyo led by associate professor Tsuyoshi Ichimura that applied artificial intelligence (AI) and mixed-precision arithmetic to accelerate the simulation of earthquake physics in urban environments. As cities continue to grow, preparedness and improved understanding of ground-shaking’s effects on buildings and urban infrastructure become increasingly important. On Summit, the Tokyo team expanded on its 2014 algorithm, which was also a Gordon Bell Finalist, to achieve a fourfold speedup and to couple the shaking of ground and urban structures during large earthquakes into the same simulation.
  • Extreme Weather. A Lawrence Berkeley National Laboratory-led collaboration that trained a deep neural network to identify extreme weather patterns from high-resolution climate simulations.The team, led by Berkeley data scientist Prabhat, plans to use the AI software to predict how extreme weather is likely to change in the future. By tapping into the specialized tensor cores built into Summit’s NVIDIA GPUs at scale, the Berkeley team achieved a peak performance of 1.13 exaops, the fastest deep-learning algorithm yet reported. Though the team applied its work to climate science, many of its innovations can be adapted for other deep-learning applications.
  • Materials Science. An ORNL team led by data scientist Robert Patton that scaled a deep-learning technique on Summit to produce intelligent software that can automatically identify materials’ atomic-level information from electron microscopy With advanced microscopes capable of producing hundreds of images per day, real-time feedback supplied by AI could give scientists the ability to fabricate materials at the atomic level. Scaled across 4,200 nodes, the team’s MENNDL algorithm achieved a speed of 152.5 petaflops with an estimated performance rate of 167 petaflops across the whole machine.
  • Physics. A team from Lawrence Berkeley and Lawrence Livermore National Laboratories led by physicists André Walker-Loud and Pavlos Vranas that developed improved algorithms to help scientists predict the lifetime of neutrons and answer fundamental questions about the universe. The team built upon its previous work using lattice quantum chromodynamics—a numerical method for calculating the underlying physics of the subatomic particles that make up protons and neutrons. In addition to optimized GPU software, the team developed lightweight, application-agnostic management software capable of managing hundreds of thousands of tasks. Using GPU-accelerated systems Sierra at Lawrence Livermore and the OLCF’s Summit, the team was able to start 1,056 four-node jobs on 4,224 nodes in 5 minutes, achieving a machine-to-machine speedup of factors of 10 and 15, respectively, over the OLCF’s previous leadership-class system, Titan. The achievement supplies nuclear physicists with the necessary computational power to support the experimental search for new physics.”

We’d be remiss not to mention the sixth Gordon finalist; it’s from a group of researchers from China who developed a graph processing framework (ShenTu) adapted for use on HPC resources. Here is a description of that very impressive work (ShenTu: Processing Multi-Trillion Edge Graphs on Millions of Cores in Seconds) taken from the SC18 web site.

“DescriptionGraphs are an important abstraction used in many scientific fields. With the magnitude of graph-structured data constantly increasing, effective data analytics requires efficient and scalable graph processing systems. Although HPC systems have long been used for scientific computing, people have only recently started to assess their potential for graph processing, a workload with inherent load imbalance, lack of locality, and access irregularity. We propose ShenTu, the first general-purpose graph processing framework that can efficiently utilize an entire petascale system to process multi-trillion edge graphs in seconds. ShenTu embodies four key innovations: hardware specializing, supernode routing, on-chip sorting, and degree-aware messaging, which together enable its unprecedented performance and scalability. It can traverse an unprecedented 70-trillion-edge graph in seconds.”

Jack Wells, OLCF

But back to Summit. Wells shared with HPCwire some of the distinguishing advantages Summit provides generally and some of which were leveraged by the Gordon Bell finalists. He noted two of the teams “were highly targeting the system’s mixed precision capabilities. The finite-element application explored ways mixed precision can boost performance by minimizing communication.”

Wells singled out four areas where Summit stands out:

  • “Because of the NVLink the users can use more system memory than they could on Titan. Connecting the Volta GPUs to the Power9 CPU using NVLink provides much higher bandwidth than possible with PCIe Gen4. NVLink provides enough bandwidth so that the three GPUs can saturate the Power9’s memory bandwidth. This enables app to use system memory in addition to the GPU’s HBM which is not practical on systems like Titan with PCIe-attached GPUs.
  • “The burst buffers, a reliable, high-speed storage layer that sits between the machine’s computing and file systems, significantly benefitted some teams who used it as a read accelerator rather than a write accelerator. Machine learning applications are read-heavy, so duplicating and moving data to the node local scratch memory was much faster than having it in GPFS.”
  • “With this generation of InfiniBand, Mellanox has vastly improved its Adaptive Routing which greatly reduces congestion and allows applications to scale better. Additionally, one of the teams extensively took advantage of Mellanox’s switch-based collective operations, which shaved significant time off synchronization operations that typically limit an application’s scalability.
  • “The Volta’s high bandwidth memory is very important. Summit’s nodes have more HBM than any other comparable system, which will allow our users to solve Gordon-Bell sized problems.”

On the software side, Summit users benefit from OCLF’s past experience with accelerators. Wells, noted, “Summit, like Titan, is a GPU-based system. Previous efforts to port and optimize codes for Titan have been beneficial for helping get codes ready for Summit. However, the Summit node is more complex, for example having multiple GPUs per node and having new features such as burst buffers and the GPU tensor cores. Adapting to this new node architecture has required effort by the code teams.”

Another issue for the Gordon Bell users, said Wells, is “[They only] had access to our relatively small test-and-development file system, not the full production file system that is undergoing acceptance testing these days. So, they had to work around this limitation. Also, the system software was still undergoing testing and debugging, so these teams were helping us identify such shortcomings and fix them.”

Information about the Summit stack – which includes XL, GNU, LLVM, PGI and NVCC compilers, LMOD, Spectrum MPI, ESSL, CUDA, LSF and JSM – is available on the web. The operating system is Red Hat Linux.

Obviously, these are early day for Summit which is still under preparation for full acceptance testing said Wells: “Users currently do not have access to the system as we attempt to finish this task. The IBM system is planned to be made available to the research community through DOE’s user programs beginning with allocations made under the Innovative and Novel Computational Impact on Theory and Experiment (INCITE) user program that will start in January 2019.”

That doesn’t mean plans aren’t afoot. They are. “For the past three years, teams have been preparing their applications to run on Summit,” said Wells. “A selection of the principal investigators of these application readiness teams includes:

  • Salman Habib of Argonne National Laboratory, whose team is modeling the large-scale structure and distribution of matter over the 13-billion-year lifespan of the universe.
  • Dmytro Bykov of Oak Ridge National Laboratory, whose team aims to describe the electronic structure of large molecular systems using quantum chemistry techniques, with targeted applications that include pharmacology and nanotechnology.
  • Abhishek Singharoy of Arizona State University, whose team is investigating the mechanics of a biological motor called ATP synthase in all-atom detail, a study which may aid the design of bioinspired clean energy technology.
  • Gaute Hagen of Oak Ridge National Laboratory, whose team is calculating the forces within atomic nuclei to study phenomena such as neutrinoless double-beta decay, a hypothesized form of radioactive decay.
  • Joe Oefelein of Georgia Tech, whose team is carrying out combustion simulations that closely match engine operating conditions to inform the design of fuel-efficient, low-emission engines.”

The Gordon Bell Prize winner will be announced at the SC2018 in Dallas in November; as you may know it’s awarded each year by the Association of Computing Machinery (ACM) to recognize outstanding achievement in high-performance computing. “The purpose of the award is to track the progress over time of parallel computing, with particular emphasis on rewarding innovation in applying high-performance computing to applications in science, engineering, and large-scale data analytics…Financial support of the $10,000 award is provided by Gordon Bell, a pioneer in high-performance and parallel computing,” says ACM.

Link to article on OCLF web site: https://www.olcf.ornl.gov/2018/09/17/uncharted-territory/

Link to Nvidia blog: https://blogs.nvidia.com/blog/2018/09/17/nvidia-volta-tensor-core-gpus-gordon-bell-finalists/

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industry updates delivered to you every week!

Nvidia’s New Blackwell GPU Can Train AI Models with Trillions of Parameters

March 18, 2024

Nvidia's latest and fastest GPU, code-named Blackwell, is here and will underpin the company's AI plans this year. The chip offers performance improvements from its predecessors, including the red-hot H100 and A100 GPUs. Read more…

Nvidia Showcases Quantum Cloud, Expanding Quantum Portfolio at GTC24

March 18, 2024

Nvidia’s barrage of quantum news at GTC24 this week includes new products, signature collaborations, and a new Nvidia Quantum Cloud for quantum developers. While Nvidia may not spring to mind when thinking of the quant Read more…

2024 Winter Classic: Meet the HPE Mentors

March 18, 2024

The latest installment of the 2024 Winter Classic Studio Update Show features our interview with the HPE mentor team who introduced our student teams to the joys (and potential sorrows) of the HPL (LINPACK) and accompany Read more…

Houston We Have a Solution: Addressing the HPC and Tech Talent Gap

March 15, 2024

Generations of Houstonian teachers, counselors, and parents have either worked in the aerospace industry or know people who do - the prospect of entering the field was normalized for boys in 1969 when the Apollo 11 missi Read more…

Apple Buys DarwinAI Deepening its AI Push According to Report

March 14, 2024

Apple has purchased Canadian AI startup DarwinAI according to a Bloomberg report today. Apparently the deal was done early this year but still hasn’t been publicly announced according to the report. Apple is preparing Read more…

Survey of Rapid Training Methods for Neural Networks

March 14, 2024

Artificial neural networks are computing systems with interconnected layers that process and learn from data. During training, neural networks utilize optimization algorithms to iteratively refine their parameters until Read more…

Nvidia’s New Blackwell GPU Can Train AI Models with Trillions of Parameters

March 18, 2024

Nvidia's latest and fastest GPU, code-named Blackwell, is here and will underpin the company's AI plans this year. The chip offers performance improvements from Read more…

Nvidia Showcases Quantum Cloud, Expanding Quantum Portfolio at GTC24

March 18, 2024

Nvidia’s barrage of quantum news at GTC24 this week includes new products, signature collaborations, and a new Nvidia Quantum Cloud for quantum developers. Wh Read more…

Houston We Have a Solution: Addressing the HPC and Tech Talent Gap

March 15, 2024

Generations of Houstonian teachers, counselors, and parents have either worked in the aerospace industry or know people who do - the prospect of entering the fi Read more…

Survey of Rapid Training Methods for Neural Networks

March 14, 2024

Artificial neural networks are computing systems with interconnected layers that process and learn from data. During training, neural networks utilize optimizat Read more…

PASQAL Issues Roadmap to 10,000 Qubits in 2026 and Fault Tolerance in 2028

March 13, 2024

Paris-based PASQAL, a developer of neutral atom-based quantum computers, yesterday issued a roadmap for delivering systems with 10,000 physical qubits in 2026 a Read more…

India Is an AI Powerhouse Waiting to Happen, but Challenges Await

March 12, 2024

The Indian government is pushing full speed ahead to make the country an attractive technology base, especially in the hot fields of AI and semiconductors, but Read more…

Charles Tahan Exits National Quantum Coordination Office

March 12, 2024

(March 1, 2024) My first official day at the White House Office of Science and Technology Policy (OSTP) was June 15, 2020, during the depths of the COVID-19 loc Read more…

AI Bias In the Spotlight On International Women’s Day

March 11, 2024

What impact does AI bias have on women and girls? What can people do to increase female participation in the AI field? These are some of the questions the tech Read more…

Alibaba Shuts Down its Quantum Computing Effort

November 30, 2023

In case you missed it, China’s e-commerce giant Alibaba has shut down its quantum computing research effort. It’s not entirely clear what drove the change. Read more…

Nvidia H100: Are 550,000 GPUs Enough for This Year?

August 17, 2023

The GPU Squeeze continues to place a premium on Nvidia H100 GPUs. In a recent Financial Times article, Nvidia reports that it expects to ship 550,000 of its lat Read more…

Analyst Panel Says Take the Quantum Computing Plunge Now…

November 27, 2023

Should you start exploring quantum computing? Yes, said a panel of analysts convened at Tabor Communications HPC and AI on Wall Street conference earlier this y Read more…

Shutterstock 1285747942

AMD’s Horsepower-packed MI300X GPU Beats Nvidia’s Upcoming H200

December 7, 2023

AMD and Nvidia are locked in an AI performance battle – much like the gaming GPU performance clash the companies have waged for decades. AMD has claimed it Read more…

DoD Takes a Long View of Quantum Computing

December 19, 2023

Given the large sums tied to expensive weapon systems – think $100-million-plus per F-35 fighter – it’s easy to forget the U.S. Department of Defense is a Read more…

Synopsys Eats Ansys: Does HPC Get Indigestion?

February 8, 2024

Recently, it was announced that Synopsys is buying HPC tool developer Ansys. Started in Pittsburgh, Pa., in 1970 as Swanson Analysis Systems, Inc. (SASI) by John Swanson (and eventually renamed), Ansys serves the CAE (Computer Aided Engineering)/multiphysics engineering simulation market. Read more…

Intel’s Server and PC Chip Development Will Blur After 2025

January 15, 2024

Intel's dealing with much more than chip rivals breathing down its neck; it is simultaneously integrating a bevy of new technologies such as chiplets, artificia Read more…

Baidu Exits Quantum, Closely Following Alibaba’s Earlier Move

January 5, 2024

Reuters reported this week that Baidu, China’s giant e-commerce and services provider, is exiting the quantum computing development arena. Reuters reported � Read more…

Leading Solution Providers

Contributors

Choosing the Right GPU for LLM Inference and Training

December 11, 2023

Accelerating the training and inference processes of deep learning models is crucial for unleashing their true potential and NVIDIA GPUs have emerged as a game- Read more…

Training of 1-Trillion Parameter Scientific AI Begins

November 13, 2023

A US national lab has started training a massive AI brain that could ultimately become the must-have computing resource for scientific researchers. Argonne N Read more…

Shutterstock 1179408610

Google Addresses the Mysteries of Its Hypercomputer 

December 28, 2023

When Google launched its Hypercomputer earlier this month (December 2023), the first reaction was, "Say what?" It turns out that the Hypercomputer is Google's t Read more…

Comparing NVIDIA A100 and NVIDIA L40S: Which GPU is Ideal for AI and Graphics-Intensive Workloads?

October 30, 2023

With long lead times for the NVIDIA H100 and A100 GPUs, many organizations are looking at the new NVIDIA L40S GPU, which it’s a new GPU optimized for AI and g Read more…

AMD MI3000A

How AMD May Get Across the CUDA Moat

October 5, 2023

When discussing GenAI, the term "GPU" almost always enters the conversation and the topic often moves toward performance and access. Interestingly, the word "GPU" is assumed to mean "Nvidia" products. (As an aside, the popular Nvidia hardware used in GenAI are not technically... Read more…

Shutterstock 1606064203

Meta’s Zuckerberg Puts Its AI Future in the Hands of 600,000 GPUs

January 25, 2024

In under two minutes, Meta's CEO, Mark Zuckerberg, laid out the company's AI plans, which included a plan to build an artificial intelligence system with the eq Read more…

Google Introduces ‘Hypercomputer’ to Its AI Infrastructure

December 11, 2023

Google ran out of monikers to describe its new AI system released on December 7. Supercomputer perhaps wasn't an apt description, so it settled on Hypercomputer Read more…

China Is All In on a RISC-V Future

January 8, 2024

The state of RISC-V in China was discussed in a recent report released by the Jamestown Foundation, a Washington, D.C.-based think tank. The report, entitled "E Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire