ORNL to Harness Computational Power of Two DGX-2s

February 7, 2019

Feb. 7 — As home to three top-ranked supercomputers of the last decade, the US Department of Energy’s (DOE’s) Oak Ridge National Laboratory (ORNL) has become synonymous with scientific computing at the largest scales.

Getting the most out of these science machines, however, requires a willingness to experiment with problems and systems of every size and scale. This is especially important as technology vendors introduce new system architectures and as scientists’ problem-solving toolkit expands to include artificial intelligence (AI) and advanced data analysis.

In that spirit, ORNL recently installed two NVIDIA DGX-2 systems, powerful GPU-accelerated appliances that will provide researchers with enhanced opportunities to conduct science—machine learning and data-intensive workloads in particular. The appliances will also provide an on-ramp to ORNL’s Summit—the world’s most powerful supercomputer—by enabling smaller and more experimental projects to be developed and tested before they run on the 200-petaflop machine. The DGX-2 appliances reside in the laboratory’s Compute and Data Environment for Science (CADES), which offers compute and data services for ORNL researchers.

“As Summit enters production, these DGX-2 systems supply ORNL with exploratory multipurpose computing resources,” said CADES director Arjun Shankar. “Early results suggest the DGX-2s will provide novel opportunities in data analysis, machine learning, and modeling and simulation that support the AI-driven transformation that is changing how science is conducted.”

The DGX-2 represents the latest step-change in AI appliances, housing 16 fully interconnected NVIDIA Tesla V100 GPUs with increased GPU memory, a powerful combination that expands the types of problems scientists can tackle in a unified environment. In addition to a standard DGX-2, ORNL received the newly available DGX-2H, which offers higher performance through upgraded CPUs and faster-clocked GPUs.

ORNL’s DGX-2 and DGX-2H appliances provide laboratory researchers with enhanced opportunities to conduct science and serve as an onramp to ORNL’s Summit supercomputer

Since NVIDIA debuted the DGX line in 2016, ORNL has deployed the appliances throughout the laboratory to connect researchers with a platform that excels at executing machine learning techniques with the potential to automate some of the time-intensive analysis inherent in research. This is especially relevant to ORNL’s world-class experimental facilities, such as the Spallation Neutron Source, which produce large, unique datasets in need of analysis and automated data workflows.

Appliance for Science

In late 2018, Arvind Ramanathan, a staff scientist in ORNL’s Computational Sciences and Engineering Division, and his team were among the first groups to receive extended time on the DGX-2s. The team used the opportunity to train and optimize algorithms that belong to a class of machine learning called reinforcement learning, in which an “agent” attempts to master its environment by performing actions and evaluating the results without any preexisting knowledge.

Reinforcement-learning algorithms, famously showcased by Google’s AlphaGo program, have proven capable of achieving prescribed goals, such as winning games, but optimizing the preset parameters that control their decision-making can be difficult. Running multiple algorithms simultaneously on the DGX-2 systems allowed Ramanathan’s team to identify superior optimization strategies via an ORNL-developed software called HyperSpace in a fraction of the time another system would require.

“We couldn’t have done this without a DGX-2 because the problem space that we were exploring was so large and sample inefficient,” Ramanathan said. “Because these GPUs can essentially be used in a unified way, we can do things that are much more difficult to do on other systems, especially in terms of moving data and doing analysis.”

Though ORNL is known for conducting leadership-scale science on its massively parallel supercomputers, there are instances when an innovative smaller machine can be useful. Refining algorithms on the DGX-2 can improve researchers’ confidence that their AI software is ready to be deployed later at scale. Additionally, workloads that may be poorly suited to run on a supercomputer—jobs that don’t scale or jobs that need to run for extended periods of time, for example—could be carried out on a DGX-2 appliance.

The DGX-2s also have something to offer to traditional modeling and simulation. Researchers can run simulations side by side with AI to extend simulations further than they would otherwise go, using AI-recognized patterns in the data to “steer” the simulation correctly. A project supported by ORNL’s Laboratory Directed Research and Development program is dedicated to a molecular dynamics framework called Molecules that can execute AI-informed simulation.

“Traditionally, running AI side by side with simulation would be too expensive,” Ramanathan said, “but state-of-the-art systems like Summit and the DGX-2 enable this in such a way that we can think of this arrangement as a fused workflow in some sense.”

Currently, CADES staff are working to integrate the appliances into the data center’s shared environment so researchers can submit jobs as easily as they can to any other CADES resource. The two DGX-2 systems have been connected by a dedicated EDR InfiniBand network to combine the systems’ capabilities.

“The idea is that researchers will be able to schedule up to 32 GPUs at one time to run in parallel,” said CADES team lead Brian Zachary.

HyperSpace software development is part of the CANcer Distributed Learning Environment project, a cancer research effort supported by the Exascale Computing Project.

UT-Battelle LLC manages ORNL for DOE’s Office of Science. The Office of Science is the single largest supporter of basic research in the physical sciences in the United States and is working to address some of the most pressing challenges of our time. For more information, please visit https://science.energy.gov.


Source: Jonathan Hines, ORNL

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industry updates delivered to you every week!

MLPerf Inference 4.0 Results Showcase GenAI; Nvidia Still Dominates

March 28, 2024

There were no startling surprises in the latest MLPerf Inference benchmark (4.0) results released yesterday. Two new workloads — Llama 2 and Stable Diffusion XL — were added to the benchmark suite as MLPerf continues Read more…

Q&A with Nvidia’s Chief of DGX Systems on the DGX-GB200 Rack-scale System

March 27, 2024

Pictures of Nvidia's new flagship mega-server, the DGX GB200, on the GTC show floor got favorable reactions on social media for the sheer amount of computing power it brings to artificial intelligence.  Nvidia's DGX Read more…

Call for Participation in Workshop on Potential NSF CISE Quantum Initiative

March 26, 2024

Editor’s Note: Next month there will be a workshop to discuss what a quantum initiative led by NSF’s Computer, Information Science and Engineering (CISE) directorate could entail. The details are posted below in a Ca Read more…

Waseda U. Researchers Reports New Quantum Algorithm for Speeding Optimization

March 25, 2024

Optimization problems cover a wide range of applications and are often cited as good candidates for quantum computing. However, the execution time for constrained combinatorial optimization applications on quantum device Read more…

NVLink: Faster Interconnects and Switches to Help Relieve Data Bottlenecks

March 25, 2024

Nvidia’s new Blackwell architecture may have stolen the show this week at the GPU Technology Conference in San Jose, California. But an emerging bottleneck at the network layer threatens to make bigger and brawnier pro Read more…

Who is David Blackwell?

March 22, 2024

During GTC24, co-founder and president of NVIDIA Jensen Huang unveiled the Blackwell GPU. This GPU itself is heavily optimized for AI work, boasting 192GB of HBM3E memory as well as the the ability to train 1 trillion pa Read more…

MLPerf Inference 4.0 Results Showcase GenAI; Nvidia Still Dominates

March 28, 2024

There were no startling surprises in the latest MLPerf Inference benchmark (4.0) results released yesterday. Two new workloads — Llama 2 and Stable Diffusion Read more…

Q&A with Nvidia’s Chief of DGX Systems on the DGX-GB200 Rack-scale System

March 27, 2024

Pictures of Nvidia's new flagship mega-server, the DGX GB200, on the GTC show floor got favorable reactions on social media for the sheer amount of computing po Read more…

NVLink: Faster Interconnects and Switches to Help Relieve Data Bottlenecks

March 25, 2024

Nvidia’s new Blackwell architecture may have stolen the show this week at the GPU Technology Conference in San Jose, California. But an emerging bottleneck at Read more…

Who is David Blackwell?

March 22, 2024

During GTC24, co-founder and president of NVIDIA Jensen Huang unveiled the Blackwell GPU. This GPU itself is heavily optimized for AI work, boasting 192GB of HB Read more…

Nvidia Looks to Accelerate GenAI Adoption with NIM

March 19, 2024

Today at the GPU Technology Conference, Nvidia launched a new offering aimed at helping customers quickly deploy their generative AI applications in a secure, s Read more…

The Generative AI Future Is Now, Nvidia’s Huang Says

March 19, 2024

We are in the early days of a transformative shift in how business gets done thanks to the advent of generative AI, according to Nvidia CEO and cofounder Jensen Read more…

Nvidia’s New Blackwell GPU Can Train AI Models with Trillions of Parameters

March 18, 2024

Nvidia's latest and fastest GPU, codenamed Blackwell, is here and will underpin the company's AI plans this year. The chip offers performance improvements from Read more…

Nvidia Showcases Quantum Cloud, Expanding Quantum Portfolio at GTC24

March 18, 2024

Nvidia’s barrage of quantum news at GTC24 this week includes new products, signature collaborations, and a new Nvidia Quantum Cloud for quantum developers. Wh Read more…

Alibaba Shuts Down its Quantum Computing Effort

November 30, 2023

In case you missed it, China’s e-commerce giant Alibaba has shut down its quantum computing research effort. It’s not entirely clear what drove the change. Read more…

Nvidia H100: Are 550,000 GPUs Enough for This Year?

August 17, 2023

The GPU Squeeze continues to place a premium on Nvidia H100 GPUs. In a recent Financial Times article, Nvidia reports that it expects to ship 550,000 of its lat Read more…

Shutterstock 1285747942

AMD’s Horsepower-packed MI300X GPU Beats Nvidia’s Upcoming H200

December 7, 2023

AMD and Nvidia are locked in an AI performance battle – much like the gaming GPU performance clash the companies have waged for decades. AMD has claimed it Read more…

DoD Takes a Long View of Quantum Computing

December 19, 2023

Given the large sums tied to expensive weapon systems – think $100-million-plus per F-35 fighter – it’s easy to forget the U.S. Department of Defense is a Read more…

Synopsys Eats Ansys: Does HPC Get Indigestion?

February 8, 2024

Recently, it was announced that Synopsys is buying HPC tool developer Ansys. Started in Pittsburgh, Pa., in 1970 as Swanson Analysis Systems, Inc. (SASI) by John Swanson (and eventually renamed), Ansys serves the CAE (Computer Aided Engineering)/multiphysics engineering simulation market. Read more…

Choosing the Right GPU for LLM Inference and Training

December 11, 2023

Accelerating the training and inference processes of deep learning models is crucial for unleashing their true potential and NVIDIA GPUs have emerged as a game- Read more…

Intel’s Server and PC Chip Development Will Blur After 2025

January 15, 2024

Intel's dealing with much more than chip rivals breathing down its neck; it is simultaneously integrating a bevy of new technologies such as chiplets, artificia Read more…

Baidu Exits Quantum, Closely Following Alibaba’s Earlier Move

January 5, 2024

Reuters reported this week that Baidu, China’s giant e-commerce and services provider, is exiting the quantum computing development arena. Reuters reported � Read more…

Leading Solution Providers

Contributors

Comparing NVIDIA A100 and NVIDIA L40S: Which GPU is Ideal for AI and Graphics-Intensive Workloads?

October 30, 2023

With long lead times for the NVIDIA H100 and A100 GPUs, many organizations are looking at the new NVIDIA L40S GPU, which it’s a new GPU optimized for AI and g Read more…

Shutterstock 1179408610

Google Addresses the Mysteries of Its Hypercomputer 

December 28, 2023

When Google launched its Hypercomputer earlier this month (December 2023), the first reaction was, "Say what?" It turns out that the Hypercomputer is Google's t Read more…

AMD MI3000A

How AMD May Get Across the CUDA Moat

October 5, 2023

When discussing GenAI, the term "GPU" almost always enters the conversation and the topic often moves toward performance and access. Interestingly, the word "GPU" is assumed to mean "Nvidia" products. (As an aside, the popular Nvidia hardware used in GenAI are not technically... Read more…

Shutterstock 1606064203

Meta’s Zuckerberg Puts Its AI Future in the Hands of 600,000 GPUs

January 25, 2024

In under two minutes, Meta's CEO, Mark Zuckerberg, laid out the company's AI plans, which included a plan to build an artificial intelligence system with the eq Read more…

Google Introduces ‘Hypercomputer’ to Its AI Infrastructure

December 11, 2023

Google ran out of monikers to describe its new AI system released on December 7. Supercomputer perhaps wasn't an apt description, so it settled on Hypercomputer Read more…

China Is All In on a RISC-V Future

January 8, 2024

The state of RISC-V in China was discussed in a recent report released by the Jamestown Foundation, a Washington, D.C.-based think tank. The report, entitled "E Read more…

Intel Won’t Have a Xeon Max Chip with New Emerald Rapids CPU

December 14, 2023

As expected, Intel officially announced its 5th generation Xeon server chips codenamed Emerald Rapids at an event in New York City, where the focus was really o Read more…

IBM Quantum Summit: Two New QPUs, Upgraded Qiskit, 10-year Roadmap and More

December 4, 2023

IBM kicks off its annual Quantum Summit today and will announce a broad range of advances including its much-anticipated 1121-qubit Condor QPU, a smaller 133-qu Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire