Super-Connecting the Supercomputers – Protect Your Network Investment

By Gilad Shainer, Mellanox Technologies

August 5, 2019

This is the third in the series of articles “Super-Connecting the Supercomputers.” In the first article published on June 10, 2019 in HPCwire [1], we introduced the three interconnect pillars—the connectivity pillar, the network pillar and the communication pillar); in the second article published on July 15, 2019 [2], we discussed the connectivity pillar in detail.

The second pillar, the network pillar, refers to the network protocol, routing capabilities, and other networking functions. Many articles debating the differences between different networking technologies and implementations have been published over the years; for example “Offloading vs. Onloading: The Case of CPU Utilization” [3] and “The Ultimate Debate – Interconnect Offloading Versus Onloading” [4].

We can categorize the networking technologies into two groups – standards-based technologies and proprietary-based technologies. Two examples of standards-based groups are InfiniBand and Ethernet. The proprietary group includes QsNet, Myrinet, Gemini, Seastar, Aries, Tofu, Omni-Path and Slingshot among others.

The ever-growing demand for higher performance in the world of supercomputing requires that interconnect solutions provide increasingly faster speeds, extreme low latency and continuous additions of smart offloading and acceleration engines. In a parallel computing environment, the interconnect is the computer and the heart of the datacenter. Proprietary networks cannot meet the abovementioned needs over time and therefore the lifetime of a proprietary network is between three to five years. In the past, extending the lifetime of the network was possible; however, with the exponential growth of data we want to analyze, together with the increased simulation complexity and the integration of artificial intelligence and deep learning into high performance computing (HPC), the lifetime of the network has been shrinking over time, and is expected to continue to shrink into the future.

Figure 1 – High Performance Computing Interconnect Development

A recent example is Intel’s Omni-Path. The roots of Omni-Path are InfiniBand technology, created by PathScale (InfiniPath adapters) and QLogic/SilverStorm (InfiniBand switch). The PathScale InfiniBand product generation lasted around ten years before Intel transformed it into Omni-Path;  once Omni-Path became a proprietary technology, its lifetime clock started ticking. Three years after its first introduction, Intel has announced that Omni-Path is no longer on the company’s roadmap [5].

There are several challenges and problems that need to be addressed when creating a new proprietary protocol. The first main challenge is to re-create the required software eco-system, including software drivers, operating system support, the creation of communication libraries, and the support of the applications vendors or open source groups. This is a very expensive and long process, and, if required to be done every three years, is a huge burden for HPC end-users. One can assume that those organizations who bought Omni-Path interconnect product in the past, would have chosen differently had they known that their investment, not only in supercomputer hardware purchases, but also in software development, adjustments, settings and troubleshooting, would need to be re-done again..

Another main challenge with proprietary networks is the need to re-invent the basic networking structure and capabilities repeatedly. This places an unnecessary burden not only on the companies doing so, but also on their funding agencies.

In the case of standards-based interconnects, InfiniBand for example, these problems do not exist. Each capability introduced in a previous generation of InfiniBand is carried into future generations, and each generation is backward and forward compatible. The Quality of Service capability for example, an inherent part of the InfiniBand specification, has been in existence since the first generation of InfiniBand, and is being further carried and optimized from one speed generation to the next. All of the software drivers, communication frameworks, native inbox support within the various operating systems, and applications optimizations and tools, continue to leverage hardware support over time, and therefore deliver the highest return on investment for their creators and users.

It is not a surprise that basic network elements, such as Quality of Service or Congestion Control, are promoted as the highlight of new proprietary interconnect technologies because they get re-invented over and over again; in fact, these basic network elements might be  served as the sole “differentiating” item for market publicity. It also does not come as a surprise that the new “benchmarks” created for these basic elements demonstrate, effectively, wasted effort. Just imagine a new car manufacturer announcing today “the invention of a round wheel…”.

On the other hand, these basic network elements are already an integral part of the long-lasting standard technologies. Efforts invested here are not wasted; rather, they enable further progress in their performance, scalability and robustness. A good example of such evolvement is the innovative development of smart In-Network Computing engines, such as Mellanox Scalable Hierarchical Aggregation and Reduction Protocol (SHARP)™ and Self-Healing technologies such as SHIELD, providing higher resiliency of supercomputers.

The early generations of InfiniBand brought the support for full network-transport offload and RDMA— capabilities that enable faster data movement, lower latency and the dramatic reduction in CPU utilization for the sake of networking operations (which translates into more CPU cycles that can be dedicated to the actual application runtime). Later, RDMA capabilities were extended to support GPUs as well, with GPUDirect® technology, enabling both a ten-fold reduction in latency and a ten-fold increase in bandwidth.

Congestion Control was first implemented and reviewed a decade ago – “Solving Hot Spot Contention Using InfiniBand Architecture Congestion Control” by Pfister, Gusat, Denzel, Craddock, Ni, Rooney,  Engbersen, Luijten, Krishnamurthy and Duato, was published in 2004, and “First experiences with Congestion Control in InfiniBand Hardware” by Gran, Eimot, Reinemo, Skeie, Lysne,  Huse and Shainer, was published in 2010. InfiniBand adaptive routing was enhanced over the last years, and the EDR InfiniBand generation was tested and verified to provide 96% network utilization using its adaptive routing, with the MPIGraph benchmark done on the Oak Ridge National Laboratory Summit supercomputer.

Therefore, InfiniBand, being a standard interconnect, provides not just word-leading performance and scalability, but also protects past investments and ensures forward compatibility, for best return on investment.

In the next article we will review in detail the major technological elements that InfiniBand offers for the networking pillar, their performance, and their reduction of overall applications runtime.

 

References:

[1] https://www.hpcwire.com/2019/06/10/super-connecting-the-supercomputers/

[2] https://www.hpcwire.com/2019/07/15/super-connecting-the-supercomputers-innovations-through-network-topologies/

[3] https://www.hpcwire.com/2016/06/18/offloading-vs-onloading-case-cpu-utilization/

[4] https://www.hpcwire.com/2016/04/12/interconnect-offloading-versus-onloading/

[5] https://www.hpcwire.com/2019/08/01/report-intel-retreats-on-omni-path/

Shares
, we introduced the three interconnect pillars—the connectivity pillar, the network pillar and the communication pillar); Read more…

" share_counter=""]
Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industry updates delivered to you every week!

MLPerf Inference 4.0 Results Showcase GenAI; Nvidia Still Dominates

March 28, 2024

There were no startling surprises in the latest MLPerf Inference benchmark (4.0) results released yesterday. Two new workloads — Llama 2 and Stable Diffusion XL — were added to the benchmark suite as MLPerf continues Read more…

Q&A with Nvidia’s Chief of DGX Systems on the DGX-GB200 Rack-scale System

March 27, 2024

Pictures of Nvidia's new flagship mega-server, the DGX GB200, on the GTC show floor got favorable reactions on social media for the sheer amount of computing power it brings to artificial intelligence.  Nvidia's DGX Read more…

Call for Participation in Workshop on Potential NSF CISE Quantum Initiative

March 26, 2024

Editor’s Note: Next month there will be a workshop to discuss what a quantum initiative led by NSF’s Computer, Information Science and Engineering (CISE) directorate could entail. The details are posted below in a Ca Read more…

Waseda U. Researchers Reports New Quantum Algorithm for Speeding Optimization

March 25, 2024

Optimization problems cover a wide range of applications and are often cited as good candidates for quantum computing. However, the execution time for constrained combinatorial optimization applications on quantum device Read more…

NVLink: Faster Interconnects and Switches to Help Relieve Data Bottlenecks

March 25, 2024

Nvidia’s new Blackwell architecture may have stolen the show this week at the GPU Technology Conference in San Jose, California. But an emerging bottleneck at the network layer threatens to make bigger and brawnier pro Read more…

Who is David Blackwell?

March 22, 2024

During GTC24, co-founder and president of NVIDIA Jensen Huang unveiled the Blackwell GPU. This GPU itself is heavily optimized for AI work, boasting 192GB of HBM3E memory as well as the the ability to train 1 trillion pa Read more…

MLPerf Inference 4.0 Results Showcase GenAI; Nvidia Still Dominates

March 28, 2024

There were no startling surprises in the latest MLPerf Inference benchmark (4.0) results released yesterday. Two new workloads — Llama 2 and Stable Diffusion Read more…

Q&A with Nvidia’s Chief of DGX Systems on the DGX-GB200 Rack-scale System

March 27, 2024

Pictures of Nvidia's new flagship mega-server, the DGX GB200, on the GTC show floor got favorable reactions on social media for the sheer amount of computing po Read more…

NVLink: Faster Interconnects and Switches to Help Relieve Data Bottlenecks

March 25, 2024

Nvidia’s new Blackwell architecture may have stolen the show this week at the GPU Technology Conference in San Jose, California. But an emerging bottleneck at Read more…

Who is David Blackwell?

March 22, 2024

During GTC24, co-founder and president of NVIDIA Jensen Huang unveiled the Blackwell GPU. This GPU itself is heavily optimized for AI work, boasting 192GB of HB Read more…

Nvidia Looks to Accelerate GenAI Adoption with NIM

March 19, 2024

Today at the GPU Technology Conference, Nvidia launched a new offering aimed at helping customers quickly deploy their generative AI applications in a secure, s Read more…

The Generative AI Future Is Now, Nvidia’s Huang Says

March 19, 2024

We are in the early days of a transformative shift in how business gets done thanks to the advent of generative AI, according to Nvidia CEO and cofounder Jensen Read more…

Nvidia’s New Blackwell GPU Can Train AI Models with Trillions of Parameters

March 18, 2024

Nvidia's latest and fastest GPU, codenamed Blackwell, is here and will underpin the company's AI plans this year. The chip offers performance improvements from Read more…

Nvidia Showcases Quantum Cloud, Expanding Quantum Portfolio at GTC24

March 18, 2024

Nvidia’s barrage of quantum news at GTC24 this week includes new products, signature collaborations, and a new Nvidia Quantum Cloud for quantum developers. Wh Read more…

Alibaba Shuts Down its Quantum Computing Effort

November 30, 2023

In case you missed it, China’s e-commerce giant Alibaba has shut down its quantum computing research effort. It’s not entirely clear what drove the change. Read more…

Nvidia H100: Are 550,000 GPUs Enough for This Year?

August 17, 2023

The GPU Squeeze continues to place a premium on Nvidia H100 GPUs. In a recent Financial Times article, Nvidia reports that it expects to ship 550,000 of its lat Read more…

Shutterstock 1285747942

AMD’s Horsepower-packed MI300X GPU Beats Nvidia’s Upcoming H200

December 7, 2023

AMD and Nvidia are locked in an AI performance battle – much like the gaming GPU performance clash the companies have waged for decades. AMD has claimed it Read more…

DoD Takes a Long View of Quantum Computing

December 19, 2023

Given the large sums tied to expensive weapon systems – think $100-million-plus per F-35 fighter – it’s easy to forget the U.S. Department of Defense is a Read more…

Synopsys Eats Ansys: Does HPC Get Indigestion?

February 8, 2024

Recently, it was announced that Synopsys is buying HPC tool developer Ansys. Started in Pittsburgh, Pa., in 1970 as Swanson Analysis Systems, Inc. (SASI) by John Swanson (and eventually renamed), Ansys serves the CAE (Computer Aided Engineering)/multiphysics engineering simulation market. Read more…

Choosing the Right GPU for LLM Inference and Training

December 11, 2023

Accelerating the training and inference processes of deep learning models is crucial for unleashing their true potential and NVIDIA GPUs have emerged as a game- Read more…

Intel’s Server and PC Chip Development Will Blur After 2025

January 15, 2024

Intel's dealing with much more than chip rivals breathing down its neck; it is simultaneously integrating a bevy of new technologies such as chiplets, artificia Read more…

Baidu Exits Quantum, Closely Following Alibaba’s Earlier Move

January 5, 2024

Reuters reported this week that Baidu, China’s giant e-commerce and services provider, is exiting the quantum computing development arena. Reuters reported � Read more…

Leading Solution Providers

Contributors

Comparing NVIDIA A100 and NVIDIA L40S: Which GPU is Ideal for AI and Graphics-Intensive Workloads?

October 30, 2023

With long lead times for the NVIDIA H100 and A100 GPUs, many organizations are looking at the new NVIDIA L40S GPU, which it’s a new GPU optimized for AI and g Read more…

Shutterstock 1179408610

Google Addresses the Mysteries of Its Hypercomputer 

December 28, 2023

When Google launched its Hypercomputer earlier this month (December 2023), the first reaction was, "Say what?" It turns out that the Hypercomputer is Google's t Read more…

AMD MI3000A

How AMD May Get Across the CUDA Moat

October 5, 2023

When discussing GenAI, the term "GPU" almost always enters the conversation and the topic often moves toward performance and access. Interestingly, the word "GPU" is assumed to mean "Nvidia" products. (As an aside, the popular Nvidia hardware used in GenAI are not technically... Read more…

Shutterstock 1606064203

Meta’s Zuckerberg Puts Its AI Future in the Hands of 600,000 GPUs

January 25, 2024

In under two minutes, Meta's CEO, Mark Zuckerberg, laid out the company's AI plans, which included a plan to build an artificial intelligence system with the eq Read more…

Google Introduces ‘Hypercomputer’ to Its AI Infrastructure

December 11, 2023

Google ran out of monikers to describe its new AI system released on December 7. Supercomputer perhaps wasn't an apt description, so it settled on Hypercomputer Read more…

China Is All In on a RISC-V Future

January 8, 2024

The state of RISC-V in China was discussed in a recent report released by the Jamestown Foundation, a Washington, D.C.-based think tank. The report, entitled "E Read more…

Intel Won’t Have a Xeon Max Chip with New Emerald Rapids CPU

December 14, 2023

As expected, Intel officially announced its 5th generation Xeon server chips codenamed Emerald Rapids at an event in New York City, where the focus was really o Read more…

IBM Quantum Summit: Two New QPUs, Upgraded Qiskit, 10-year Roadmap and More

December 4, 2023

IBM kicks off its annual Quantum Summit today and will announce a broad range of advances including its much-anticipated 1121-qubit Condor QPU, a smaller 133-qu Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire