Incoming: 10 Gigabit Ethernet for HPC

By Dan Tuchler

November 20, 2008

Sure, it’s proven, and a lot of people use it. But like many proprietary technologies, it also has some unappealing characteristics. It demands specialized expertise. It’s not always as fast as advertised. It’s not completely reliable. It certainly doesn’t work and play well with others. Yes, we are talking about InfiniBand.

InfiniBand has been a comfort zone for those tightly-coupled HPC applications that can’t live without their addiction to low latency and high speed. If your application is a science experiment with good funding and no firm schedule, that’s OK. If your application involves business, deadlines, and ROI, it’s time to break out of that comfort zone and acquaint yourself with 10 Gigabit Ethernet (10GE).

Fifty seven percent of the TOP500 Supercomputer Sites are already using Gigabit Ethernet interconnects. For many of those sites, that one Gigabit will give them all the speed they’ll ever need. For those applications that really do need a higher performance interconnect, 10GE can provide it — and a host of other advantages as well.

Technology You Know

InfiniBand might be the obvious choice in the HPC world, but compared to the well-known and widely-used Ethernet standard, the comfort level of InfiniBand might be equated to sleeping on a bed of nails. IT executives, network administrators, server vendors, and managed service providers around the world understand Ethernet. They can integrate and update an Ethernet installation within an existing infrastructure, without specialized gateways. They know how to manage it and how to benchmark it. They know what to do if something goes wrong. When it comes to Ethernet, familiarity breeds peace of mind.

More Cost-Effective and Already There

With costs going up, power at a premium, and manageability critical, it’s no surprise that converging and consolidating have become central themes in networking. Converged voice and data networks are becoming pervasive. Fibre Channel over Ethernet (FCoE) will merge storage and standard network applications on a common infrastructure. Virtualization is fast becoming a critical technology to support server and storage consolidation in the datacenter.

When it comes to reducing capital and operating expenses, one infrastructure is simply better than two — or more — and the HPC environment is no exception. High-performance computing clusters that use an InfiniBand interconnect also use Ethernet. Ethernet is necessary for user and storage connectivity, and for the management network that orchestrates the cluster. Replacing the InfiniBand interconnect with 10GE to create a single, all-inclusive infrastructure will cut hardware and power costs, and simplify manageability. And, that infrastructure combines high performance with low power needs and a sufficiently low latency for many HPC applications, making it an excellent fit for technical and budget requirements.

Prices are Plummeting

As with many technologies, 10GE was not initially cost-effective for widespread use. In fact, at one point a 10GE connection cost more than the server. But that ship has sailed. Now 10GE is so cost-effective that server vendors are starting to include the technology as a built-in standard feature. And switch prices are falling too. A number of switch vendors are offering 10 GE switches with a list price less than $500 per port.

Stable Network Interface

Some early adopters of 10GE were discouraged by problems with network interface cards (NICs). These problems were related to immature hardware and software drivers and have since been corrected. NIC vendors that could not adapt have dropped out of the market, and 10GE now has a stable network interface environment.

Physical Layer Selected

Many users expected that 10GBase-T would provide a simple, cost-effective solution, but were disappointed with the high cost, high power requirements and 2.6 µsec latency per cable hop. Multiple optics standards also led to some customer confusion, with XenPak, X2, XFP, and now SFP+.

It took a while for 10GE to converge on a single type of attachment, but many users today believe that SFP+ Direct Attach Cable (also known as twinax) is the right solution. SFP+ Direct Attach Cable is a low-cost, low-latency, interoperable solution that uses existing SFP+ sockets and addresses most 10GE challenges for distances up to around ten meters.

Broad Vendor Support

Every networking vendor supports Ethernet, and that support will extend to 40GE and 100GE in the future. Support for InfiniBand is limited to a handful of vendors. With less competition, there will be less innovation, and less incentive to offer competitive pricing and to increase reliability.

It’s Time

New products and advancing technologies have overcome the last hurdles that prevented 10GE from addressing HPC needs:

  • NIC prices will continue to drop as LAN-on-Motherboard (LOM) technology lets NIC vendors reach the high volumes they need to keep costs down.
  • Switch prices are also dropping–prices under $500 per port make 10GE switches cost-effective for business applications.
  • The emerging SFP+ Direct Attach cabling standard avoids the problem of expensive optics and presents a clear solution.
  • Many HPC applications such as Reuter’s RMDS and others achieve equivalent performance numbers on 10GbE and InfiniBand
  • Using some of the latest server blades and comparing DDR InfiniBand with RDMA to 10GE iWarp using a BLADE Network Technologies switch, we have observed similar performance on common HPC applications such as FLUENT, PAM CRASH, VASP and others.

It’s time to bring the benefits of ubiquitous 10GE to the HPC community. For most clusters and most applications, Ethernet brings the advantages of better pricing, higher reliability, plenty of performance, and lower operating costs than InfiniBand. A holistic approach with a single infrastructure will also contribute to reduced costs, while widespread Ethernet expertise will reduce management headaches and support a more efficient environment.

When InfiniBand is Necessary

Some applications will still require the extremely high performance and low latency that is achievable only with InfiniBand. However, IT teams should evaluate project needs carefully before selecting an interconnect technology. Situations that invite InfiniBand include:

The Need For Speed — Performance is a key driver for InfiniBand, but designers need to be aware of the bottom line for performance statistics. Double Data Rate (DDR) technology claims to provide 20 Gbps performance, but the reality is closer to 16 Gbps. Hardware limits in the PCIe gen1 bus in many machines reduce this number even further. With a measurable performance of about 13 Gbps, InfiniBand isn’t a whole lot faster than 10GE. Similarly, Quad Data Rate (QDR) claims 40 Gbps performance, but actually delivers closer to 26 Gbps due to the PCIe gen2 bus–not that much faster than a two-port 10GE switch.

Really Low Latency — Some compute-intensive, tightly-coupled applications, such as those found in the largest national science labs, do demand the extremely low latency that only InfiniBand can provide. However, other applications that are either loosely coupled or simply don’t have an excessive demand for low latency can run perfectly well over 10GE. Many TCP/IP-based applications fall into this category, and many more can be supported by adapters that offload TCP/IP processing. In fact, some TCP/IP applications actually run faster and with lower latency over 10GE than over InfiniBand.

About the Author

Dan Tuchler is Vice President of Strategy and Product Management at BLADE Network Technologies (www.bladenetwork.net). His technology experience spans emerging Ethernet standards, InfiniBand, security, load balancing, content-aware networking, and high-availability.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industry updates delivered to you every week!

MLPerf Inference 4.0 Results Showcase GenAI; Nvidia Still Dominates

March 28, 2024

There were no startling surprises in the latest MLPerf Inference benchmark (4.0) results released yesterday. Two new workloads — Llama 2 and Stable Diffusion XL — were added to the benchmark suite as MLPerf continues Read more…

Q&A with Nvidia’s Chief of DGX Systems on the DGX-GB200 Rack-scale System

March 27, 2024

Pictures of Nvidia's new flagship mega-server, the DGX GB200, on the GTC show floor got favorable reactions on social media for the sheer amount of computing power it brings to artificial intelligence.  Nvidia's DGX Read more…

Call for Participation in Workshop on Potential NSF CISE Quantum Initiative

March 26, 2024

Editor’s Note: Next month there will be a workshop to discuss what a quantum initiative led by NSF’s Computer, Information Science and Engineering (CISE) directorate could entail. The details are posted below in a Ca Read more…

Waseda U. Researchers Reports New Quantum Algorithm for Speeding Optimization

March 25, 2024

Optimization problems cover a wide range of applications and are often cited as good candidates for quantum computing. However, the execution time for constrained combinatorial optimization applications on quantum device Read more…

NVLink: Faster Interconnects and Switches to Help Relieve Data Bottlenecks

March 25, 2024

Nvidia’s new Blackwell architecture may have stolen the show this week at the GPU Technology Conference in San Jose, California. But an emerging bottleneck at the network layer threatens to make bigger and brawnier pro Read more…

Who is David Blackwell?

March 22, 2024

During GTC24, co-founder and president of NVIDIA Jensen Huang unveiled the Blackwell GPU. This GPU itself is heavily optimized for AI work, boasting 192GB of HBM3E memory as well as the the ability to train 1 trillion pa Read more…

MLPerf Inference 4.0 Results Showcase GenAI; Nvidia Still Dominates

March 28, 2024

There were no startling surprises in the latest MLPerf Inference benchmark (4.0) results released yesterday. Two new workloads — Llama 2 and Stable Diffusion Read more…

Q&A with Nvidia’s Chief of DGX Systems on the DGX-GB200 Rack-scale System

March 27, 2024

Pictures of Nvidia's new flagship mega-server, the DGX GB200, on the GTC show floor got favorable reactions on social media for the sheer amount of computing po Read more…

NVLink: Faster Interconnects and Switches to Help Relieve Data Bottlenecks

March 25, 2024

Nvidia’s new Blackwell architecture may have stolen the show this week at the GPU Technology Conference in San Jose, California. But an emerging bottleneck at Read more…

Who is David Blackwell?

March 22, 2024

During GTC24, co-founder and president of NVIDIA Jensen Huang unveiled the Blackwell GPU. This GPU itself is heavily optimized for AI work, boasting 192GB of HB Read more…

Nvidia Looks to Accelerate GenAI Adoption with NIM

March 19, 2024

Today at the GPU Technology Conference, Nvidia launched a new offering aimed at helping customers quickly deploy their generative AI applications in a secure, s Read more…

The Generative AI Future Is Now, Nvidia’s Huang Says

March 19, 2024

We are in the early days of a transformative shift in how business gets done thanks to the advent of generative AI, according to Nvidia CEO and cofounder Jensen Read more…

Nvidia’s New Blackwell GPU Can Train AI Models with Trillions of Parameters

March 18, 2024

Nvidia's latest and fastest GPU, codenamed Blackwell, is here and will underpin the company's AI plans this year. The chip offers performance improvements from Read more…

Nvidia Showcases Quantum Cloud, Expanding Quantum Portfolio at GTC24

March 18, 2024

Nvidia’s barrage of quantum news at GTC24 this week includes new products, signature collaborations, and a new Nvidia Quantum Cloud for quantum developers. Wh Read more…

Alibaba Shuts Down its Quantum Computing Effort

November 30, 2023

In case you missed it, China’s e-commerce giant Alibaba has shut down its quantum computing research effort. It’s not entirely clear what drove the change. Read more…

Nvidia H100: Are 550,000 GPUs Enough for This Year?

August 17, 2023

The GPU Squeeze continues to place a premium on Nvidia H100 GPUs. In a recent Financial Times article, Nvidia reports that it expects to ship 550,000 of its lat Read more…

Shutterstock 1285747942

AMD’s Horsepower-packed MI300X GPU Beats Nvidia’s Upcoming H200

December 7, 2023

AMD and Nvidia are locked in an AI performance battle – much like the gaming GPU performance clash the companies have waged for decades. AMD has claimed it Read more…

DoD Takes a Long View of Quantum Computing

December 19, 2023

Given the large sums tied to expensive weapon systems – think $100-million-plus per F-35 fighter – it’s easy to forget the U.S. Department of Defense is a Read more…

Synopsys Eats Ansys: Does HPC Get Indigestion?

February 8, 2024

Recently, it was announced that Synopsys is buying HPC tool developer Ansys. Started in Pittsburgh, Pa., in 1970 as Swanson Analysis Systems, Inc. (SASI) by John Swanson (and eventually renamed), Ansys serves the CAE (Computer Aided Engineering)/multiphysics engineering simulation market. Read more…

Choosing the Right GPU for LLM Inference and Training

December 11, 2023

Accelerating the training and inference processes of deep learning models is crucial for unleashing their true potential and NVIDIA GPUs have emerged as a game- Read more…

Intel’s Server and PC Chip Development Will Blur After 2025

January 15, 2024

Intel's dealing with much more than chip rivals breathing down its neck; it is simultaneously integrating a bevy of new technologies such as chiplets, artificia Read more…

Baidu Exits Quantum, Closely Following Alibaba’s Earlier Move

January 5, 2024

Reuters reported this week that Baidu, China’s giant e-commerce and services provider, is exiting the quantum computing development arena. Reuters reported � Read more…

Leading Solution Providers

Contributors

Comparing NVIDIA A100 and NVIDIA L40S: Which GPU is Ideal for AI and Graphics-Intensive Workloads?

October 30, 2023

With long lead times for the NVIDIA H100 and A100 GPUs, many organizations are looking at the new NVIDIA L40S GPU, which it’s a new GPU optimized for AI and g Read more…

Shutterstock 1179408610

Google Addresses the Mysteries of Its Hypercomputer 

December 28, 2023

When Google launched its Hypercomputer earlier this month (December 2023), the first reaction was, "Say what?" It turns out that the Hypercomputer is Google's t Read more…

AMD MI3000A

How AMD May Get Across the CUDA Moat

October 5, 2023

When discussing GenAI, the term "GPU" almost always enters the conversation and the topic often moves toward performance and access. Interestingly, the word "GPU" is assumed to mean "Nvidia" products. (As an aside, the popular Nvidia hardware used in GenAI are not technically... Read more…

Shutterstock 1606064203

Meta’s Zuckerberg Puts Its AI Future in the Hands of 600,000 GPUs

January 25, 2024

In under two minutes, Meta's CEO, Mark Zuckerberg, laid out the company's AI plans, which included a plan to build an artificial intelligence system with the eq Read more…

Google Introduces ‘Hypercomputer’ to Its AI Infrastructure

December 11, 2023

Google ran out of monikers to describe its new AI system released on December 7. Supercomputer perhaps wasn't an apt description, so it settled on Hypercomputer Read more…

China Is All In on a RISC-V Future

January 8, 2024

The state of RISC-V in China was discussed in a recent report released by the Jamestown Foundation, a Washington, D.C.-based think tank. The report, entitled "E Read more…

Intel Won’t Have a Xeon Max Chip with New Emerald Rapids CPU

December 14, 2023

As expected, Intel officially announced its 5th generation Xeon server chips codenamed Emerald Rapids at an event in New York City, where the focus was really o Read more…

IBM Quantum Summit: Two New QPUs, Upgraded Qiskit, 10-year Roadmap and More

December 4, 2023

IBM kicks off its annual Quantum Summit today and will announce a broad range of advances including its much-anticipated 1121-qubit Condor QPU, a smaller 133-qu Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire