Tuning InfiniBand Interconnects Using Congestion Control

By Adam Dorsey

July 26, 2017

InfiniBand is among the most common and well-known cluster interconnect technologies. However, the complexities of an InfiniBand (IB) network can frustrate the most experienced cluster administrators. Maintaining a balanced fabric topology, dealing with underperforming hosts and links, and chasing potential improvements keeps all of us on our toes. Sometimes, though, a little research and experimentation can find unexpected performance and stability gains.

For example, consider a 1,300-node cluster using Intel TrueScale IB for job communication and a Panasas ActiveStor filesystem for storage. Panasas only communicates to clients via Ethernet and not IB, so a group of Mellanox switches act as gateways from the Panasas Ethernet to the TrueScale IB.

Every system has bottlenecks; in our case, the links to and from these IB/Ethernet gateways showed congestion due to the large amount of disk traffic. This adversely affects the whole cluster — jobs can’t get the data they need, and the increased congestion interferes with other IB traffic as well.

Fortunately, InfiniBand provides a congestion control mechanism that can help mitigate the effects of severe congestion on the fabric. We were able to implement this feature to save the expense and trouble of adding additional IB/Ethernet gateways.

What Is InfiniBand Congestion Control?

InfiniBand is intended to be a lossless fabric. IB switches won’t drop packets for flow control unless they absolutely have to, usually in cases of hardware failure or malformed packets. Instead of dropping packets and retransmitting, like Ethernet does, InfiniBand uses a system of credits to perform flow control.

Communication occurs between IB endpoints, which in turn are issued credits based on the amount of buffer space the receiving device has. If the credit cost of the data to be transmitted is less than the credits remaining on the receiving device, the data is transmitted. Otherwise, the transmitting device holds on to the data until the receiving device has sufficient credits free.

This method of flow control works well for normal loads on well-balanced, non-oversubscribed IB fabrics. However, if the fabric is unbalanced or oversubscribed or just heavily loaded, some links may be oversaturated with traffic beyond the ability of the credit mechanism to help.

Congestion can be observed by checking the IB error counters. When an IB device attempts to transmit data but the receiving device cannot receive data due to congestion, the PortXmitWait counter is incremented. If the congestion is so bad that the data cannot be transmitted before the time-to-live on the packet expires, the packet is discarded and the PortXmitDiscards counter is incremented. If you’re seeing high values of PortXmitWait and PortXmitDiscards counters, enabling congestion control may help manage congestion on your InfiniBand fabric.

How Does InfiniBand Congestion Control Work?

When an IB switch detects congestion on a link, it enables a special bit, called the Forward Explicit Congestion Notification (FECN) bit, which informs the destination device that congestion has been detected on the link. When the destination receives a packet marked with the FECN bit, the destination device notifies the sending device of the congestion via a Backwards Explicit Congestion Notification bit (BECN.)

When the source receives the BECN bit notification from the destination, the sending (source) device begins to throttle the amount of data it sends to the destination. The mechanism it uses is the credits system – by reducing the credits available to the destination, the size and rate of the packets are effectively decreased. The sending device may also add a delay between packets to provide the destination device time to catch up on data.

Over time, the source device increases credits for the destination device, gradually increasing the amount of packets sent. If the destination device continues to receive FECN packets from its switch, it again transmits BECN packets to the source device and the throttling is increased again. Without the reception of BECN packets from the destination device, the source device eventually returns to normal packet transmission. This balancing act is managed by congestion control parameters which require tuning for each environment.

After enabling InfiniBand congestion control and proper tuning, we realized a 15 percent improvement in our Panasas file system benchmark testing. PortXmitDiscards counters were completely clear, and PortXmitWait counters were significantly smaller, indicating that congestion control was doing its job.

Given that no additional hardware or other costs were required to achieve these results, a speed increase of 15 percent plus increased stability of the IB fabric was a nice result.

How Can I Enable InfiniBand Congestion Control?

Congestion control must be enabled on all IB devices and hosts, as well as on the IB subnet manager. This process includes turning on congestion control and setting a congestion control key on each device, as well as tuning the congestion control tables and parameters on each host and switch.

After congestion control is enabled on each IB device, the OpenSM configuration file must be modified to tune the subnet manager’s congestion control manager. Please note that mistuned parameters will either wreak havoc on a fabric or be completely ineffectual, so be careful – and do plenty of testing on a safe “test” system. Never attempt this on a live or production system.

Enabling InfiniBand congestion control had an immediate positive effect on our IB fabric. If you are suffering from issues with fabric congestion, enabling congestion control may provide the similar relief for your fabric as well, without the cost of adding additional hardware.

About the Author

Adam Dorsey is a systems administrator and site lead for RedLine Performance Solutions.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Nvidia Debuts Turing Architecture, Focusing on Real-Time Ray Tracing

August 16, 2018

From the SIGGRAPH professional graphics conference in Vancouver this week, Nvidia CEO Jensen Huang unveiled Turing, the company's next-gen GPU platform that introduces new RT Cores to accelerate ray tracing and new Tenso Read more…

By Tiffany Trader

HPC Coding: The Power of L(o)osing Control

August 16, 2018

Exascale roadmaps, exascale projects and exascale lobbyists ask, on-again-off-again, for a fundamental rewrite of major code building blocks. Otherwise, so they claim, codes will not scale up. Naturally, some exascale pr Read more…

By Tobias Weinzierl

STAQ(ing) the Quantum Computing Deck

August 16, 2018

Quantum computers – at least for now – remain noisy. That’s another way of saying unreliable and in diverse ways that often depend on the specific quantum technology used. One idea is to mitigate noisiness and perh Read more…

By John Russell

HPE Extreme Performance Solutions

Introducing the First Integrated System Management Software for HPC Clusters from HPE

How do you manage your complex, growing cluster environments? Answer that big challenge with the new HPC cluster management solution: HPE Performance Cluster Manager. Read more…

IBM Accelerated Insights

Super Problem Solving

You might think that tackling the world’s toughest problems is a job only for superheroes, but at special places such as the Oak Ridge National Laboratory, supercomputers are the real heroes. Read more…

NREL ‘Eagle’ Supercomputer to Advance Energy Tech R&D

August 14, 2018

The U.S. Department of Energy (DOE) National Renewable Energy Laboratory (NREL) has contracted with Hewlett Packard Enterprise (HPE) for a new 8-petaflops (peak) supercomputer that will be used to advance early-stage R&a Read more…

By Tiffany Trader

STAQ(ing) the Quantum Computing Deck

August 16, 2018

Quantum computers – at least for now – remain noisy. That’s another way of saying unreliable and in diverse ways that often depend on the specific quantum Read more…

By John Russell

NREL ‘Eagle’ Supercomputer to Advance Energy Tech R&D

August 14, 2018

The U.S. Department of Energy (DOE) National Renewable Energy Laboratory (NREL) has contracted with Hewlett Packard Enterprise (HPE) for a new 8-petaflops (peak Read more…

By Tiffany Trader

CERN Project Sees Orders-of-Magnitude Speedup with AI Approach

August 14, 2018

An award-winning effort at CERN has demonstrated potential to significantly change how the physics based modeling and simulation communities view machine learni Read more…

By Rob Farber

Intel Announces Cooper Lake, Advances AI Strategy

August 9, 2018

Intel's chief datacenter exec Navin Shenoy kicked off the company's Data-Centric Innovation Summit Wednesday, the day-long program devoted to Intel's datacenter Read more…

By Tiffany Trader

SLATE Update: Making Math Libraries Exascale-ready

August 9, 2018

Practically-speaking, achieving exascale computing requires enabling HPC software to effectively use accelerators – mostly GPUs at present – and that remain Read more…

By John Russell

Summertime in Washington: Some Unexpected Advanced Computing News

August 8, 2018

Summertime in Washington DC is known for its heat and humidity. That is why most people get away to either the mountains or the seashore and things slow down. H Read more…

By Alex R. Larzelere

NSF Invests $15 Million in Quantum STAQ

August 7, 2018

Quantum computing development is in full ascent as global backers aim to transcend the limitations of classical computing by leveraging the magical-seeming prop Read more…

By Tiffany Trader

By the Numbers: Cray Would Like Exascale to Be the Icing on the Cake

August 1, 2018

On its earnings call held for investors yesterday, Cray gave an accounting for its latest quarterly financials, offered future guidance and provided an update o Read more…

By Tiffany Trader

Leading Solution Providers

SC17 Booth Video Tours Playlist

Altair @ SC17

Altair

AMD @ SC17

AMD

ASRock Rack @ SC17

ASRock Rack

CEJN @ SC17

CEJN

DDN Storage @ SC17

DDN Storage

Huawei @ SC17

Huawei

IBM @ SC17

IBM

IBM Power Systems @ SC17

IBM Power Systems

Intel @ SC17

Intel

Lenovo @ SC17

Lenovo

Mellanox Technologies @ SC17

Mellanox Technologies

Microsoft @ SC17

Microsoft

Penguin Computing @ SC17

Penguin Computing

Pure Storage @ SC17

Pure Storage

Supericro @ SC17

Supericro

Tyan @ SC17

Tyan

Univa @ SC17

Univa

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This