Next-Generation 10GbE Switching

By Saqib Jang

June 22, 2007

High-end computing environments for high-performance cluster computing (HPCC) and internet data center (IDC) applications have embraced the scale-out server model. As a result, they have witnessed their data centers expand to include hundreds to thousands of servers running diverse operating systems and applications. As the number of servers has grown, so has the cost of operations, which includes people, space, power, and cooling.

In response, IT organizations are increasingly turning to technologies like utility computing, virtualization, and data center grids to transform data center resources from monolithic systems into agile, shared computing resource pools, which consist of uniform components that can be dynamically aggregated, tiered, provisioned, and accessed.

Harnessed together, these technologies have the potential to dramatically increase performance levels, maximize return on investment, and allow IT organizations to rapidly deploy and scale resources on-demand.

Scalable 10GbE Networking For High-Performance Data Centers

Central to the vision of virtualization and automation of HPCC and IDC data center resources is the deployment of grid computing environments with hundreds to thousands of servers running tightly coupled applications that are highly sensitive to client/server and server-to-storage communications bandwidth, as well as to inter-process message latency.

While a number of specialized interconnect technologies, such as InfiniBand and Fibre Channel, are available for building high performance data center networks, Gigabit Ethernet remains the dominant networking technology in HPCC and IDC environments. The continued evolution towards 10 Gigabit Ethernet (10GbE) server networking with the promise of rapidly declining prices and a stream of innovations, such as power and space-efficient network interface controllers (NICs) for dense server form factors, and support of RDMA, TCP offload engine (TOE), iSCSI, and I/O virtualization capabilities, make it the obvious choice as the upgrade path for high-performance data centers.

Chelsio Communications, a leading provider of 10GbE adapter solutions, offers a family of ‘unified wire’ adapters targeted at the massive installed base of Gigabit Ethernet networking infrastructure. Chelsio’s adapters enable the replacement of disparate fabric technologies such as Fibre Channel and InfiniBand in a wide range of applications, including NAS filers, SAN arrays, high performance cluster computing, and blade servers.

“Chelsio’s focus is on delivering the promise of the unified wire, enabling the convergence of server networking, storage networking and cluster computing interconnect onto a single 10GbE fabric,” said Kianoosh Naghshineh, president and CEO of Chelsio. “Ethernet is the fabric of convergence and we have sucessfully developed a broad family of adapter solutions that deliver all the required critical features, such as low latency, high transaction rate, and reduced cost.”

While vendors such as Chelsio are driving major improvements in 10GbE adapter cost and performance, 10GbE deployment, however, continues to be limited to aggregation of gigabit ports in HPCC and IDC environments, and inter-switch connectivity. A number of obstacles have to be overcome in order to fulfill the promise of broad-based 10GbE data center networking deployment within high-end data centers. The most important among these is the availability of affordable switching infrastructure delivering the scalability, price, performance, and resiliency required for end-to-end 10GbE deployment within high-end IDC and HPCC data centers.

Scaling Ethernet Networks

While the 10GbE switching segment has seen intensifying competition with new switch chip and system products every quarter, switch port density remains a challenge, especially when you compare them to switching options based on Gigabit Ethernet, Fibre Channel, and InfiniBand technologies. The biggest non-blocking 10GbE LAN switch has 128 ports and is priced at thousands of dollars per port. Even with Gigabit Ethernet, the largest non-blocking switch has about 600 ports.

While the bandwidth of a L2 Ethernet network is limited to the bandwidth of the largest Ethernet switch in the core, Fibre Channel and InfiniBand protocols and products enable fabrics (multi-stage meshes of switches) with non-blocking throughput supported across thousands of ports. But the problem with Fibre Channel and InfiniBand is that they don’t do IP very well and they don’t do Ethernet at all. And, there is another challenge with Fibre Channel: With low-cost scale-out servers, the cost of Fibre Channel host bus adapters (HBAs) is commonly more than 25 percent of the cost of the server.

So what is it about Ethernet that prevents multi-path mesh fabrics when both InfiniBand and Fibre Channel support them? It’s a consequence of the plug-and-play nature of Ethernet. Ethernet frames, per the standards, do not keep history. An Ethernet frame does not track elapsed time or count hops. This means that if Ethernet switches are connected in a loop (which multi-path meshes typically have), it is possible that a packet forwarded from the output port of a Ethernet switch may ultimately arrive to an input port of the same Ethernet switch and, without any other mechanisms to resolve the situation, the packet will flow indefinitely in a loop consuming bandwidth and ultimately creating a fully congested network unable to forward any other traffic.

To avoid this, Ethernet switches run the spanning tree protocol (STP) during initialization and periodically thereafter to accommodate changes in the physical network to detect and disable multiple paths automatically. Spanning tree disables paths that could create a loop.

Theoretically, you can build a mesh of spanning tree-enabled Ethernet switches, but because of the single-path constraint, the Ethernet network becomes subjected to numerous congestion points leading to dropped packets and throttling of injection rates in transmit nodes which leads to less than wire speed and non-deterministic performance. In other words, the spanning tree algorithm does not allow the use of Layer 2 Ethernet switches to build multi-path mesh fabrics, which is why the overall bandwidth of a Layer 2 Ethernet network is limited to the capacity of the largest switch in the core.

In order to overcome this limitation of Layer 2 Ethernet switches, it is necessary to use Layer 3 (and higher) IP switches. Using Layer 3 10GbE switches, even switches of 128 ports, severely impacts price and performance metrics (both bandwidth and latency) critical in the data center. According to IDC, Layer 3 switch ports cost, on average, five times as much as Layer 2 ports.

L3 routing functionality requires complex processing based on store-and-forward methods which thereby preclude the use of “cut-through” switches. The requirement for store-and-forward and advanced processing introduces significant switch latency, makes performance non-deterministic and greatly increases the cost of these switches.

Converged I/O Using InfiniBand: Congestion Collapse?

In contrast, InfiniBand fabrics do not have this problem because they include a Subnet Manager that is used to discover the physical topology and set up all the paths by configuring the forwarding tables in each InfiniBand switch.

InfiniBand’s fast point-to-point communication speeds have made it a favorite among the high-end HPCC environments. HPC applications typically operate doing short, discrete calculations on clusters of servers. Since servers quickly complete their calculations and must quickly refresh with new calculations, constantly passing them additional data to keep them occupied is a key part of making sure an HPC farm is run optimally. InfiniBand high bandwidth and low latency make it ideal to pass arguments between HPC servers.

But InfiniBand has a major problem which is now becoming apparent as InfiniBand is being considered for broad-based deployment within HPCC and IDC data centers: InfiniBand switches cannot drop packets to deal with congestion. As a result, switch buffers can fill up, block upstream switches and even block flows that are not contending for the congested link.

For small-scale application environments with predictable load, this is not a problem. However, for large-scale deployments spanning hundreds to thousands of servers supporting a range of applications, the lack of congestion control is potentially disastrous. Data presented at the Open Fabrics conference held in April 2007 in Sonoma shows that congestion in IB networks can occur with as few as 24 servers, dramatically increasing latency and decreasing throughput.

A converged fabric requires significant congestion control to prevent congestion collapse as I/O increases. This is especially important for enterprise-class components such as databases and application servers that may be deployed in mission-critical, customer-facing IDC environments with variable demand. A financial institution that runs a key trading application on a fabric without sufficient congestion control may suffer severe performance degradation or a complete fabric collapse during a heavy trading day.

Need for Low Latency

While latency has been a much-discussed aspect of high-end data center networking, it would be helpful to review the different ways latency is measured as well as its impact on real-world HPCC and Enterprise applications.

First, latency means different things to different people. The application vendors measure how long it takes from requesting the data to when they receive it. If the request is for a big block of data, obviously the pipe size has an impact. The adapter vendors measure the round trip time for a small packet with two back to back servers and divide it by two to estimate the one-way latency. Low latency is better than high latency, but this measure won’t always predict applications performance.

The network switch vendors measure the time from the head of a frame entering one switch, to the head of the frame exiting the switch. What this ignores is the impact of bandwidth on applications performance. As an example, if the DBMS cluster is moving 32K byte data blocks around, they will get to their destination a lot faster with a 10 Gbps pipe than they will with 1 Gbps.

However, scalable support of high-performance applications running on server grids is driving the need for dynamic, fine-grained calibration of end-to-end latency within data centers. While millisecond-scale latency across the WAN can be measured, that isn’t near good enough in the data center. In very controlled conditions, with very expensive test equipment, the latency across a data center network can be measured one port at a time. But in the real world, nobody knows what the latency of their data center network is. While server to server pings can generate round trip time (RTT) latencies, however, microsecond-scale, let alone nanosecond-scale, one-way latencies cannot be measured without special test gear. All that can be typically done is to measure the application response time, and guess where time is being consumed.

The importance of latency is also often debated. In the view of application vendors, the latency they worry about is in hundreds of milliseconds, so fixating on the LAN latency is in their view a waste of time. However, what they fail to take into account is that an application-level transaction may include hundreds of read/write acknowledgements. In these cases, high “stack-up” latencies can result in reduced application performance.

Latency is becoming more important because in scale-out datacenters as the system bus of large multiprocessor systems is effectively being replaced by the LAN. Further, in the grid computing vision, servers across the enterprise network can, in theory, can be dynamically added to the DBMS cluster. But it will only deliver acceptable performance if the combined latency across the multiple switch hops is consistently low.

The need for optimizing end-to-end network availability and performance, including latency, is also being driven by the evolution of server virtualization to enable data center automation. For example, VMWare VMotion capability allows application-ready software modules to be moved seamlessly between physical and virtual computing resources, dynamically provisioned on one or more servers, autonomously updated and patched according to user-definable compliance and security policies, and scheduled, executed, and tracked according to logical sequences, events, dependencies, and geographic hierarchies.

During a VMotion migration, the memory contents of the running virtual machine are transferred over the network to the target server. Having a robust, high-performance network available for this task is critical for ensuring that VMotion operations can complete in a timely manner and result in a successful migration. The bandwidth, latency, and availability of the network determine the effectiveness of the dynamic load-balancing and stateful failover capability enabled by VMotion.

High Cost of Bandwidth

Moore’s Law doesn’t exactly predict performance improvements, but a corollary to transistors doubling every 24 months does. Between clock rate improvements and architectural improvements, CPU performance has typically doubled about every 18 months. Although clock rate improvements are slowing down, multi-core processors and better chip I/O are helping keep performance gains on the same trajectory. What isn’t so obvious is that this means that every five years or so, processor performance goes up 10X.

The last big change in Ethernet speed got well underway by 1999 when IT departments were deploying Gigabit Ethernet server connections using the recently ratified 1000BaseTX standard for Category 5 UTP cable. In 1999 several GbE startups were hitting their stride as newly-minted public companies or business units of incumbent vendors, as their LAN switch sales grew at dizzying rates. While it wasn’t until 2003 that the shipments of Gigabit Ethernet LAN switch ports equaled 100Mbps Ethernet, by then, most IT organizations had lots of servers connected with Gigabit Ethernet.

In servers, the transition to GbE NICs is already complete, while 10GbE remains in the early stages of adoption, with a range of incumbent and startup vendors offering 10GbE server adapters. While 10GbE NIC shipments are growing from a small base, prices are falling rapidly, and the impending volume shipments of 10GBaseT and 10GbE-based blade server products foretell a dynamic and high-growth market.

However, other than the market leader, LAN switch suppliers likely do not have the ability to invest in R&D to develop the new architectures, the chips, and the software needed to improve the price/performance needed to make 10GbE a viable alternative for converged data center networking. The high per-port cost of incumbents’ enterprise-class LAN switches is inherited from their architectures, which are based on either buffered single-stage crossbars or WAN-based 10GbE switch design heritage incorporating sophisticated multi-class quality of service (QoS) capabilities and large, high-performance buffers for handling longer distance.

With every generation of LAN switching, new products have come from startups. In fact, the market leader has bought at least one such company every time. But the dotcom/telecom bubble aftermath has stifled 10GbE LAN switch startup investments. For all practical purposes there aren’t any.

Next-Gen 10GbE LAN Switching For Dynamic, Scale-Out Computing

While a decade ago, Ethernet switching was promoted as the answer to the problems with latency, complexity, and cost caused by using Layer 3 routers to connect compute farms together, LAN switch market incumbents have come to advocate Layer 3 routing over Ethernet switching, likely to gain higher revenues per connection, and to protect their turf from smaller competitors. The strongest arguments in favor of routing are that it eliminates broadcast storms, and it avoids spanning tree protocol-related problems as described above.

But Layer 3 protocols add their own complexity and restrict the flexibility to dynamically reconfigure servers and networks promoted by utility computing visionaries. For example, it’s a lot easier to add a server to an application cluster or enable dynamic migration of virtual servers within the same Layer 2 subnet. Lots of subnets lead to redundant servers for every subnet scattered around the datacenter.

What is a needed is a new approach coming most likely from innovative startups. Incumbents are too focused on protecting their revenue streams and their proprietary turf, which is why the adventuresome are, by process of elimination, looking outside the Ethernet world for the capabilities they need.

One of the start-ups looking to address the limitations of 10GbE LAN switching technology is Woven Systems, a Santa Clara, CA-based startup developing an Ethernet-based mesh network product.

“Woven’s focus is to deliver the best features of Fibre Channel and InfiniBand on a 10G Ethernet fabric,” says Dan Maltbie, founder and Sr. VP of Engineering, Woven Systems.

Woven is developing Layer 2-based 10G Ethernet data center switches that use special algorithms that allow the boxes to deliver a resilient multipath Ethernet fabric solution that delivers the low latency and scalability of InfiniBand coupled with the reliability of Fibre Channel. “Multiple paths can be established among switches in the fabric, allowing bandwidth to be allocated more dynamically over the paths, since traffic lanes are not shut down, as in spanning tree-based Ethernet”, Maltbie says.

In summary, IT capacity and requirements are on a collision course for high-performance computing and internet datacenters. Industry trends such as multi-core CPUs, blade computing, and virtualization are significant advances that have increased datacenter capacity. But the demand for IT compute and storage capabilities is expected to increase at an even faster rate. Ethernet, the staple of datacenter networking infrastructure, has kept pace with the availability of a range of 10GbE server networking adapters, but 10GbE switches have until recently lacked the performance, affordability, scalability and reliability required for pervasive 10GbE deployment within enterprise data centers.


About the Author

Saqib Jang is founder and principal at Margalla Communications, a Woodside, CA-based strategic and technical marketing consulting firm focused on storage and server networking. He can be contacted at [email protected].

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Mira Supercomputer Enables Cancer Research Breakthrough

November 11, 2019

Dynamic partial-wave spectroscopic (PWS) microscopy allows researchers to observe intracellular structures as small as 20 nanometers – smaller than those visible by optical microscopes – in three dimensions at a mill Read more…

By Staff report

IBM Adds Support for Ion Trap Quantum Technology to Qiskit

November 11, 2019

After years of percolating in the shadow of quantum computing research based on superconducting semiconductors – think IBM, Rigetti, Google, and D-Wave (quantum annealing) – ion trap technology is edging into the QC Read more…

By John Russell

Tackling HPC’s Memory and I/O Bottlenecks with On-Node, Non-Volatile RAM

November 8, 2019

On-node, non-volatile memory (NVRAM) is a game-changing technology that can remove many I/O and memory bottlenecks and provide a key enabler for exascale. That’s the conclusion drawn by the scientists and researcher Read more…

By Jan Rowell

What’s New in HPC Research: Cosmic Magnetism, Cryptanalysis, Car Navigation & More

November 8, 2019

In this bimonthly feature, HPCwire highlights newly published research in the high-performance computing community and related domains. From parallel programming to exascale to quantum computing, the details are here. Read more…

By Oliver Peckham

Machine Learning Fuels a Booming HPC Market

November 7, 2019

Enterprise infrastructure investments for training machine learning models have grown more than 50 percent annually over the past two years, and are expected to shortly surpass $10 billion, according to a new market fore Read more…

By George Leopold

AWS Solution Channel

Making High Performance Computing Affordable and Accessible for Small and Medium Businesses with HPC on AWS

High performance computing (HPC) brings a powerful set of tools to a broad range of industries, helping to drive innovation and boost revenue in finance, genomics, oil and gas extraction, and other fields. Read more…

IBM Accelerated Insights

Atom by Atom, Supercomputers Shed Light on Alloys

November 7, 2019

Alloys are at the heart of human civilization, but developing alloys in the Information Age is much different than it was in the Bronze Age. Trial-by-error smelting has given way to the use of high-performance computing Read more…

By Oliver Peckham

IBM Adds Support for Ion Trap Quantum Technology to Qiskit

November 11, 2019

After years of percolating in the shadow of quantum computing research based on superconducting semiconductors – think IBM, Rigetti, Google, and D-Wave (quant Read more…

By John Russell

Tackling HPC’s Memory and I/O Bottlenecks with On-Node, Non-Volatile RAM

November 8, 2019

On-node, non-volatile memory (NVRAM) is a game-changing technology that can remove many I/O and memory bottlenecks and provide a key enabler for exascale. Th Read more…

By Jan Rowell

MLPerf Releases First Inference Benchmark Results; Nvidia Touts its Showing

November 6, 2019, the young AI-benchmarking consortium, today issued the first round of results for its inference test suite. Among organizations with submissions wer Read more…

By John Russell

Azure Cloud First with AMD Epyc Rome Processors

November 6, 2019

At Ignite 2019 this week, Microsoft's Azure cloud team and AMD announced an expansion of their partnership that began in 2017 when Azure debuted Epyc-backed ins Read more…

By Tiffany Trader

Nvidia Launches Credit Card-Sized 21 TOPS Jetson System for Edge Devices

November 6, 2019

Nvidia has launched a new addition to its Jetson product line: a credit card-sized (70x45mm) form factor delivering up to 21 trillion operations/second (TOPS) o Read more…

By Doug Black

In Memoriam: Steve Tuecke, Globus Co-founder

November 4, 2019

HPCwire is deeply saddened to report that Steve Tuecke, longtime scientist at Argonne National Lab and University of Chicago, has passed away at age 52. Tuecke Read more…

By Tiffany Trader

Spending Spree: Hyperscalers Bought $57B of IT in 2018, $10B+ by Google – But Is Cloud on Horizon?

October 31, 2019

Hyperscalers are the masters of the IT universe, gravitational centers of increasing pull in the emerging age of data-driven compute and AI.  In the high-stake Read more…

By Doug Black

Cray Debuts ClusterStor E1000 Finishing Remake of Portfolio for ‘Exascale Era’

October 30, 2019

Cray, now owned by HPE, today introduced the ClusterStor E1000 storage platform, which leverages Cray software and mixes hard disk drives (HDD) and flash memory Read more…

By John Russell

Supercomputer-Powered AI Tackles a Key Fusion Energy Challenge

August 7, 2019

Fusion energy is the Holy Grail of the energy world: low-radioactivity, low-waste, zero-carbon, high-output nuclear power that can run on hydrogen or lithium. T Read more…

By Oliver Peckham

Using AI to Solve One of the Most Prevailing Problems in CFD

October 17, 2019

How can artificial intelligence (AI) and high-performance computing (HPC) solve mesh generation, one of the most commonly referenced problems in computational engineering? A new study has set out to answer this question and create an industry-first AI-mesh application... Read more…

By James Sharpe

Cray Wins NNSA-Livermore ‘El Capitan’ Exascale Contract

August 13, 2019

Cray has won the bid to build the first exascale supercomputer for the National Nuclear Security Administration (NNSA) and Lawrence Livermore National Laborator Read more…

By Tiffany Trader

DARPA Looks to Propel Parallelism

September 4, 2019

As Moore’s law runs out of steam, new programming approaches are being pursued with the goal of greater hardware performance with less coding. The Defense Advanced Projects Research Agency is launching a new programming effort aimed at leveraging the benefits of massive distributed parallelism with less sweat. Read more…

By George Leopold

AMD Launches Epyc Rome, First 7nm CPU

August 8, 2019

From a gala event at the Palace of Fine Arts in San Francisco yesterday (Aug. 7), AMD launched its second-generation Epyc Rome x86 chips, based on its 7nm proce Read more…

By Tiffany Trader

D-Wave’s Path to 5000 Qubits; Google’s Quantum Supremacy Claim

September 24, 2019

On the heels of IBM’s quantum news last week come two more quantum items. D-Wave Systems today announced the name of its forthcoming 5000-qubit system, Advantage (yes the name choice isn’t serendipity), at its user conference being held this week in Newport, RI. Read more…

By John Russell

Ayar Labs to Demo Photonics Chiplet in FPGA Package at Hot Chips

August 19, 2019

Silicon startup Ayar Labs continues to gain momentum with its DARPA-backed optical chiplet technology that puts advanced electronics and optics on the same chip Read more…

By Tiffany Trader

Crystal Ball Gazing: IBM’s Vision for the Future of Computing

October 14, 2019

Dario Gil, IBM’s relatively new director of research, painted a intriguing portrait of the future of computing along with a rough idea of how IBM thinks we’ Read more…

By John Russell

Leading Solution Providers

ISC 2019 Virtual Booth Video Tour


Intel Confirms Retreat on Omni-Path

August 1, 2019

Intel Corp.’s plans to make a big splash in the network fabric market for linking HPC and other workloads has apparently belly-flopped. The chipmaker confirmed to us the outlines of an earlier report by the website CRN that it has jettisoned plans for a second-generation version of its Omni-Path interconnect... Read more…

By Staff report

Kubernetes, Containers and HPC

September 19, 2019

Software containers and Kubernetes are important tools for building, deploying, running and managing modern enterprise applications at scale and delivering enterprise software faster and more reliably to the end user — while using resources more efficiently and reducing costs. Read more…

By Daniel Gruber, Burak Yenier and Wolfgang Gentzsch, UberCloud

Dell Ramps Up HPC Testing of AMD Rome Processors

October 21, 2019

Dell Technologies is wading deeper into the AMD-based systems market with a growing evaluation program for the latest Epyc (Rome) microprocessors from AMD. In a Read more…

By John Russell

Intel Debuts Pohoiki Beach, Its 8M Neuron Neuromorphic Development System

July 17, 2019

Neuromorphic computing has received less fanfare of late than quantum computing whose mystery has captured public attention and which seems to have generated mo Read more…

By John Russell

Rise of NIH’s Biowulf Mirrors the Rise of Computational Biology

July 29, 2019

The story of NIH’s supercomputer Biowulf is fascinating, important, and in many ways representative of the transformation of life sciences and biomedical res Read more…

By John Russell

Xilinx vs. Intel: FPGA Market Leaders Launch Server Accelerator Cards

August 6, 2019

The two FPGA market leaders, Intel and Xilinx, both announced new accelerator cards this week designed to handle specialized, compute-intensive workloads and un Read more…

By Doug Black

When Dense Matrix Representations Beat Sparse

September 9, 2019

In our world filled with unintended consequences, it turns out that saving memory space to help deal with GPU limitations, knowing it introduces performance pen Read more…

By James Reinders

With the Help of HPC, Astronomers Prepare to Deflect a Real Asteroid

September 26, 2019

For years, NASA has been running simulations of asteroid impacts to understand the risks (and likelihoods) of asteroids colliding with Earth. Now, NASA and the European Space Agency (ESA) are preparing for the next, crucial step in planetary defense against asteroid impacts: physically deflecting a real asteroid. Read more…

By Oliver Peckham

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This