Next-Generation 10GbE Switching

By Saqib Jang

June 22, 2007

High-end computing environments for high-performance cluster computing (HPCC) and internet data center (IDC) applications have embraced the scale-out server model. As a result, they have witnessed their data centers expand to include hundreds to thousands of servers running diverse operating systems and applications. As the number of servers has grown, so has the cost of operations, which includes people, space, power, and cooling.

In response, IT organizations are increasingly turning to technologies like utility computing, virtualization, and data center grids to transform data center resources from monolithic systems into agile, shared computing resource pools, which consist of uniform components that can be dynamically aggregated, tiered, provisioned, and accessed.

Harnessed together, these technologies have the potential to dramatically increase performance levels, maximize return on investment, and allow IT organizations to rapidly deploy and scale resources on-demand.

Scalable 10GbE Networking For High-Performance Data Centers

Central to the vision of virtualization and automation of HPCC and IDC data center resources is the deployment of grid computing environments with hundreds to thousands of servers running tightly coupled applications that are highly sensitive to client/server and server-to-storage communications bandwidth, as well as to inter-process message latency.

While a number of specialized interconnect technologies, such as InfiniBand and Fibre Channel, are available for building high performance data center networks, Gigabit Ethernet remains the dominant networking technology in HPCC and IDC environments. The continued evolution towards 10 Gigabit Ethernet (10GbE) server networking with the promise of rapidly declining prices and a stream of innovations, such as power and space-efficient network interface controllers (NICs) for dense server form factors, and support of RDMA, TCP offload engine (TOE), iSCSI, and I/O virtualization capabilities, make it the obvious choice as the upgrade path for high-performance data centers.

Chelsio Communications, a leading provider of 10GbE adapter solutions, offers a family of ‘unified wire’ adapters targeted at the massive installed base of Gigabit Ethernet networking infrastructure. Chelsio’s adapters enable the replacement of disparate fabric technologies such as Fibre Channel and InfiniBand in a wide range of applications, including NAS filers, SAN arrays, high performance cluster computing, and blade servers.

“Chelsio’s focus is on delivering the promise of the unified wire, enabling the convergence of server networking, storage networking and cluster computing interconnect onto a single 10GbE fabric,” said Kianoosh Naghshineh, president and CEO of Chelsio. “Ethernet is the fabric of convergence and we have sucessfully developed a broad family of adapter solutions that deliver all the required critical features, such as low latency, high transaction rate, and reduced cost.”

While vendors such as Chelsio are driving major improvements in 10GbE adapter cost and performance, 10GbE deployment, however, continues to be limited to aggregation of gigabit ports in HPCC and IDC environments, and inter-switch connectivity. A number of obstacles have to be overcome in order to fulfill the promise of broad-based 10GbE data center networking deployment within high-end data centers. The most important among these is the availability of affordable switching infrastructure delivering the scalability, price, performance, and resiliency required for end-to-end 10GbE deployment within high-end IDC and HPCC data centers.

Scaling Ethernet Networks

While the 10GbE switching segment has seen intensifying competition with new switch chip and system products every quarter, switch port density remains a challenge, especially when you compare them to switching options based on Gigabit Ethernet, Fibre Channel, and InfiniBand technologies. The biggest non-blocking 10GbE LAN switch has 128 ports and is priced at thousands of dollars per port. Even with Gigabit Ethernet, the largest non-blocking switch has about 600 ports.

While the bandwidth of a L2 Ethernet network is limited to the bandwidth of the largest Ethernet switch in the core, Fibre Channel and InfiniBand protocols and products enable fabrics (multi-stage meshes of switches) with non-blocking throughput supported across thousands of ports. But the problem with Fibre Channel and InfiniBand is that they don’t do IP very well and they don’t do Ethernet at all. And, there is another challenge with Fibre Channel: With low-cost scale-out servers, the cost of Fibre Channel host bus adapters (HBAs) is commonly more than 25 percent of the cost of the server.

So what is it about Ethernet that prevents multi-path mesh fabrics when both InfiniBand and Fibre Channel support them? It’s a consequence of the plug-and-play nature of Ethernet. Ethernet frames, per the standards, do not keep history. An Ethernet frame does not track elapsed time or count hops. This means that if Ethernet switches are connected in a loop (which multi-path meshes typically have), it is possible that a packet forwarded from the output port of a Ethernet switch may ultimately arrive to an input port of the same Ethernet switch and, without any other mechanisms to resolve the situation, the packet will flow indefinitely in a loop consuming bandwidth and ultimately creating a fully congested network unable to forward any other traffic.

To avoid this, Ethernet switches run the spanning tree protocol (STP) during initialization and periodically thereafter to accommodate changes in the physical network to detect and disable multiple paths automatically. Spanning tree disables paths that could create a loop.

Theoretically, you can build a mesh of spanning tree-enabled Ethernet switches, but because of the single-path constraint, the Ethernet network becomes subjected to numerous congestion points leading to dropped packets and throttling of injection rates in transmit nodes which leads to less than wire speed and non-deterministic performance. In other words, the spanning tree algorithm does not allow the use of Layer 2 Ethernet switches to build multi-path mesh fabrics, which is why the overall bandwidth of a Layer 2 Ethernet network is limited to the capacity of the largest switch in the core.

In order to overcome this limitation of Layer 2 Ethernet switches, it is necessary to use Layer 3 (and higher) IP switches. Using Layer 3 10GbE switches, even switches of 128 ports, severely impacts price and performance metrics (both bandwidth and latency) critical in the data center. According to IDC, Layer 3 switch ports cost, on average, five times as much as Layer 2 ports.

L3 routing functionality requires complex processing based on store-and-forward methods which thereby preclude the use of “cut-through” switches. The requirement for store-and-forward and advanced processing introduces significant switch latency, makes performance non-deterministic and greatly increases the cost of these switches.

Converged I/O Using InfiniBand: Congestion Collapse?

In contrast, InfiniBand fabrics do not have this problem because they include a Subnet Manager that is used to discover the physical topology and set up all the paths by configuring the forwarding tables in each InfiniBand switch.

InfiniBand’s fast point-to-point communication speeds have made it a favorite among the high-end HPCC environments. HPC applications typically operate doing short, discrete calculations on clusters of servers. Since servers quickly complete their calculations and must quickly refresh with new calculations, constantly passing them additional data to keep them occupied is a key part of making sure an HPC farm is run optimally. InfiniBand high bandwidth and low latency make it ideal to pass arguments between HPC servers.

But InfiniBand has a major problem which is now becoming apparent as InfiniBand is being considered for broad-based deployment within HPCC and IDC data centers: InfiniBand switches cannot drop packets to deal with congestion. As a result, switch buffers can fill up, block upstream switches and even block flows that are not contending for the congested link.

For small-scale application environments with predictable load, this is not a problem. However, for large-scale deployments spanning hundreds to thousands of servers supporting a range of applications, the lack of congestion control is potentially disastrous. Data presented at the Open Fabrics conference held in April 2007 in Sonoma shows that congestion in IB networks can occur with as few as 24 servers, dramatically increasing latency and decreasing throughput.

A converged fabric requires significant congestion control to prevent congestion collapse as I/O increases. This is especially important for enterprise-class components such as databases and application servers that may be deployed in mission-critical, customer-facing IDC environments with variable demand. A financial institution that runs a key trading application on a fabric without sufficient congestion control may suffer severe performance degradation or a complete fabric collapse during a heavy trading day.

Need for Low Latency

While latency has been a much-discussed aspect of high-end data center networking, it would be helpful to review the different ways latency is measured as well as its impact on real-world HPCC and Enterprise applications.

First, latency means different things to different people. The application vendors measure how long it takes from requesting the data to when they receive it. If the request is for a big block of data, obviously the pipe size has an impact. The adapter vendors measure the round trip time for a small packet with two back to back servers and divide it by two to estimate the one-way latency. Low latency is better than high latency, but this measure won’t always predict applications performance.

The network switch vendors measure the time from the head of a frame entering one switch, to the head of the frame exiting the switch. What this ignores is the impact of bandwidth on applications performance. As an example, if the DBMS cluster is moving 32K byte data blocks around, they will get to their destination a lot faster with a 10 Gbps pipe than they will with 1 Gbps.

However, scalable support of high-performance applications running on server grids is driving the need for dynamic, fine-grained calibration of end-to-end latency within data centers. While millisecond-scale latency across the WAN can be measured, that isn’t near good enough in the data center. In very controlled conditions, with very expensive test equipment, the latency across a data center network can be measured one port at a time. But in the real world, nobody knows what the latency of their data center network is. While server to server pings can generate round trip time (RTT) latencies, however, microsecond-scale, let alone nanosecond-scale, one-way latencies cannot be measured without special test gear. All that can be typically done is to measure the application response time, and guess where time is being consumed.

The importance of latency is also often debated. In the view of application vendors, the latency they worry about is in hundreds of milliseconds, so fixating on the LAN latency is in their view a waste of time. However, what they fail to take into account is that an application-level transaction may include hundreds of read/write acknowledgements. In these cases, high “stack-up” latencies can result in reduced application performance.

Latency is becoming more important because in scale-out datacenters as the system bus of large multiprocessor systems is effectively being replaced by the LAN. Further, in the grid computing vision, servers across the enterprise network can, in theory, can be dynamically added to the DBMS cluster. But it will only deliver acceptable performance if the combined latency across the multiple switch hops is consistently low.

The need for optimizing end-to-end network availability and performance, including latency, is also being driven by the evolution of server virtualization to enable data center automation. For example, VMWare VMotion capability allows application-ready software modules to be moved seamlessly between physical and virtual computing resources, dynamically provisioned on one or more servers, autonomously updated and patched according to user-definable compliance and security policies, and scheduled, executed, and tracked according to logical sequences, events, dependencies, and geographic hierarchies.

During a VMotion migration, the memory contents of the running virtual machine are transferred over the network to the target server. Having a robust, high-performance network available for this task is critical for ensuring that VMotion operations can complete in a timely manner and result in a successful migration. The bandwidth, latency, and availability of the network determine the effectiveness of the dynamic load-balancing and stateful failover capability enabled by VMotion.

High Cost of Bandwidth

Moore’s Law doesn’t exactly predict performance improvements, but a corollary to transistors doubling every 24 months does. Between clock rate improvements and architectural improvements, CPU performance has typically doubled about every 18 months. Although clock rate improvements are slowing down, multi-core processors and better chip I/O are helping keep performance gains on the same trajectory. What isn’t so obvious is that this means that every five years or so, processor performance goes up 10X.

The last big change in Ethernet speed got well underway by 1999 when IT departments were deploying Gigabit Ethernet server connections using the recently ratified 1000BaseTX standard for Category 5 UTP cable. In 1999 several GbE startups were hitting their stride as newly-minted public companies or business units of incumbent vendors, as their LAN switch sales grew at dizzying rates. While it wasn’t until 2003 that the shipments of Gigabit Ethernet LAN switch ports equaled 100Mbps Ethernet, by then, most IT organizations had lots of servers connected with Gigabit Ethernet.

In servers, the transition to GbE NICs is already complete, while 10GbE remains in the early stages of adoption, with a range of incumbent and startup vendors offering 10GbE server adapters. While 10GbE NIC shipments are growing from a small base, prices are falling rapidly, and the impending volume shipments of 10GBaseT and 10GbE-based blade server products foretell a dynamic and high-growth market.

However, other than the market leader, LAN switch suppliers likely do not have the ability to invest in R&D to develop the new architectures, the chips, and the software needed to improve the price/performance needed to make 10GbE a viable alternative for converged data center networking. The high per-port cost of incumbents’ enterprise-class LAN switches is inherited from their architectures, which are based on either buffered single-stage crossbars or WAN-based 10GbE switch design heritage incorporating sophisticated multi-class quality of service (QoS) capabilities and large, high-performance buffers for handling longer distance.

With every generation of LAN switching, new products have come from startups. In fact, the market leader has bought at least one such company every time. But the dotcom/telecom bubble aftermath has stifled 10GbE LAN switch startup investments. For all practical purposes there aren’t any.

Next-Gen 10GbE LAN Switching For Dynamic, Scale-Out Computing

While a decade ago, Ethernet switching was promoted as the answer to the problems with latency, complexity, and cost caused by using Layer 3 routers to connect compute farms together, LAN switch market incumbents have come to advocate Layer 3 routing over Ethernet switching, likely to gain higher revenues per connection, and to protect their turf from smaller competitors. The strongest arguments in favor of routing are that it eliminates broadcast storms, and it avoids spanning tree protocol-related problems as described above.

But Layer 3 protocols add their own complexity and restrict the flexibility to dynamically reconfigure servers and networks promoted by utility computing visionaries. For example, it’s a lot easier to add a server to an application cluster or enable dynamic migration of virtual servers within the same Layer 2 subnet. Lots of subnets lead to redundant servers for every subnet scattered around the datacenter.

What is a needed is a new approach coming most likely from innovative startups. Incumbents are too focused on protecting their revenue streams and their proprietary turf, which is why the adventuresome are, by process of elimination, looking outside the Ethernet world for the capabilities they need.

One of the start-ups looking to address the limitations of 10GbE LAN switching technology is Woven Systems, a Santa Clara, CA-based startup developing an Ethernet-based mesh network product.

“Woven’s focus is to deliver the best features of Fibre Channel and InfiniBand on a 10G Ethernet fabric,” says Dan Maltbie, founder and Sr. VP of Engineering, Woven Systems.

Woven is developing Layer 2-based 10G Ethernet data center switches that use special algorithms that allow the boxes to deliver a resilient multipath Ethernet fabric solution that delivers the low latency and scalability of InfiniBand coupled with the reliability of Fibre Channel. “Multiple paths can be established among switches in the fabric, allowing bandwidth to be allocated more dynamically over the paths, since traffic lanes are not shut down, as in spanning tree-based Ethernet”, Maltbie says.

In summary, IT capacity and requirements are on a collision course for high-performance computing and internet datacenters. Industry trends such as multi-core CPUs, blade computing, and virtualization are significant advances that have increased datacenter capacity. But the demand for IT compute and storage capabilities is expected to increase at an even faster rate. Ethernet, the staple of datacenter networking infrastructure, has kept pace with the availability of a range of 10GbE server networking adapters, but 10GbE switches have until recently lacked the performance, affordability, scalability and reliability required for pervasive 10GbE deployment within enterprise data centers.

—–

About the Author

Saqib Jang is founder and principal at Margalla Communications, a Woodside, CA-based strategic and technical marketing consulting firm focused on storage and server networking. He can be contacted at [email protected].

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Nvidia Aims Clara Healthcare at Drug Discovery, Imaging via DGX

April 12, 2021

Nvidia Corp. continues to expand its Clara healthcare platform with the addition of computational drug discovery and medical imaging tools based on its DGX A100 platform, related InfiniBand networking and its AGX develop Read more…

Nvidia Serves Up Its First Arm Datacenter CPU ‘Grace’ During Kitchen Keynote

April 12, 2021

Today at Nvidia’s annual spring GPU technology conference, held virtually once more due to the ongoing pandemic, the company announced its first ever Arm-based CPU, called Grace in honor of the famous American programmer Grace Hopper. Read more…

Nvidia Debuts BlueField-3 – Its Next DPU with Big Plans for an Expanded Role

April 12, 2021

Nvidia today announced its next generation data processing unit (DPU) – BlueField-3 – adding more substance to its evolving concept of the DPU as a full-fledged partner to CPUs and GPUs in delivering advanced computi Read more…

Nvidia’s Newly DPU-Enabled SuperPOD Is a Multi-Tenant, Cloud-Native Supercomputer

April 12, 2021

At GTC 2021, Nvidia has announced an upgraded iteration of its DGX SuperPods, calling the new offering “the first cloud-native, multi-tenant supercomputer.” The newly announced SuperPods come just two years after the Read more…

Tune in to Watch Nvidia’s GTC21 Keynote with Jensen Huang – Recording Now Available

April 12, 2021

Join HPCwire right here on Monday, April 12, at 8:30 am PT to see the Nvidia GTC21 keynote from Nvidia’s CEO, Jensen Huang, livestreamed in its entirety. Hosted by HPCwire, you can click to join the Huang keynote on our livestream to hear Nvidia’s expected news and... Read more…

AWS Solution Channel

Volkswagen Passenger Cars Uses NICE DCV for High-Performance 3D Remote Visualization

 

Volkswagen Passenger Cars has been one of the world’s largest car manufacturers for over 70 years. The company delivers more than 6 million automobiles to global customers every year, from 50 production locations on five continents. Read more…

The US Places Seven Additional Chinese Supercomputing Entities on Blacklist

April 8, 2021

As tensions between the U.S. and China continue to simmer, the U.S. government today added seven Chinese supercomputing entities to an economic blacklist. The U.S. Entity List bars U.S. firms from supplying key technolog Read more…

Nvidia Serves Up Its First Arm Datacenter CPU ‘Grace’ During Kitchen Keynote

April 12, 2021

Today at Nvidia’s annual spring GPU technology conference, held virtually once more due to the ongoing pandemic, the company announced its first ever Arm-based CPU, called Grace in honor of the famous American programmer Grace Hopper. Read more…

Nvidia Debuts BlueField-3 – Its Next DPU with Big Plans for an Expanded Role

April 12, 2021

Nvidia today announced its next generation data processing unit (DPU) – BlueField-3 – adding more substance to its evolving concept of the DPU as a full-fle Read more…

Nvidia’s Newly DPU-Enabled SuperPOD Is a Multi-Tenant, Cloud-Native Supercomputer

April 12, 2021

At GTC 2021, Nvidia has announced an upgraded iteration of its DGX SuperPods, calling the new offering “the first cloud-native, multi-tenant supercomputer.” Read more…

Tune in to Watch Nvidia’s GTC21 Keynote with Jensen Huang – Recording Now Available

April 12, 2021

Join HPCwire right here on Monday, April 12, at 8:30 am PT to see the Nvidia GTC21 keynote from Nvidia’s CEO, Jensen Huang, livestreamed in its entirety. Hosted by HPCwire, you can click to join the Huang keynote on our livestream to hear Nvidia’s expected news and... Read more…

The US Places Seven Additional Chinese Supercomputing Entities on Blacklist

April 8, 2021

As tensions between the U.S. and China continue to simmer, the U.S. government today added seven Chinese supercomputing entities to an economic blacklist. The U Read more…

Habana’s AI Silicon Comes to San Diego Supercomputer Center

April 8, 2021

Habana Labs, an Intel-owned AI company, has partnered with server maker Supermicro to provide high-performance, high-efficiency AI computing in the form of new Read more…

Intel Partners Debut Latest Servers Based on the New Intel Gen 3 ‘Ice Lake’ Xeons

April 7, 2021

Fresh from Intel’s launch of the company’s latest third-generation Xeon Scalable “Ice Lake” processors on April 6 (Tuesday), Intel server partners Cisco, Dell EMC, HPE and Lenovo simultaneously unveiled their first server models built around the latest chips. And though arch-rival AMD may... Read more…

Intel Launches 10nm ‘Ice Lake’ Datacenter CPU with Up to 40 Cores

April 6, 2021

The wait is over. Today Intel officially launched its 10nm datacenter CPU, the third-generation Intel Xeon Scalable processor, codenamed Ice Lake. With up to 40 Read more…

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from Julialang.org. I Read more…

Intel Launches 10nm ‘Ice Lake’ Datacenter CPU with Up to 40 Cores

April 6, 2021

The wait is over. Today Intel officially launched its 10nm datacenter CPU, the third-generation Intel Xeon Scalable processor, codenamed Ice Lake. With up to 40 Read more…

CERN Is Betting Big on Exascale

April 1, 2021

The European Organization for Nuclear Research (CERN) involves 23 countries, 15,000 researchers, billions of dollars a year, and the biggest machine in the worl Read more…

Programming the Soon-to-Be World’s Fastest Supercomputer, Frontier

January 5, 2021

What’s it like designing an app for the world’s fastest supercomputer, set to come online in the United States in 2021? The University of Delaware’s Sunita Chandrasekaran is leading an elite international team in just that task. Chandrasekaran, assistant professor of computer and information sciences, recently was named... Read more…

HPE Launches Storage Line Loaded with IBM’s Spectrum Scale File System

April 6, 2021

HPE today launched a new family of storage solutions bundled with IBM’s Spectrum Scale Erasure Code Edition parallel file system (description below) and featu Read more…

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

Saudi Aramco Unveils Dammam 7, Its New Top Ten Supercomputer

January 21, 2021

By revenue, oil and gas giant Saudi Aramco is one of the largest companies in the world, and it has historically employed commensurate amounts of supercomputing Read more…

Quantum Computer Start-up IonQ Plans IPO via SPAC

March 8, 2021

IonQ, a Maryland-based quantum computing start-up working with ion trap technology, plans to go public via a Special Purpose Acquisition Company (SPAC) merger a Read more…

Leading Solution Providers

Contributors

Can Deep Learning Replace Numerical Weather Prediction?

March 3, 2021

Numerical weather prediction (NWP) is a mainstay of supercomputing. Some of the first applications of the first supercomputers dealt with climate modeling, and Read more…

Livermore’s El Capitan Supercomputer to Debut HPE ‘Rabbit’ Near Node Local Storage

February 18, 2021

A near node local storage innovation called Rabbit factored heavily into Lawrence Livermore National Laboratory’s decision to select Cray’s proposal for its CORAL-2 machine, the lab’s first exascale-class supercomputer, El Capitan. Details of this new storage technology were revealed... Read more…

New Deep Learning Algorithm Solves Rubik’s Cube

July 25, 2018

Solving (and attempting to solve) Rubik’s Cube has delighted millions of puzzle lovers since 1974 when the cube was invented by Hungarian sculptor and archite Read more…

African Supercomputing Center Inaugurates ‘Toubkal,’ Most Powerful Supercomputer on the Continent

February 25, 2021

Historically, Africa hasn’t exactly been synonymous with supercomputing. There are only a handful of supercomputers on the continent, with few ranking on the Read more…

The History of Supercomputing vs. COVID-19

March 9, 2021

The COVID-19 pandemic poses a greater challenge to the high-performance computing community than any before. HPCwire's coverage of the supercomputing response t Read more…

HPE Names Justin Hotard New HPC Chief as Pete Ungaro Departs

March 2, 2021

HPE CEO Antonio Neri announced today (March 2, 2021) the appointment of Justin Hotard as general manager of HPC, mission critical solutions and labs, effective Read more…

AMD Launches Epyc ‘Milan’ with 19 SKUs for HPC, Enterprise and Hyperscale

March 15, 2021

At a virtual launch event held today (Monday), AMD revealed its third-generation Epyc “Milan” CPU lineup: a set of 19 SKUs -- including the flagship 64-core, 280-watt 7763 part --  aimed at HPC, enterprise and cloud workloads. Notably, the third-gen Epyc Milan chips achieve 19 percent... Read more…

Microsoft, HPE Bringing AI, Edge, Cloud to Earth Orbit in Preparation for Mars Missions

February 12, 2021

The International Space Station will soon get a delivery of powerful AI, edge and cloud computing tools from HPE and Microsoft Azure to expand technology experi Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire