Next-Generation 10GbE Switching

By Saqib Jang

June 22, 2007

High-end computing environments for high-performance cluster computing (HPCC) and internet data center (IDC) applications have embraced the scale-out server model. As a result, they have witnessed their data centers expand to include hundreds to thousands of servers running diverse operating systems and applications. As the number of servers has grown, so has the cost of operations, which includes people, space, power, and cooling.

In response, IT organizations are increasingly turning to technologies like utility computing, virtualization, and data center grids to transform data center resources from monolithic systems into agile, shared computing resource pools, which consist of uniform components that can be dynamically aggregated, tiered, provisioned, and accessed.

Harnessed together, these technologies have the potential to dramatically increase performance levels, maximize return on investment, and allow IT organizations to rapidly deploy and scale resources on-demand.

Scalable 10GbE Networking For High-Performance Data Centers

Central to the vision of virtualization and automation of HPCC and IDC data center resources is the deployment of grid computing environments with hundreds to thousands of servers running tightly coupled applications that are highly sensitive to client/server and server-to-storage communications bandwidth, as well as to inter-process message latency.

While a number of specialized interconnect technologies, such as InfiniBand and Fibre Channel, are available for building high performance data center networks, Gigabit Ethernet remains the dominant networking technology in HPCC and IDC environments. The continued evolution towards 10 Gigabit Ethernet (10GbE) server networking with the promise of rapidly declining prices and a stream of innovations, such as power and space-efficient network interface controllers (NICs) for dense server form factors, and support of RDMA, TCP offload engine (TOE), iSCSI, and I/O virtualization capabilities, make it the obvious choice as the upgrade path for high-performance data centers.

Chelsio Communications, a leading provider of 10GbE adapter solutions, offers a family of ‘unified wire’ adapters targeted at the massive installed base of Gigabit Ethernet networking infrastructure. Chelsio’s adapters enable the replacement of disparate fabric technologies such as Fibre Channel and InfiniBand in a wide range of applications, including NAS filers, SAN arrays, high performance cluster computing, and blade servers.

“Chelsio’s focus is on delivering the promise of the unified wire, enabling the convergence of server networking, storage networking and cluster computing interconnect onto a single 10GbE fabric,” said Kianoosh Naghshineh, president and CEO of Chelsio. “Ethernet is the fabric of convergence and we have sucessfully developed a broad family of adapter solutions that deliver all the required critical features, such as low latency, high transaction rate, and reduced cost.”

While vendors such as Chelsio are driving major improvements in 10GbE adapter cost and performance, 10GbE deployment, however, continues to be limited to aggregation of gigabit ports in HPCC and IDC environments, and inter-switch connectivity. A number of obstacles have to be overcome in order to fulfill the promise of broad-based 10GbE data center networking deployment within high-end data centers. The most important among these is the availability of affordable switching infrastructure delivering the scalability, price, performance, and resiliency required for end-to-end 10GbE deployment within high-end IDC and HPCC data centers.

Scaling Ethernet Networks

While the 10GbE switching segment has seen intensifying competition with new switch chip and system products every quarter, switch port density remains a challenge, especially when you compare them to switching options based on Gigabit Ethernet, Fibre Channel, and InfiniBand technologies. The biggest non-blocking 10GbE LAN switch has 128 ports and is priced at thousands of dollars per port. Even with Gigabit Ethernet, the largest non-blocking switch has about 600 ports.

While the bandwidth of a L2 Ethernet network is limited to the bandwidth of the largest Ethernet switch in the core, Fibre Channel and InfiniBand protocols and products enable fabrics (multi-stage meshes of switches) with non-blocking throughput supported across thousands of ports. But the problem with Fibre Channel and InfiniBand is that they don’t do IP very well and they don’t do Ethernet at all. And, there is another challenge with Fibre Channel: With low-cost scale-out servers, the cost of Fibre Channel host bus adapters (HBAs) is commonly more than 25 percent of the cost of the server.

So what is it about Ethernet that prevents multi-path mesh fabrics when both InfiniBand and Fibre Channel support them? It’s a consequence of the plug-and-play nature of Ethernet. Ethernet frames, per the standards, do not keep history. An Ethernet frame does not track elapsed time or count hops. This means that if Ethernet switches are connected in a loop (which multi-path meshes typically have), it is possible that a packet forwarded from the output port of a Ethernet switch may ultimately arrive to an input port of the same Ethernet switch and, without any other mechanisms to resolve the situation, the packet will flow indefinitely in a loop consuming bandwidth and ultimately creating a fully congested network unable to forward any other traffic.

To avoid this, Ethernet switches run the spanning tree protocol (STP) during initialization and periodically thereafter to accommodate changes in the physical network to detect and disable multiple paths automatically. Spanning tree disables paths that could create a loop.

Theoretically, you can build a mesh of spanning tree-enabled Ethernet switches, but because of the single-path constraint, the Ethernet network becomes subjected to numerous congestion points leading to dropped packets and throttling of injection rates in transmit nodes which leads to less than wire speed and non-deterministic performance. In other words, the spanning tree algorithm does not allow the use of Layer 2 Ethernet switches to build multi-path mesh fabrics, which is why the overall bandwidth of a Layer 2 Ethernet network is limited to the capacity of the largest switch in the core.

In order to overcome this limitation of Layer 2 Ethernet switches, it is necessary to use Layer 3 (and higher) IP switches. Using Layer 3 10GbE switches, even switches of 128 ports, severely impacts price and performance metrics (both bandwidth and latency) critical in the data center. According to IDC, Layer 3 switch ports cost, on average, five times as much as Layer 2 ports.

L3 routing functionality requires complex processing based on store-and-forward methods which thereby preclude the use of “cut-through” switches. The requirement for store-and-forward and advanced processing introduces significant switch latency, makes performance non-deterministic and greatly increases the cost of these switches.

Converged I/O Using InfiniBand: Congestion Collapse?

In contrast, InfiniBand fabrics do not have this problem because they include a Subnet Manager that is used to discover the physical topology and set up all the paths by configuring the forwarding tables in each InfiniBand switch.

InfiniBand’s fast point-to-point communication speeds have made it a favorite among the high-end HPCC environments. HPC applications typically operate doing short, discrete calculations on clusters of servers. Since servers quickly complete their calculations and must quickly refresh with new calculations, constantly passing them additional data to keep them occupied is a key part of making sure an HPC farm is run optimally. InfiniBand high bandwidth and low latency make it ideal to pass arguments between HPC servers.

But InfiniBand has a major problem which is now becoming apparent as InfiniBand is being considered for broad-based deployment within HPCC and IDC data centers: InfiniBand switches cannot drop packets to deal with congestion. As a result, switch buffers can fill up, block upstream switches and even block flows that are not contending for the congested link.

For small-scale application environments with predictable load, this is not a problem. However, for large-scale deployments spanning hundreds to thousands of servers supporting a range of applications, the lack of congestion control is potentially disastrous. Data presented at the Open Fabrics conference held in April 2007 in Sonoma shows that congestion in IB networks can occur with as few as 24 servers, dramatically increasing latency and decreasing throughput.

A converged fabric requires significant congestion control to prevent congestion collapse as I/O increases. This is especially important for enterprise-class components such as databases and application servers that may be deployed in mission-critical, customer-facing IDC environments with variable demand. A financial institution that runs a key trading application on a fabric without sufficient congestion control may suffer severe performance degradation or a complete fabric collapse during a heavy trading day.

Need for Low Latency

While latency has been a much-discussed aspect of high-end data center networking, it would be helpful to review the different ways latency is measured as well as its impact on real-world HPCC and Enterprise applications.

First, latency means different things to different people. The application vendors measure how long it takes from requesting the data to when they receive it. If the request is for a big block of data, obviously the pipe size has an impact. The adapter vendors measure the round trip time for a small packet with two back to back servers and divide it by two to estimate the one-way latency. Low latency is better than high latency, but this measure won’t always predict applications performance.

The network switch vendors measure the time from the head of a frame entering one switch, to the head of the frame exiting the switch. What this ignores is the impact of bandwidth on applications performance. As an example, if the DBMS cluster is moving 32K byte data blocks around, they will get to their destination a lot faster with a 10 Gbps pipe than they will with 1 Gbps.

However, scalable support of high-performance applications running on server grids is driving the need for dynamic, fine-grained calibration of end-to-end latency within data centers. While millisecond-scale latency across the WAN can be measured, that isn’t near good enough in the data center. In very controlled conditions, with very expensive test equipment, the latency across a data center network can be measured one port at a time. But in the real world, nobody knows what the latency of their data center network is. While server to server pings can generate round trip time (RTT) latencies, however, microsecond-scale, let alone nanosecond-scale, one-way latencies cannot be measured without special test gear. All that can be typically done is to measure the application response time, and guess where time is being consumed.

The importance of latency is also often debated. In the view of application vendors, the latency they worry about is in hundreds of milliseconds, so fixating on the LAN latency is in their view a waste of time. However, what they fail to take into account is that an application-level transaction may include hundreds of read/write acknowledgements. In these cases, high “stack-up” latencies can result in reduced application performance.

Latency is becoming more important because in scale-out datacenters as the system bus of large multiprocessor systems is effectively being replaced by the LAN. Further, in the grid computing vision, servers across the enterprise network can, in theory, can be dynamically added to the DBMS cluster. But it will only deliver acceptable performance if the combined latency across the multiple switch hops is consistently low.

The need for optimizing end-to-end network availability and performance, including latency, is also being driven by the evolution of server virtualization to enable data center automation. For example, VMWare VMotion capability allows application-ready software modules to be moved seamlessly between physical and virtual computing resources, dynamically provisioned on one or more servers, autonomously updated and patched according to user-definable compliance and security policies, and scheduled, executed, and tracked according to logical sequences, events, dependencies, and geographic hierarchies.

During a VMotion migration, the memory contents of the running virtual machine are transferred over the network to the target server. Having a robust, high-performance network available for this task is critical for ensuring that VMotion operations can complete in a timely manner and result in a successful migration. The bandwidth, latency, and availability of the network determine the effectiveness of the dynamic load-balancing and stateful failover capability enabled by VMotion.

High Cost of Bandwidth

Moore’s Law doesn’t exactly predict performance improvements, but a corollary to transistors doubling every 24 months does. Between clock rate improvements and architectural improvements, CPU performance has typically doubled about every 18 months. Although clock rate improvements are slowing down, multi-core processors and better chip I/O are helping keep performance gains on the same trajectory. What isn’t so obvious is that this means that every five years or so, processor performance goes up 10X.

The last big change in Ethernet speed got well underway by 1999 when IT departments were deploying Gigabit Ethernet server connections using the recently ratified 1000BaseTX standard for Category 5 UTP cable. In 1999 several GbE startups were hitting their stride as newly-minted public companies or business units of incumbent vendors, as their LAN switch sales grew at dizzying rates. While it wasn’t until 2003 that the shipments of Gigabit Ethernet LAN switch ports equaled 100Mbps Ethernet, by then, most IT organizations had lots of servers connected with Gigabit Ethernet.

In servers, the transition to GbE NICs is already complete, while 10GbE remains in the early stages of adoption, with a range of incumbent and startup vendors offering 10GbE server adapters. While 10GbE NIC shipments are growing from a small base, prices are falling rapidly, and the impending volume shipments of 10GBaseT and 10GbE-based blade server products foretell a dynamic and high-growth market.

However, other than the market leader, LAN switch suppliers likely do not have the ability to invest in R&D to develop the new architectures, the chips, and the software needed to improve the price/performance needed to make 10GbE a viable alternative for converged data center networking. The high per-port cost of incumbents’ enterprise-class LAN switches is inherited from their architectures, which are based on either buffered single-stage crossbars or WAN-based 10GbE switch design heritage incorporating sophisticated multi-class quality of service (QoS) capabilities and large, high-performance buffers for handling longer distance.

With every generation of LAN switching, new products have come from startups. In fact, the market leader has bought at least one such company every time. But the dotcom/telecom bubble aftermath has stifled 10GbE LAN switch startup investments. For all practical purposes there aren’t any.

Next-Gen 10GbE LAN Switching For Dynamic, Scale-Out Computing

While a decade ago, Ethernet switching was promoted as the answer to the problems with latency, complexity, and cost caused by using Layer 3 routers to connect compute farms together, LAN switch market incumbents have come to advocate Layer 3 routing over Ethernet switching, likely to gain higher revenues per connection, and to protect their turf from smaller competitors. The strongest arguments in favor of routing are that it eliminates broadcast storms, and it avoids spanning tree protocol-related problems as described above.

But Layer 3 protocols add their own complexity and restrict the flexibility to dynamically reconfigure servers and networks promoted by utility computing visionaries. For example, it’s a lot easier to add a server to an application cluster or enable dynamic migration of virtual servers within the same Layer 2 subnet. Lots of subnets lead to redundant servers for every subnet scattered around the datacenter.

What is a needed is a new approach coming most likely from innovative startups. Incumbents are too focused on protecting their revenue streams and their proprietary turf, which is why the adventuresome are, by process of elimination, looking outside the Ethernet world for the capabilities they need.

One of the start-ups looking to address the limitations of 10GbE LAN switching technology is Woven Systems, a Santa Clara, CA-based startup developing an Ethernet-based mesh network product.

“Woven’s focus is to deliver the best features of Fibre Channel and InfiniBand on a 10G Ethernet fabric,” says Dan Maltbie, founder and Sr. VP of Engineering, Woven Systems.

Woven is developing Layer 2-based 10G Ethernet data center switches that use special algorithms that allow the boxes to deliver a resilient multipath Ethernet fabric solution that delivers the low latency and scalability of InfiniBand coupled with the reliability of Fibre Channel. “Multiple paths can be established among switches in the fabric, allowing bandwidth to be allocated more dynamically over the paths, since traffic lanes are not shut down, as in spanning tree-based Ethernet”, Maltbie says.

In summary, IT capacity and requirements are on a collision course for high-performance computing and internet datacenters. Industry trends such as multi-core CPUs, blade computing, and virtualization are significant advances that have increased datacenter capacity. But the demand for IT compute and storage capabilities is expected to increase at an even faster rate. Ethernet, the staple of datacenter networking infrastructure, has kept pace with the availability of a range of 10GbE server networking adapters, but 10GbE switches have until recently lacked the performance, affordability, scalability and reliability required for pervasive 10GbE deployment within enterprise data centers.


About the Author

Saqib Jang is founder and principal at Margalla Communications, a Woodside, CA-based strategic and technical marketing consulting firm focused on storage and server networking. He can be contacted at [email protected].

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

University of Chicago Researchers Generate First Computational Model of Entire SARS-CoV-2 Virus

January 15, 2021

Over the course of the last year, many detailed computational models of SARS-CoV-2 have been produced with the help of supercomputers, but those models have largely focused on critical elements of the virus, such as its Read more…

By Oliver Peckham

Pat Gelsinger Returns to Intel as CEO

January 14, 2021

The Intel board of directors has appointed a new CEO. Intel alum Pat Gelsinger is leaving his post as CEO of VMware to rejoin the company that he parted ways with 11 years ago. Gelsinger will succeed Bob Swan, who will remain CEO until Feb. 15. Gelsinger previously spent 30 years... Read more…

By Tiffany Trader

Roar Supercomputer to Support Naval Aircraft Research

January 14, 2021

One might not think “aircraft” when picturing the U.S. Navy, but the military branch actually has thousands of aircraft currently in service – and now, supercomputing will help future naval aircraft operate faster, Read more…

By Staff report

DOE and NOAA Extend Computing Partnership, Plan for New Supercomputer

January 14, 2021

The National Climate-Computing Research Center (NCRC), hosted by Oak Ridge National Laboratory (ORNL), has been supporting the climate research of the National Oceanic and Atmospheric Administration (NOAA) for the last 1 Read more…

By Oliver Peckham

Using Micro-Combs, Researchers Demonstrate World’s Fastest Optical Neuromorphic Processor for AI

January 13, 2021

Neuromorphic computing, which uses chips that mimic the behavior of the human brain using virtual “neurons,” is growing in popularity thanks to high-profile efforts from Intel and others. Now, a team of researchers l Read more…

By Oliver Peckham

AWS Solution Channel

Now Available – Amazon EC2 C6gn Instances with 100 Gbps Networking

Amazon EC2 C6gn instances powered by AWS Graviton2 processors are now available!

Compared to C6g instances, this new instance type provides 4x higher network bandwidth, 4x higher packet processing performance, and 2x higher EBS bandwidth. Read more…

Intel® HPC + AI Pavilion

Intel Keynote Address

Intel is the foundation of HPC – from the workstation to the cloud to the backbone of the Top500. At SC20, Intel’s Trish Damkroger, VP and GM of high performance computing, addresses the audience to show how Intel and its partners are building the future of HPC today, through hardware and software technologies that accelerate the broad deployment of advanced HPC systems. Read more…

Honing In on AI, US Launches National Artificial Intelligence Initiative Office

January 13, 2021

To drive American leadership in the field of AI into the future, the National Artificial Intelligence Initiative Office has been launched by the White House Office of Science and Technology Policy (OSTP). The new agen Read more…

By Todd R. Weiss

Pat Gelsinger Returns to Intel as CEO

January 14, 2021

The Intel board of directors has appointed a new CEO. Intel alum Pat Gelsinger is leaving his post as CEO of VMware to rejoin the company that he parted ways with 11 years ago. Gelsinger will succeed Bob Swan, who will remain CEO until Feb. 15. Gelsinger previously spent 30 years... Read more…

By Tiffany Trader

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from I Read more…

By John Russell

Intel ‘Ice Lake’ Server Chips in Production, Set for Volume Ramp This Quarter

January 12, 2021

Intel Corp. used this week’s virtual CES 2021 event to reassert its dominance of the datacenter with the formal roll out of its next-generation server chip, the 10nm Xeon Scalable processor that targets AI and HPC workloads. The third-generation “Ice Lake” family... Read more…

By George Leopold

Researchers Say It Won’t Be Possible to Control Superintelligent AI

January 11, 2021

Worries about out-of-control AI aren’t new. Many prominent figures have suggested caution when unleashing AI. One quote that keeps cropping up is (roughly) th Read more…

By John Russell

AMD Files Patent on New GPU Chiplet Approach

January 5, 2021

Advanced Micro Devices is accelerating the GPU chiplet race with the release of a U.S. patent application for a device that incorporates high-bandwidth intercon Read more…

By George Leopold

Programming the Soon-to-Be World’s Fastest Supercomputer, Frontier

January 5, 2021

What’s it like designing an app for the world’s fastest supercomputer, set to come online in the United States in 2021? The University of Delaware’s Sunita Chandrasekaran is leading an elite international team in just that task. Chandrasekaran, assistant professor of computer and information sciences, recently was named... Read more…

By Tracey Bryant

Intel Touts Optane Performance, Teases Next-gen “Crow Pass”

January 5, 2021

Competition to leverage new memory and storage hardware with new or improved software to create better storage/memory schemes has steadily gathered steam during Read more…

By John Russell

Farewell 2020: Bleak, Yes. But a Lot of Good Happened Too

December 30, 2020

Here on the cusp of the new year, the catchphrase ‘2020 hindsight’ has a distinctly different feel. Good riddance, yes. But also proof of science’s power Read more…

By John Russell

Esperanto Unveils ML Chip with Nearly 1,100 RISC-V Cores

December 8, 2020

At the RISC-V Summit today, Art Swift, CEO of Esperanto Technologies, announced a new, RISC-V based chip aimed at machine learning and containing nearly 1,100 low-power cores based on the open-source RISC-V architecture. Esperanto Technologies, headquartered in... Read more…

By Oliver Peckham

Azure Scaled to Record 86,400 Cores for Molecular Dynamics

November 20, 2020

A new record for HPC scaling on the public cloud has been achieved on Microsoft Azure. Led by Dr. Jer-Ming Chia, the cloud provider partnered with the Beckman I Read more…

By Oliver Peckham

NICS Unleashes ‘Kraken’ Supercomputer

April 4, 2008

A Cray XT4 supercomputer, dubbed Kraken, is scheduled to come online in mid-summer at the National Institute for Computational Sciences (NICS). The soon-to-be petascale system, and the resulting NICS organization, are the result of an NSF Track II award of $65 million to the University of Tennessee and its partners to provide next-generation supercomputing for the nation's science community. Read more…

Is the Nvidia A100 GPU Performance Worth a Hardware Upgrade?

October 16, 2020

Over the last decade, accelerators have seen an increasing rate of adoption in high-performance computing (HPC) platforms, and in the June 2020 Top500 list, eig Read more…

By Hartwig Anzt, Ahmad Abdelfattah and Jack Dongarra

Aurora’s Troubles Move Frontier into Pole Exascale Position

October 1, 2020

Intel’s 7nm node delay has raised questions about the status of the Aurora supercomputer that was scheduled to be stood up at Argonne National Laboratory next year. Aurora was in the running to be the United States’ first exascale supercomputer although it was on a contemporaneous timeline with... Read more…

By Tiffany Trader

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from I Read more…

By John Russell

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

By Doug Black

Programming the Soon-to-Be World’s Fastest Supercomputer, Frontier

January 5, 2021

What’s it like designing an app for the world’s fastest supercomputer, set to come online in the United States in 2021? The University of Delaware’s Sunita Chandrasekaran is leading an elite international team in just that task. Chandrasekaran, assistant professor of computer and information sciences, recently was named... Read more…

By Tracey Bryant

Leading Solution Providers


Top500: Fugaku Keeps Crown, Nvidia’s Selene Climbs to #5

November 16, 2020

With the publication of the 56th Top500 list today from SC20's virtual proceedings, Japan's Fugaku supercomputer – now fully deployed – notches another win, Read more…

By Tiffany Trader

Texas A&M Announces Flagship ‘Grace’ Supercomputer

November 9, 2020

Texas A&M University has announced its next flagship system: Grace. The new supercomputer, named for legendary programming pioneer Grace Hopper, is replacing the Ada system (itself named for mathematician Ada Lovelace) as the primary workhorse for Texas A&M’s High Performance Research Computing (HPRC). Read more…

By Oliver Peckham

At Oak Ridge, ‘End of Life’ Sometimes Isn’t

October 31, 2020

Sometimes, the old dog actually does go live on a farm. HPC systems are often cursed with short lifespans, as they are continually supplanted by the latest and Read more…

By Oliver Peckham

Nvidia and EuroHPC Team for Four Supercomputers, Including Massive ‘Leonardo’ System

October 15, 2020

The EuroHPC Joint Undertaking (JU) serves as Europe’s concerted supercomputing play, currently comprising 32 member states and billions of euros in funding. I Read more…

By Oliver Peckham

Gordon Bell Special Prize Goes to Massive SARS-CoV-2 Simulations

November 19, 2020

2020 has proven a harrowing year – but it has produced remarkable heroes. To that end, this year, the Association for Computing Machinery (ACM) introduced the Read more…

By Oliver Peckham

Nvidia-Arm Deal a Boon for RISC-V?

October 26, 2020

The $40 billion blockbuster acquisition deal that will bring chipmaker Arm into the Nvidia corporate family could provide a boost for the competing RISC-V architecture. As regulators in the U.S., China and the European Union begin scrutinizing the impact of the blockbuster deal on semiconductor industry competition and innovation, the deal has at the very least... Read more…

By George Leopold

Intel Xe-HP GPU Deployed for Aurora Exascale Development

November 17, 2020

At SC20, Intel announced that it is making its Xe-HP high performance discrete GPUs available to early access developers. Notably, the new chips have been deplo Read more…

By Tiffany Trader

HPE, AMD and EuroHPC Partner for Pre-Exascale LUMI Supercomputer

October 21, 2020

Not even a week after Nvidia announced that it would be providing hardware for the first four of the eight planned EuroHPC systems, HPE and AMD are announcing a Read more…

By Oliver Peckham

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This