Next-Generation 10GbE Switching

By Saqib Jang

June 22, 2007

High-end computing environments for high-performance cluster computing (HPCC) and internet data center (IDC) applications have embraced the scale-out server model. As a result, they have witnessed their data centers expand to include hundreds to thousands of servers running diverse operating systems and applications. As the number of servers has grown, so has the cost of operations, which includes people, space, power, and cooling.

In response, IT organizations are increasingly turning to technologies like utility computing, virtualization, and data center grids to transform data center resources from monolithic systems into agile, shared computing resource pools, which consist of uniform components that can be dynamically aggregated, tiered, provisioned, and accessed.

Harnessed together, these technologies have the potential to dramatically increase performance levels, maximize return on investment, and allow IT organizations to rapidly deploy and scale resources on-demand.

Scalable 10GbE Networking For High-Performance Data Centers

Central to the vision of virtualization and automation of HPCC and IDC data center resources is the deployment of grid computing environments with hundreds to thousands of servers running tightly coupled applications that are highly sensitive to client/server and server-to-storage communications bandwidth, as well as to inter-process message latency.

While a number of specialized interconnect technologies, such as InfiniBand and Fibre Channel, are available for building high performance data center networks, Gigabit Ethernet remains the dominant networking technology in HPCC and IDC environments. The continued evolution towards 10 Gigabit Ethernet (10GbE) server networking with the promise of rapidly declining prices and a stream of innovations, such as power and space-efficient network interface controllers (NICs) for dense server form factors, and support of RDMA, TCP offload engine (TOE), iSCSI, and I/O virtualization capabilities, make it the obvious choice as the upgrade path for high-performance data centers.

Chelsio Communications, a leading provider of 10GbE adapter solutions, offers a family of ‘unified wire’ adapters targeted at the massive installed base of Gigabit Ethernet networking infrastructure. Chelsio’s adapters enable the replacement of disparate fabric technologies such as Fibre Channel and InfiniBand in a wide range of applications, including NAS filers, SAN arrays, high performance cluster computing, and blade servers.

“Chelsio’s focus is on delivering the promise of the unified wire, enabling the convergence of server networking, storage networking and cluster computing interconnect onto a single 10GbE fabric,” said Kianoosh Naghshineh, president and CEO of Chelsio. “Ethernet is the fabric of convergence and we have sucessfully developed a broad family of adapter solutions that deliver all the required critical features, such as low latency, high transaction rate, and reduced cost.”

While vendors such as Chelsio are driving major improvements in 10GbE adapter cost and performance, 10GbE deployment, however, continues to be limited to aggregation of gigabit ports in HPCC and IDC environments, and inter-switch connectivity. A number of obstacles have to be overcome in order to fulfill the promise of broad-based 10GbE data center networking deployment within high-end data centers. The most important among these is the availability of affordable switching infrastructure delivering the scalability, price, performance, and resiliency required for end-to-end 10GbE deployment within high-end IDC and HPCC data centers.

Scaling Ethernet Networks

While the 10GbE switching segment has seen intensifying competition with new switch chip and system products every quarter, switch port density remains a challenge, especially when you compare them to switching options based on Gigabit Ethernet, Fibre Channel, and InfiniBand technologies. The biggest non-blocking 10GbE LAN switch has 128 ports and is priced at thousands of dollars per port. Even with Gigabit Ethernet, the largest non-blocking switch has about 600 ports.

While the bandwidth of a L2 Ethernet network is limited to the bandwidth of the largest Ethernet switch in the core, Fibre Channel and InfiniBand protocols and products enable fabrics (multi-stage meshes of switches) with non-blocking throughput supported across thousands of ports. But the problem with Fibre Channel and InfiniBand is that they don’t do IP very well and they don’t do Ethernet at all. And, there is another challenge with Fibre Channel: With low-cost scale-out servers, the cost of Fibre Channel host bus adapters (HBAs) is commonly more than 25 percent of the cost of the server.

So what is it about Ethernet that prevents multi-path mesh fabrics when both InfiniBand and Fibre Channel support them? It’s a consequence of the plug-and-play nature of Ethernet. Ethernet frames, per the standards, do not keep history. An Ethernet frame does not track elapsed time or count hops. This means that if Ethernet switches are connected in a loop (which multi-path meshes typically have), it is possible that a packet forwarded from the output port of a Ethernet switch may ultimately arrive to an input port of the same Ethernet switch and, without any other mechanisms to resolve the situation, the packet will flow indefinitely in a loop consuming bandwidth and ultimately creating a fully congested network unable to forward any other traffic.

To avoid this, Ethernet switches run the spanning tree protocol (STP) during initialization and periodically thereafter to accommodate changes in the physical network to detect and disable multiple paths automatically. Spanning tree disables paths that could create a loop.

Theoretically, you can build a mesh of spanning tree-enabled Ethernet switches, but because of the single-path constraint, the Ethernet network becomes subjected to numerous congestion points leading to dropped packets and throttling of injection rates in transmit nodes which leads to less than wire speed and non-deterministic performance. In other words, the spanning tree algorithm does not allow the use of Layer 2 Ethernet switches to build multi-path mesh fabrics, which is why the overall bandwidth of a Layer 2 Ethernet network is limited to the capacity of the largest switch in the core.

In order to overcome this limitation of Layer 2 Ethernet switches, it is necessary to use Layer 3 (and higher) IP switches. Using Layer 3 10GbE switches, even switches of 128 ports, severely impacts price and performance metrics (both bandwidth and latency) critical in the data center. According to IDC, Layer 3 switch ports cost, on average, five times as much as Layer 2 ports.

L3 routing functionality requires complex processing based on store-and-forward methods which thereby preclude the use of “cut-through” switches. The requirement for store-and-forward and advanced processing introduces significant switch latency, makes performance non-deterministic and greatly increases the cost of these switches.

Converged I/O Using InfiniBand: Congestion Collapse?

In contrast, InfiniBand fabrics do not have this problem because they include a Subnet Manager that is used to discover the physical topology and set up all the paths by configuring the forwarding tables in each InfiniBand switch.

InfiniBand’s fast point-to-point communication speeds have made it a favorite among the high-end HPCC environments. HPC applications typically operate doing short, discrete calculations on clusters of servers. Since servers quickly complete their calculations and must quickly refresh with new calculations, constantly passing them additional data to keep them occupied is a key part of making sure an HPC farm is run optimally. InfiniBand high bandwidth and low latency make it ideal to pass arguments between HPC servers.

But InfiniBand has a major problem which is now becoming apparent as InfiniBand is being considered for broad-based deployment within HPCC and IDC data centers: InfiniBand switches cannot drop packets to deal with congestion. As a result, switch buffers can fill up, block upstream switches and even block flows that are not contending for the congested link.

For small-scale application environments with predictable load, this is not a problem. However, for large-scale deployments spanning hundreds to thousands of servers supporting a range of applications, the lack of congestion control is potentially disastrous. Data presented at the Open Fabrics conference held in April 2007 in Sonoma shows that congestion in IB networks can occur with as few as 24 servers, dramatically increasing latency and decreasing throughput.

A converged fabric requires significant congestion control to prevent congestion collapse as I/O increases. This is especially important for enterprise-class components such as databases and application servers that may be deployed in mission-critical, customer-facing IDC environments with variable demand. A financial institution that runs a key trading application on a fabric without sufficient congestion control may suffer severe performance degradation or a complete fabric collapse during a heavy trading day.

Need for Low Latency

While latency has been a much-discussed aspect of high-end data center networking, it would be helpful to review the different ways latency is measured as well as its impact on real-world HPCC and Enterprise applications.

First, latency means different things to different people. The application vendors measure how long it takes from requesting the data to when they receive it. If the request is for a big block of data, obviously the pipe size has an impact. The adapter vendors measure the round trip time for a small packet with two back to back servers and divide it by two to estimate the one-way latency. Low latency is better than high latency, but this measure won’t always predict applications performance.

The network switch vendors measure the time from the head of a frame entering one switch, to the head of the frame exiting the switch. What this ignores is the impact of bandwidth on applications performance. As an example, if the DBMS cluster is moving 32K byte data blocks around, they will get to their destination a lot faster with a 10 Gbps pipe than they will with 1 Gbps.

However, scalable support of high-performance applications running on server grids is driving the need for dynamic, fine-grained calibration of end-to-end latency within data centers. While millisecond-scale latency across the WAN can be measured, that isn’t near good enough in the data center. In very controlled conditions, with very expensive test equipment, the latency across a data center network can be measured one port at a time. But in the real world, nobody knows what the latency of their data center network is. While server to server pings can generate round trip time (RTT) latencies, however, microsecond-scale, let alone nanosecond-scale, one-way latencies cannot be measured without special test gear. All that can be typically done is to measure the application response time, and guess where time is being consumed.

The importance of latency is also often debated. In the view of application vendors, the latency they worry about is in hundreds of milliseconds, so fixating on the LAN latency is in their view a waste of time. However, what they fail to take into account is that an application-level transaction may include hundreds of read/write acknowledgements. In these cases, high “stack-up” latencies can result in reduced application performance.

Latency is becoming more important because in scale-out datacenters as the system bus of large multiprocessor systems is effectively being replaced by the LAN. Further, in the grid computing vision, servers across the enterprise network can, in theory, can be dynamically added to the DBMS cluster. But it will only deliver acceptable performance if the combined latency across the multiple switch hops is consistently low.

The need for optimizing end-to-end network availability and performance, including latency, is also being driven by the evolution of server virtualization to enable data center automation. For example, VMWare VMotion capability allows application-ready software modules to be moved seamlessly between physical and virtual computing resources, dynamically provisioned on one or more servers, autonomously updated and patched according to user-definable compliance and security policies, and scheduled, executed, and tracked according to logical sequences, events, dependencies, and geographic hierarchies.

During a VMotion migration, the memory contents of the running virtual machine are transferred over the network to the target server. Having a robust, high-performance network available for this task is critical for ensuring that VMotion operations can complete in a timely manner and result in a successful migration. The bandwidth, latency, and availability of the network determine the effectiveness of the dynamic load-balancing and stateful failover capability enabled by VMotion.

High Cost of Bandwidth

Moore’s Law doesn’t exactly predict performance improvements, but a corollary to transistors doubling every 24 months does. Between clock rate improvements and architectural improvements, CPU performance has typically doubled about every 18 months. Although clock rate improvements are slowing down, multi-core processors and better chip I/O are helping keep performance gains on the same trajectory. What isn’t so obvious is that this means that every five years or so, processor performance goes up 10X.

The last big change in Ethernet speed got well underway by 1999 when IT departments were deploying Gigabit Ethernet server connections using the recently ratified 1000BaseTX standard for Category 5 UTP cable. In 1999 several GbE startups were hitting their stride as newly-minted public companies or business units of incumbent vendors, as their LAN switch sales grew at dizzying rates. While it wasn’t until 2003 that the shipments of Gigabit Ethernet LAN switch ports equaled 100Mbps Ethernet, by then, most IT organizations had lots of servers connected with Gigabit Ethernet.

In servers, the transition to GbE NICs is already complete, while 10GbE remains in the early stages of adoption, with a range of incumbent and startup vendors offering 10GbE server adapters. While 10GbE NIC shipments are growing from a small base, prices are falling rapidly, and the impending volume shipments of 10GBaseT and 10GbE-based blade server products foretell a dynamic and high-growth market.

However, other than the market leader, LAN switch suppliers likely do not have the ability to invest in R&D to develop the new architectures, the chips, and the software needed to improve the price/performance needed to make 10GbE a viable alternative for converged data center networking. The high per-port cost of incumbents’ enterprise-class LAN switches is inherited from their architectures, which are based on either buffered single-stage crossbars or WAN-based 10GbE switch design heritage incorporating sophisticated multi-class quality of service (QoS) capabilities and large, high-performance buffers for handling longer distance.

With every generation of LAN switching, new products have come from startups. In fact, the market leader has bought at least one such company every time. But the dotcom/telecom bubble aftermath has stifled 10GbE LAN switch startup investments. For all practical purposes there aren’t any.

Next-Gen 10GbE LAN Switching For Dynamic, Scale-Out Computing

While a decade ago, Ethernet switching was promoted as the answer to the problems with latency, complexity, and cost caused by using Layer 3 routers to connect compute farms together, LAN switch market incumbents have come to advocate Layer 3 routing over Ethernet switching, likely to gain higher revenues per connection, and to protect their turf from smaller competitors. The strongest arguments in favor of routing are that it eliminates broadcast storms, and it avoids spanning tree protocol-related problems as described above.

But Layer 3 protocols add their own complexity and restrict the flexibility to dynamically reconfigure servers and networks promoted by utility computing visionaries. For example, it’s a lot easier to add a server to an application cluster or enable dynamic migration of virtual servers within the same Layer 2 subnet. Lots of subnets lead to redundant servers for every subnet scattered around the datacenter.

What is a needed is a new approach coming most likely from innovative startups. Incumbents are too focused on protecting their revenue streams and their proprietary turf, which is why the adventuresome are, by process of elimination, looking outside the Ethernet world for the capabilities they need.

One of the start-ups looking to address the limitations of 10GbE LAN switching technology is Woven Systems, a Santa Clara, CA-based startup developing an Ethernet-based mesh network product.

“Woven’s focus is to deliver the best features of Fibre Channel and InfiniBand on a 10G Ethernet fabric,” says Dan Maltbie, founder and Sr. VP of Engineering, Woven Systems.

Woven is developing Layer 2-based 10G Ethernet data center switches that use special algorithms that allow the boxes to deliver a resilient multipath Ethernet fabric solution that delivers the low latency and scalability of InfiniBand coupled with the reliability of Fibre Channel. “Multiple paths can be established among switches in the fabric, allowing bandwidth to be allocated more dynamically over the paths, since traffic lanes are not shut down, as in spanning tree-based Ethernet”, Maltbie says.

In summary, IT capacity and requirements are on a collision course for high-performance computing and internet datacenters. Industry trends such as multi-core CPUs, blade computing, and virtualization are significant advances that have increased datacenter capacity. But the demand for IT compute and storage capabilities is expected to increase at an even faster rate. Ethernet, the staple of datacenter networking infrastructure, has kept pace with the availability of a range of 10GbE server networking adapters, but 10GbE switches have until recently lacked the performance, affordability, scalability and reliability required for pervasive 10GbE deployment within enterprise data centers.


About the Author

Saqib Jang is founder and principal at Margalla Communications, a Woodside, CA-based strategic and technical marketing consulting firm focused on storage and server networking. He can be contacted at [email protected].

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

AI Chip Start-up Groq to Detail Technology Progress in Fall

August 13, 2020

AI chip startup Groq announced yesterday it had closed its most recent funding round, saying the new investments will help it double in size by the end of this year and double again by the end of next year as it transiti Read more…

By John Russell

Intel Speeds NAMD by 1.8x: Saves Xeon Processor Users Millions of Compute Hours

August 12, 2020

Potentially saving datacenters millions of CPU node hours, Intel and the University of Illinois at Urbana–Champaign (UIUC) have collaborated to develop AVX-512 optimizations for the NAMD scalable molecular dynamics cod Read more…

By Rob Farber

Intel’s Optane/DAOS Solution Tops Latest IO500

August 11, 2020

Intel’s persistent memory technology, Optane, and its DAOS (Distributed Asynchronous Object Storage) stack continue to impress and gain market traction. Yesterday, Intel reported an Optane and DAOS-based system finishe Read more…

By John Russell

Summit Now Offers Virtual Tours

August 10, 2020

Summit, the second most powerful publicly ranked supercomputer in the world, now has a virtual tour. The tour, implemented by 3D platform Matterport, allows users to virtually “walk” around the massive supercomputer Read more…

By Oliver Peckham

Supercomputer Simulations Examine Changes in Chesapeake Bay

August 8, 2020

The Chesapeake Bay, the largest estuary in the continental United States, weaves its way south from Maryland, collecting waters from West Virginia, Delaware, DC, Pennsylvania and New York along the way. Like many major e Read more…

By Oliver Peckham

AWS Solution Channel

University of Adelaide Provides Seamless Bioinformatics Training Using AWS

The University of Adelaide, established in South Australia in 1874, maintains a rich history of scientific innovation. For more than 140 years, the institution and its researchers have had an impact all over the world—making vital contributions to the invention of X-ray crystallography, insulin, penicillin, and the Olympic torch. Read more…

Intel® HPC + AI Pavilion

Supercomputing the Pandemic: Scientific Community Tackles COVID-19 from Multiple Perspectives

Since their inception, supercomputers have taken on the biggest, most complex, and most data-intensive computing challenges—from confirming Einstein’s theories about gravitational waves to predicting the impacts of climate change. Read more…

Student Success from ‘Scratch’: CHPC’s Proof is in the Pudding

August 7, 2020

Happy Sithole, who directs the South African Centre for High Performance Computing (SA-CHPC), called the 13th annual CHPC National conference to order on December 1, 2019, at the Birchwood Conference Centre in Kempton Pa Read more…

By Elizabeth Leake

AI Chip Start-up Groq to Detail Technology Progress in Fall

August 13, 2020

AI chip startup Groq announced yesterday it had closed its most recent funding round, saying the new investments will help it double in size by the end of this Read more…

By John Russell

Intel Speeds NAMD by 1.8x: Saves Xeon Processor Users Millions of Compute Hours

August 12, 2020

Potentially saving datacenters millions of CPU node hours, Intel and the University of Illinois at Urbana–Champaign (UIUC) have collaborated to develop AVX-51 Read more…

By Rob Farber

Intel’s Optane/DAOS Solution Tops Latest IO500

August 11, 2020

Intel’s persistent memory technology, Optane, and its DAOS (Distributed Asynchronous Object Storage) stack continue to impress and gain market traction. Yeste Read more…

By John Russell

Summit Now Offers Virtual Tours

August 10, 2020

Summit, the second most powerful publicly ranked supercomputer in the world, now has a virtual tour. The tour, implemented by 3D platform Matterport, allows use Read more…

By Oliver Peckham

Research: A Survey of Numerical Methods Utilizing Mixed Precision Arithmetic

August 5, 2020

Within the past years, hardware vendors have started designing low precision special function units in response to the demand of the machine learning community Read more…

By Hartwig Anzt and Jack Dongarra

Implement Photonic Tensor Cores for Machine Learning?

August 5, 2020

Researchers from George Washington University have reported an approach for building photonic tensor cores that leverages phase change photonic memory to implem Read more…

By John Russell

HPE Keeps Cray Brand Promise, Reveals HPE Cray Supercomputing Line

August 4, 2020

The HPC community, ever-affectionate toward Cray and its eponymous founder, can breathe a (virtual) sigh of relief. The Cray brand will live on, encompassing th Read more…

By Tiffany Trader

Machines, Connections, Data, and Especially People: OAC Acting Director Amy Friedlander Charts Office’s Blueprint for Innovation

August 3, 2020

The path to innovation in cyberinfrastructure (CI) will require continued focus on building HPC systems and secure connections between them, in addition to the Read more…

By Ken Chiacchia, Pittsburgh Supercomputing Center/XSEDE

Supercomputer Modeling Tests How COVID-19 Spreads in Grocery Stores

April 8, 2020

In the COVID-19 era, many people are treating simple activities like getting gas or groceries with caution as they try to heed social distancing mandates and protect their own health. Still, significant uncertainty surrounds the relative risk of different activities, and conflicting information is prevalent. A team of Finnish researchers set out to address some of these uncertainties by... Read more…

By Oliver Peckham

Supercomputer-Powered Research Uncovers Signs of ‘Bradykinin Storm’ That May Explain COVID-19 Symptoms

July 28, 2020

Doctors and medical researchers have struggled to pinpoint – let alone explain – the deluge of symptoms induced by COVID-19 infections in patients, and what Read more…

By Oliver Peckham

Nvidia Said to Be Close on Arm Deal

August 3, 2020

GPU leader Nvidia Corp. is in talks to buy U.K. chip designer Arm from parent company Softbank, according to several reports over the weekend. If consummated Read more…

By George Leopold

Intel’s 7nm Slip Raises Questions About Ponte Vecchio GPU, Aurora Supercomputer

July 30, 2020

During its second-quarter earnings call, Intel announced a one-year delay of its 7nm process technology, which it says it will create an approximate six-month shift for its CPU product timing relative to prior expectations. The primary issue is a defect mode in the 7nm process that resulted in yield degradation... Read more…

By Tiffany Trader

Supercomputer Simulations Reveal the Fate of the Neanderthals

May 25, 2020

For hundreds of thousands of years, neanderthals roamed the planet, eventually (almost 50,000 years ago) giving way to homo sapiens, which quickly became the do Read more…

By Oliver Peckham

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

By Doug Black

HPE Keeps Cray Brand Promise, Reveals HPE Cray Supercomputing Line

August 4, 2020

The HPC community, ever-affectionate toward Cray and its eponymous founder, can breathe a (virtual) sigh of relief. The Cray brand will live on, encompassing th Read more…

By Tiffany Trader

Neocortex Will Be First-of-Its-Kind 800,000-Core AI Supercomputer

June 9, 2020

Pittsburgh Supercomputing Center (PSC - a joint research organization of Carnegie Mellon University and the University of Pittsburgh) has won a $5 million award Read more…

By Tiffany Trader

Leading Solution Providers


Nvidia’s Ampere A100 GPU: Up to 2.5X the HPC, 20X the AI

May 14, 2020

Nvidia's first Ampere-based graphics card, the A100 GPU, packs a whopping 54 billion transistors on 826mm2 of silicon, making it the world's largest seven-nanom Read more…

By Tiffany Trader

Australian Researchers Break All-Time Internet Speed Record

May 26, 2020

If you’ve been stuck at home for the last few months, you’ve probably become more attuned to the quality (or lack thereof) of your internet connection. Even Read more…

By Oliver Peckham

15 Slides on Programming Aurora and Exascale Systems

May 7, 2020

Sometime in 2021, Aurora, the first planned U.S. exascale system, is scheduled to be fired up at Argonne National Laboratory. Cray (now HPE) and Intel are the k Read more…

By John Russell

‘Billion Molecules Against COVID-19’ Challenge to Launch with Massive Supercomputing Support

April 22, 2020

Around the world, supercomputing centers have spun up and opened their doors for COVID-19 research in what may be the most unified supercomputing effort in hist Read more…

By Oliver Peckham

Joliot-Curie Supercomputer Used to Build First Full, High-Fidelity Aircraft Engine Simulation

July 14, 2020

When industrial designers plan the design of a new element of a vehicle’s propulsion or exterior, they typically use fluid dynamics to optimize airflow and in Read more…

By Oliver Peckham

John Martinis Reportedly Leaves Google Quantum Effort

April 21, 2020

John Martinis, who led Google’s quantum computing effort since establishing its quantum hardware group in 2014, has left Google after being moved into an advi Read more…

By John Russell

$100B Plan Submitted for Massive Remake and Expansion of NSF

May 27, 2020

Legislation to reshape, expand - and rename - the National Science Foundation has been submitted in both the U.S. House and Senate. The proposal, which seems to Read more…

By John Russell

Google Cloud Debuts 16-GPU Ampere A100 Instances

July 7, 2020

On the heels of the Nvidia’s Ampere A100 GPU launch in May, Google Cloud is announcing alpha availability of the A100 “Accelerator Optimized” VM A2 instance family on Google Compute Engine. The instances are powered by the HGX A100 16-GPU platform, which combines two HGX A100 8-GPU baseboards using... Read more…

By Tiffany Trader

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This