Visit additional Tabor Communication Publications
September 03, 2009
This week Mellanox announced a refinement of its ConnectX line with the ConnectX-2 architecture. This latest evolution enhances its combination InfiniBand/Ethernet network adapter cards with new features, such added support for IEEE DCB standards and enhanced RDMA access, while maintaining the advantages of the previous line for those needing to support multiple protocols with limited server real estate.
The ConnectX-2 family of controller chips and adapter cards comes in a variety of flavors supporting Ethernet, InfiniBand, and (most interestingly) both. The ConnectX-2 EN/ENt cards support 10 gigabit Ethernet (GbE) with options for CX4, SFP+, and 10GBASE-T connections, while the IB cards support 10/20/40Gb/s InfiniBand with CX4 or QSFP connections. The favorite children in the family, however, are clearly the ConnectX-2 Virtual Protocol Interface (VPI) cards that support both 10/20/40Gb/s InfiniBand and 10 GbE on a single Converged Network Adapter (CNA). Each card sports two ports, one IB and one Ethernet, and come in either CX4 variant (both ports) or a QSFP/SFP+ version.
So, why might one want a single card that supports both interconnects? There is a lot of talk about something called convergence, most of which centers on whether or not everything will, or will not, eventually end up running on Ethernet. Even if you aren't a datacenter networks person, you have probably heard of Fibre Channel over Ethernet (FCoE), and there are other examples as well. Proponents say that Ethernet is already deployed everywhere and a single fabric will focus R&D efforts and streamline deployments. Opponents say that one size never really does fit all, and by the time you finish fixing the problems of Ethernet relative to purpose-built protocols (Ethernet is a best effort protocol with no flow control), you've lost all the advantages of convergence in lower performance and system complexity.
Whichever side you come down on here (if indeed you have a side at all), there is a clear advantage for HPC and cluster builders with the ConnectX-2 family of adapters, and that's in server real estate and cabling. Although many applications will use the ConnectX-2 in either Ethernet or IB mode, the VPI card supports both simultaneously. In the latest TOP500 list, 30 percent of clusters have InfiniBand interconnects, and the VPI card will allow cluster designers to have an IB network for cluster communications and support access to Lustre storage over 10 GbE, or other permutations (an Ethernet control network and an IB network for data communications, and so on). In fact, Lawrence Livermore is using the VPI card in precisely this mode:
"This technology allows us to provide greater high-performance computing resources to researchers in our national security programs by simplifying the design, and lowering the cost and power requirements of our scalable units for scientific simulation clusters," said Mark Seager, assistant department head for advanced technology at Lawrence Livermore National Laboratory. "In addition, these new adapters enable higher Lustre file system performance with greater connection flexibility between the InfiniBand cluster interconnect and our 10 Gigabit Ethernet storage area network."
You could also imagine, for example, provisioning a cluster with two data communications networks, and tailoring the network to the workload.
Among the improvements in this version of the product family are support for IEEE's 802.1 Data Center Bridging (DCB) specifications and hardware offload support for improved FCoE performance. The new cards also use less power: 35 percent less on the 10GbE side, and 15 percent less for IB. The InfiniBand port supports up to 40 Gb/s bandwidth with 1 microsecond latencies; on the Ethernet port the cards support 10Gb/s bandwidth with 6 microsecond TCP latency or 3 microsecond RDMA latency. Kernel bypass is also available for Low Latency Ethernet environments. ConnectX-2 samples are available today, and the products are expected to be generally available in October.
There are other vendors offering converged networking solutions, but, in general, the available solutions today -- including Mellanox's offering -- are outstanding in only a few of the possible areas of interest. For example, Brocade offers a CNA that works well for storage and server networks with support for both FCoE and iSCSI. The Mellanox ConnectX-2 family seems to hold a lot of promise for combined storage and low latency server networking.
As Brian Sparks of Mellanox said when I talked with him about this announcement, "It really is hard for a single technology to be great at both LAN and high performance local interconnect." The analogy he used in our discussion was the displacement of magnetic disk drives by new technologies like optical and SSD. Each time, the new technologies have opened up new areas of application, and taken a little share from the magnetic incumbents, but at the end of the day, there was a place where each technology was clearly superior. If there is an ultimate convergence, it will be a long time out, but until then, Mellanox is well positioned to sell to all sides of the debate.
May 22, 2013 |
At some point in the not-too-distant future, building powerful, miniature computing systems will be considered a hobby for high schoolers, just as robotics or even Lego-building are today. That could be made possible through recent advancements made with the Raspberry Pi computers.
May 16, 2013 |
When it comes to cloud, long distances mean unacceptably high latencies. Researchers from the University of Bonn in Germany examined those latency issues of doing CFD modeling in the cloud by utilizing a common CFD and its utilization in HPC instance types including both CPU and GPU cores of Amazon EC2.
May 15, 2013 |
Supercomputers at the Department of Energy’s National Energy Research Scientific Computing Center (NERSC) have worked on important computational problems such as collapse of the atomic state, the optimization of chemical catalysts, and now modeling popping bubbles.
May 10, 2013 |
Program provides cash awards up to $10,000 for the best open-source end-user applications deployed on 100G network.
05/10/2013 | Cleversafe, Cray, DDN, NetApp, & Panasas | From Wall Street to Hollywood, drug discovery to homeland security, companies and organizations of all sizes and stripes are coming face to face with the challenges – and opportunities – afforded by Big Data. Before anyone can utilize these extraordinary data repositories, however, they must first harness and manage their data stores, and do so utilizing technologies that underscore affordability, security, and scalability.
04/15/2013 | Bull | “50% of HPC users say their largest jobs scale to 120 cores or less.” How about yours? Are your codes ready to take advantage of today’s and tomorrow’s ultra-parallel HPC systems? Download this White Paper by Analysts Intersect360 Research to see what Bull and Intel’s Center for Excellence in Parallel Programming can do for your codes.
In this demonstration of SGI DMF ZeroWatt disk solution, Dr. Eng Lim Goh, SGI CTO, discusses a function of SGI DMF software to reduce costs and power consumption in an exascale (Big Data) storage datacenter.
The Cray CS300-AC cluster supercomputer offers energy efficient, air-cooled design based on modular, industry-standard platforms featuring the latest processor and network technologies and a wide range of datacenter cooling requirements.