On Tuesday Appro announced the latest update to its Xtreme-X1 line of supercomputers. The new X1 has the Nehalem EPs of course (Xeon 5500s), but one of the significant innovations was the use of dual quad data rate (QDR) InfiniBand chips on the motherboard to create two InfiniBand networks (or "rails") in the system.

With single data rate and double data rate (DDR) InfiniBand, it was possible to use a single InfiniBand host channel adapter (HCA) -- or a single chip with multiple ports -- connected to a single PCI Express bus and not swamp the bandwidth of the PCI Express link. Appro has offered multi-rail clusters in the past, configured in a variety of ways, including using multiple HCAs, a single IB chip and PCI Express channel combination directly on the motherboard with a separate IB link via an HCA card and, most recently, a single Mellanox ConnectX chip on the motherboard with two DDR ports on a single PCI Express link.

With this latest product announcement, however, Appro moves to two QDR Mellanox ConnectX InfiniBand chips directly mounted on the motherboard, each connected to the node via its own x8 PCI Express v2.0 channel. The custom motherboards Appro is using for this application (built by a "large contract manufacturer in Asia," who the company wasn't interested in naming) also provide another x8 PCI Express v2.0 channel for whatever additional capability a specific cluster might require; Fibre Channel storage, for example, or a 10 Gbps Ethernet network.

For a while this week Appro had the only product on the market with dual QDR IB chips mounted on the motherboard with their own PCI Express channels. Then Bull announced its bullx family of HPC servers based on a new blade architecture in which a single node supports two x16 and two x8 PCI Express interfaces, enabling the node to support both the dual QDR IB chips on the motherboard and dual on-board GPUs.

Why directly mount the InfiniBand silicon to the motherboards? After all, since they are still using the PCI Express interface, why not just plug in HCAs and avoid the added engineering? John Lee, vice president of Appro's Advanced Technology Solutions Group, says that, while performance is "just about the same" in either the on-board or HCA-based implementations, reliability and density in very large clusters improves with fewer physical connectors. Also, cost is a driving factor. Lee indicates that there is a "20-30% savings" from using ConnectX chips directly on the motherboard versus buying InfiniBand ports on HCAs. In a very large cluster -- the kind that Appro has been building recently for DoE and the University of Tsukuba in Japan -- that cost savings can really add up.

One can draw a line from the idea that buying the ConnectX chips and directly mounting them on the motherboards saves money to a future revenue problem for Mellanox's HCA business. Of course, they make the silicon, too, so they are in the deal either way, but profits are probably lower for the chip. Appro and Bull aren't the only ones to have done the math on this, and many motherboard manufacturers have been directly mounting IB silicon on their motherboards for some time now. This probably helps to explain Mellanox's recent moves to diversify its revenue, for example moving into 10 Gb Ethernet, InfiniBand switches and multi-protocol gateways.

It is interesting to see two vendors launch dual-rail QDR InfiniBand cluster architectures at the same time, both using InfiniBand chips mounted directly on the motherboards rather than via HCAs. Appro's Lee explained in an interview with HPCwire that while "there is not a lot of leadership in multi-rail adoption right now" they expect to see increasing customer uptake in the market for multi-rail clusters in late 2009 and 2010. Although the idea itself is not new, the increasing socket count in clusters will drive growth in the amount of data that needs to be moved around a cluster at a faster rate than the InfiniBand roadmap will increase bandwidth.