Russian supercomputer maker T-Platforms is continuing its push into the elite end of the HPC market. On Monday, the company unveiled a joint venture with a group at the University of Heidelberg to develop a new ultra-fast interconnect for high-end supercomputing. The goal is to bring the technology to market in the form of an ASIC, which can be incorporated into a network interface controller (NIC) for HPC servers.

The technology, called EXTOLL, for Extreme Low Latency Interconnect, was born out of a research project led by Prof. Ulrich Brüning in the university's Computing Architecture Group. The team there has created a working FPGA-based prototype for demonstration and software development purposes, but the final goal is to develop an ASIC for commercial production.

In a nutshell, EXTOLL is a switchless interconnect designed for ultra-low-latency, high bandwidth, and extreme scalability. The latter attribute is especially critical to the construction of large-scale HPC machines, which may contain tens of thousands of compute servers. According to Anton Korzh, a hardware architect at T-Platforms, the initial EXTOLL implementation will support up to 64 thousand nodes and at least a hundred cores per node. Even with the current generation of processors, that would allow systems to reach well into the multi-petaflop realm.

There is already an EXTOLL MPI software implementation in place, which has been developed and tested on the FPGA prototype. Since this technology is destined for petascale supercomputing, support for PGAS (Global Address Space) language environments is also in the works. Some support has to be baked into the operating system, and T-Platforms is planning to incorporate EXTOLL awareness into its own custom OS for HPC, ClustrX. In general, the proprietary software stack would be the biggest impediment to wider use of the technology.

Latency-wise, EXTOLL is aiming for sub-microsecond territory. The Heidelberg researchers believe they can achieve 500 nanoseconds (ns), NIC to NIC. No external switches are needed (switch logic is part of the NIC ASIC), and each hop in the network adds just 60 ns of delay. So even the worst cast latency for a 10,000 node supercomputer would be in the neighborhood of 3 microseconds -- assuming a 3D torus network design, which is what T-Platforms has in mind for its implementation.

Bandwidth, too, is aimed at the upper end of the spectrum. Each of the six EXTOLL links on the NIC will be capable of transferring 120 gigabits/second, which works out to about 90 GB/sec for a single device. That's about 2.5 times the speed of the current generation QDR InfiniBand, and puts it in the realm of the 160 GB/sec Cray Gemini interconnect used in the company's newest XE6 "Baker" supercomputers.

The original EXTOLL design was based on the HyperTransport protocol, which would have effectively limited its use in supercomputing to AMD Opteron-based servers. T-Platforms persuaded the university researchers to incorporate PCI-Express (PCIe) support as well, so they could build hardware with Intel silicon. The current roadmap will include support for both PCIe 3.0 and HyperTransport 3.0. It's worth noting that Cray's future system interconnect, named "Aries," will also support PCIe, and for exactly the same reason. The Aries technology is the follow-on to the Gemini interconnect, and will be used in the upcoming Cascade-class supercomputers.

While it's a stretch to start comparing T-Platforms to Cray, the Russian vendor seems to be following the Cray model of layering a proprietary interconnect on top of commodity x86 parts for its top-of-the-line supercomputers. The big difference is that T-Platforms bought into the technology rather than developing it in-house. The arrangement between T-Platforms and the University of Heidelberg gives the company an equity position in the joint venture. In exchange, T-Platforms is investing an undisclosed sum in the project to help move the technology into commercial production.

The idea is for the supercomputer maker to get first dibs on the new interconnect so that it can be incorporated into a future blade product aimed at the upper end of the HPC market. T-Platforms intends to keep its InfiniBand-based blade, as well, for those systems that don't require extreme scalability. According to Korzh, the company intends to move to a more modular blade design such that either interconnect technology (or perhaps even both) can be accommodated on the same basic motherboard.

The EXTOLL-based offering is already under development and is slated for launch in Q4 2011, when production of the NIC ASICs are scheduled to commence. The EXTOLL group is also interested in producing a stand-alone product that presumably would take the form of a PCIe-based network adapter that could be plugged into standard servers. That would make for a rather interesting setup for, say, a medium-sized supercomputer. Although the EXTOLL NICs are bound to be more expensive than their commodity InfiniBand or Ethernet brethren, the fact that one can do away with external switching could make for a compelling scale-out cluster model.

Getting the technology off the ground, however, is not going to come cheap. Typical costs for ASIC development alone can easily reach into the millions of dollars. Conveniently, T-Platforms recently announced an infusion of money from the state-run "Bank for Development and Foreign Economic Affairs" (Vneshekonombank), and although specific projects were not called out, the stated purpose of the investment was for "expanding T-Platforms' supercomputing research and development efforts, along with providing support for the company's expansion into the global HPC market." These maneuverings appear designed as part of a concerted strategy to expand the Russian company's reach into the European Union and beyond.

For the time being, though, Europe appears to be the primary target for T-Platforms' supercomputing aspirations. The Russian (and former Soviet Union) high-end supercomputing market is too small, while the US one, represented mostly by DOE labs, DoD research centers, and NSF supercomputing centers, is under the implicit mandate to buy American. That said, a differentiated high-end offering from T-Platforms could shake up the positions of established European market players like Bull, Cray and IBM, in particular, and create a more diverse set of supercomputing choices than even the US enjoys.