HPCwire: InfiniBand Goes Long

The Leading Source for Global News and Information Covering the Ecosystem of High Productivity Computing

Leading HPC
Solution Providers

HPCwire >> Topic >> Interconnects

May 06, 2008

InfiniBand Goes Long

Michael Feldman, HPCwire Editor

Since InfiniBand came onto the scene, users have focused their efforts on using the high performance network fabric to connect compute and storage boxes within the datacenter. But a couple of enterprising companies, Network Equipment Technologies and Obsidian Research Corp., have developed InfiniBand connectivity for wide area networks (WANs). In both instances the vendors have developed solutions that can transparently connect IB clusters and storage over long distances -- hundreds or thousands of miles. From the application's viewpoint, the remote compute and storage nodes look and (more or less) act as if they're sitting right next to each other.

The benefits of long distance InfiniBand mirror its advantages in the datacenter -- namely high bandwidth and low latency. While the WAN InfiniBand performance won't always match local performance, solutions have demonstrated user data rates of up to 8 Gbps over thousands of miles across SONET OC-192 or 10 GbE backbones. Bandwidth and latency tends to drop a bit the farther you go, but unlike TCP/IP implementations, Quality of Service (QoS) is maintained.

Obsidian's Longbow IB WAN solution has been deployed at NASA Ames, Arizona State University and the University of Florida, and is being researched by Oak Ridge National Laboratory and Ohio State. The Longbow product has also been a feature at the last three Supercomputing (SC) conferences. Last year, Canadian-based Obsidian set up a subsidiary to go after the lucrative U.S. federal, intelligence and defense market spaces.

Network Equipment Technologies (NET) has a competitive product, the NX5010 InfiniBand bridge, a $100K+ box that is already fairly well-established in the U.S. DoD and Intelligence Community market. NET, a provider of a range of telecommunication platforms, got into the long distance InfiniBand market about a year and a half ago when its government customers started demanding long haul InfiniBand capability. Many of these federal organizations maintain a network of HPC sites dispersed across the country. These customers have developed a need to use wide area clusters to run some of their most critical MPI-based programs. Although the three-letter agencies don't talk about specific applications, wide area InfiniBand is a good fit for things such as dispersed intelligence gathering, network centric warfare, and general data mining.

NET's current InfiniBand offering, the 2U NX5010 box, works with any standard IB protocol. To the subnet manager, the NX5010 looks like a two-port InfiniBand switch. The device acts as a network bridge, converting the InfiniBand stream to the subnet manager's WAN protocol -- ATM, 10 Gigabit Ethernet, or whatever. At the other end, the companion NX5010 box attached to the remote cluster or SAN reverses the conversion. The magic is that the translation to and from the subnet protocol is performed at the 10 Gbps line rate, without losing the InfiniBand semantics or incurring a big latency penalty.

NET says they've sold about 100 NX systems so far. That's hardly a commodity market, but the company now thinks it can drive its solution into the commercial space. As InfiniBand adoption grows beyond HPC, NET is eyeing the demand for real time data capture on remote InfiniBand-equipped storage area networks. The company is looking at the financial market, where there is a real demand to synchronize streaming data in real time across storage silos. In particular, for these institutions, the need for remote disaster recovery (DR) may turn out to be the first killer app for long distance InfiniBand.

In the dot-com days, a number of financial firms on Wall Street bought a lot of dark fiber, which is still underutilized. NET is pitching them the idea of using this capacity for InfiniBand-enabled DR. "They have the bandwidth," says Haseeb Budhani, director of strategic planning for NET. "They just don't have a way to push the data." The traditional TCP/IP solution, which was never intended for high performance data transfer, incurs a heavy latency penalty, especially at longer distances.

NET is looking to piggyback onto deployments from Oracle, SAP, EMC, NetApp and system vendors as a way to enter the commercial market. The recent decision by Colfax International to offer NX 5000 systems alongside its high performance cluster gear is a development NET would like to see repeated with other system integrators and OEMs.

While NET is excited about connecting remote storage over IB, at this point, the company doesn't perceive a big demand for long haul computing over InfiniBand outside the government space. But in that market, the need for speed is unrelenting. NET is planning to introduce NX bridges that support 40 Gbps data rate later this year. These devices will be especially handy if you happen to be connected to a next-generation 40G OC-768 backbone.

But for most organizations, remote computing over high performance networks is still a bit too expensive. While NET expects to drive its NX boxes below $100K at some point, it still makes sense for the average HPC customer to expand their compute capacity on-site. As high performance network infrastructure becomes more commonplace and the InfiniBand ecosystem continues to mature, we may see a more general demand for IB-based wide area networking.

As the only two vendors of WAN InfiniBand gear, Obsidian and NET are in a good position to take advantage of those opportunities. From NET's perspective, Budhani would welcome more players, if only to validate the business opportunity. "More competitors absolutely make a case for the market," he says.

Article Tools

Share & Save Options

Discussion

There are 0 discussion items posted.

Join the Discussion

You must be a registered user in order to comment on HPCWire stories.

Members enjoy:

	Access to Special Content, including Podcasts, Webcasts, and Whitepapers and more!
	Ability to take an active role in the discussions that are shaping the HPC technology discipline!
	Special offers available only to HPCWire members!

Become a Registered User Today!

Registered Users Log in Here to Comment

Email Address:	Password (case sensitive)

Remember me	Forgot Password?

RSS Feeds

Feeds by Topic

Feeds by Industry

Feeds by Content Type

Feature Articles

The Linux HPC Empire Strikes Back

While the Microsoft juggernaut has been touting the joys of its new Windows HPC Server 2008, the Linux HPC contingent has been somewhat less vocal of late. But now Red Hat has come up with its version of an integrated cluster solution.
Read More...

Nexsan Looks to Scare Up HPC Customers With Storage Beast

Even though the cost of servers still dominates the datacenter budget, storage is actually on a steeper growth curve. HPC storage, in particular, is being singled out as high-growth opportunity. Vendors are scrambling to keep up.
Read More...

The Week in Review

Google datacenters most energy efficient; Cluster Resources to demo Moab Hybrid Cluster; Red Hat Linux releases HPC distro. John West recaps those stories and more in our weekly wrap-up.
Read More...

Read more HPCwire features...

Top Headlines

Oracle and HP's Database Machine Predicated on Voltaire

Oct 06 | The Register | Does the HP Oracle Database Machine represent InfiniBand's big chance to break out its HPC niche? Read more...

3D Imaging Spreads to Fashion and Beyond

Oct 06 | BusinessWeek | A body scan can save a lot of time in the fitting room, and fields from medicine to architecture are adopting 3D computing applications. Read more...

Structural Engineers and Computer Scientists Hope to Integrate Disciplines to 'Revolutionize Building Construction'

Oct 03 | UCSD News | Despite the evolution of computer science over the past 30 years, structural engineering -- hindered by a reluctance to adapt to digital innovations -- has remained relatively unchanged as a discipline. Read more...

Credit Crisis Spreads a Pall Over Silicon Valley

Oct 02 | New York Times | Silcon Valley is starting to feel the effects of the credit crunch. Read more...

Google: 'The World's Most Efficient Data Centers'

Oct 01 | Data Center Knowledge | Google today disclosed details of its data center energy usage, confirming that it operates some of the most efficient facilities in the world. Read more...

Read more headlines...

Featured Whitepapers

Panasas® Tiered Parity™ Architecture

Sep 04 | | Disk drives are approximately 250 times denser today than a decade ago. This is good news for users who are creating, manipulating and storing more data than ever before. It gives them an opportunity to derive more value from their stored data and lowers the capital acquisition and operating expense associated with that data.

View the White Paper Library

Multimedia

Video White Paper: Architecting a Better Network Storage Solution

BlueArc's Titan architecture represents an evolutionary step in file servers by creating a hardware-based file system that can scale bandwidth, IOPS, and overall data capacity well beyond conventional software-based devices. With its ability to virtualize a massive storage pool of up to four usable petabytes of tiered storage, Titan can scale with growing data requirements, offering a competitive advantage for businesses, researchers, or other enterprises seeking to better manage data growth while still ensuring optimal performance.