The combination of quad-core Opterons and DDR Infiniband is re-landscaping the HPC terrain and is propelling the largest clusters to the top of the high performance heap. A rash of recent announcements of big system purchases suggests good times ahead for HPC cluster vendors.

Or does it? Let's look at the good news first.

In the space of five days, over 500 teraflops of systems were announced -- all, coincidentally, for DOE labs. Last Friday, the Pacific Northwest National Laboratory (PNNL) revealed it had purchased a 163 teraflop HP Proliant server cluster. Then on Tuesday, Appro announced it had been awarded a contract to deliver 438 teraflops of computing hardware to three other DOE labs -- Lawrence Livermore, Los Alamos and Sandia. Add this to the 500 teraflop Ranger system currently being installed at the Texas Advanced Computing Center (TACC) and you have over a petaflop computing power -- all based on the new quad-core Opterons and DDR InfiniBand technology.

The Appro contract will result in eight new Linux clusters for the National Nuclear Security Administration (NNSA) Advanced Simulation and Computing (ASC) program, which supports the NNSA's mission to maintain the country's nuclear weapons. Three clusters are headed to Lawrence Livermore, two to Los Alamos and the remaining three to Sandia. Of these, the largest is a 162 teraflop machine for Livermore. The contract also contains an option for perhaps three additional clusters, representing 200 additional teraflops.

Prior to this, Appro had a presence at all ASC DOE labs, but nothing on this scale at Los Alamos and Sandia. The company's large three-cluster Peloton deployment at Livermore in 2006 was just a prelude to this week's ASC award. It's a huge win for Appro.

The contract also represents a new approach to purchasing computing resources by the ASC program, which chose to make a single large purchase for its capacity computing requirements for the next two years, instead of six smaller purchases over the same time period. Our feature article this week delves more deeply into the new clusters and the strategy behind the unified procurement for the three labs.

HP's 163 petaflop cluster, headed to PNNL in Richland, Washington, is just one of two 100-plus teraflop systems that the company will be delivering over the next year. The PNNL system will be used to drive the lab's molecular science research in applications such as aerosol formation, bioremediation, catalysis, climate change and hydrogen storage. The machine will be deployed in two phases and won't be fully operational until September 2008. The other system, a 182 teraflop blade-based cluster for an unnamed Swedish government agency, is expected to be fully operational as early as next month. That could give HP a more respectable presence in the top 10 of the Top500 list. The current top-ranked HP system is the 20.5 teraflop ASCI Q machine at number 62.

The dynamic duo of multicore x86 processors and InfiniBand should give clusters a real boost over the next few years. At least in raw performance, the top commodity-based clusters are now on par with the fastest high-end supercomputers from Cray and IBM. Until the petaflop-class XT and Blue Gene/P systems come online, the capability machines and the scaled-out clusters will vie for peak performance leadership. And while these clusters may not provide the same level of sustained application performance as their more expensive supercomputing brethren, their price-performance is about twice as good.

But the sub-100 teraflop cluster market is where the real market action is, and these systems are just getting less expensive and more popular every day. This makes them an increasingly attractive proposition for a wide range of users, which in turn makes them an increasingly attractive proposition for a wide range of vendors.

In a sense, that's the bad news. The promise of a fast growing market has attracted an abundance of companies. In general, the HPC cluster market is now over-served. There are a lot of cluster vendors out there without much fundamental product differentiation among them. Standard 64-bit x86 processors and InfiniBand interconnect technology have become the basis for hundreds of cluster computing offerings. These two elements -- compute and communications -- dominate how well a cluster performs.

Companies that have broken out of this mold, like SiCortex and (to a certain extent) Liquid Computing, have come up with truly innovative solutions that are unlike the traditional model. The challenge for them is gaining a foothold in a market that is risk-averse and extremely price sensitive. These companies have chosen a tough path, but at least they avoid direct competition with the 50 other system vendors selling what are essentially the same boxes.

To be fair, for the more mainstream cluster offerings, product differentiation does exist in technology, customization options, configuration, support, service, software bundling, storage offerings, etc., and these elements are all being applied for competitive advantage. But sometimes these advantages are only temporary. Rackable created a unique line of x86-based systems, with innovative power and cooling technology at competitive price points. Then everyone jumped onto the power/cooling bandwagon, dulling the benefit of Rackable's offerings and making them easy prey to aggressive pricing by Dell and HP. More recently, Rackable is fighting back with a mobile datacenter called Concentro, designed to go up against Sun Microsystems' new Blackbox offering. The lesson here is that vendors copy good ideas from each other to the extent possible, which, over time, marginalizes competitive advantages.

Since the HPC market -- and more specifically, the HPC cluster market -- is growing at double-digit rates, there has been little pressure for consolidation. Rapid growth is not conductive to the "normal" maturation process of eliminating weaker competitors and products. But I suspect the next sustained economic contraction will change that. With signs of inflation and rising interest rates starting to appear, an economic slowdown, if not full-blown recession, could happen next year.

Even in the absence of a deteriorating economy, there is constant pressure on buyers to look for purchasing efficiencies. The ASC program procurement strategy, mentioned above, is one such example. It could provide a model for how large organizations will buy high performance computing in the future. In the ASC case, the labs were specifically not looking for innovative computing. They already bought that in their high-end supercomputer systems, typified by Lawrence Livermore's Blue Gene/L. What they were looking for was a standard, replicable architecture for capacity computing that they could buy in bulk. But by replacing a number of smaller purchases with one large one, they explicitly limited the number of vendors in the ASC program.

Overall, the larger tier 1 system vendors seem to be least at risk, since they can offer more complete solutions, better support and a variety of services. They sell the same product lines into a broad customer base in both HPC and traditional enterprise computing. On the other hand, they are often less able to compete on price. The tier 3 shops fill a need for users looking for a one-night stand in high performance computing and can offer rock bottom prices. Tier 2 vendors provide an attractive middle ground, as was the case for the ASC procurement.

Whether market forces thin the herd quickly, through a recession, or more slowly, via a steady maturation of the ecosystem, depends upon future economic conditions. Over the long term, the larger vendors are going to be the most resilient. The smaller companies with narrower customer bases are already looking over their shoulder. Whatever the situation, if you're an x86 system vendor, hang on to your clusters. It's going to be a bumpy ride.

-----

As always, comments about HPCwire are welcomed and encouraged. Write to me, Michael Feldman, at editor@hpcwire.com.