There must have been a few companies that were a bit put off by the announcement of the Beacon computer as the Green500 champ last November. Everyone, that is, except Intel Corp.

Beacon, developed by the National Institute for Computational Sciences (NICS) at the University of Tennessee, started out as a research project to port software code to small clusters of Intel Many Integrated Core (MIC3) chips. When the group started working with the MICs, now better known as the Xeon Phi processor line, it found some compelling energy-saving possibilities. The NICS group decided to use the technology to create a small HPC cluster.

On Friday Eurotech and NVIDIA announced a new HPC system, called Eurora, that the companies say beats Beacon's energy efficiency by 26% – using NVIDIA GPUs instead of Intel Xeons, of course. With Eurora pumping out 315 gigaflops per watt versus Beacon's 299 gigaflops per watt, NVIDIA executives are happy that they can now claim the most energy-efficient HPC system uses their GPUs.

The most powerful supercomputer in the world, the Titan system at Oak Ridge National Lab, uses the same K20 GPUs as Eurora. Titan, however, finished third on the Green500 list. Eurora is NVIDIA's chance to take the title back from Intel.

Even with the advent of multicore CPUs, accelerators seem to be critical to creating today's most efficient HPC systems. NVIDIA technology, in particular, has made its presence felt on recent Green500 lists. "GPUs have been the biggest innovation in energy efficiency in the last five years," says Sumit Gupta, NVIDIA's general manager for Tesla GPU Accelerated Computing.

Eurora's efficiency was not just about GPUs, though. Everything about the system was created with the intention of optimizing power draw. The machine is part of Eurotech's latest Aurora Tigon line of HPC systems. Aurora was designed to decrease energy cost by using Eurotech's custom-built hot-water cooling system, enabling a datacenter PUE as low as 1.05.

To create Eurora, Eurotech replaced older generation NVIDIA GPUs in its previous Aurora systems with the latest Tesla K20 processors. NVIDIA's K20 Kepler generation GPU for the HPC market was also designed specifically to increase FLOPS per watt.

Next >>

The Eurora system was delivered to Cineca, an Italian non-profit consortium of universities and a member of the Partnership for Advanced Computing in Europe (PRACE). Cineca provides access to HPC systems for research institutions across Europe and prides itself as usually having one of the most powerful supercomputers in the world. Now, it seems, it can add energy efficiency to its boast. It's an important consideration in Europe, where energy prices tend to be much higher than those in the U.S., and where governments are increasingly demanding green technology.

Still, Eurora is essentially a prototype upon which to build. NVIDIA says it shipped a special version of its GPU to Eurotech. And Eurora, with a Linpack rating of 110 teraflops, is a small system compared to Titan at 17 petaflops. That makes the Oak Ridge machine 150 times more powerful than Cineca's.

NVIDIA's Gupta says it's not really fair to compare the reported energy efficiency for Beacon with that of other systems like Titan and Eurora. "Beacon was measured only for the Green500," he points out. "You can always create the ideal situation for that. Cineca is doing what they would do in production. Cineca is government funded and has regulations. They have to be mindful of the data they report."

Gupta notes that paradoxically, the energy-efficiency gains in HPC have been driven in large part by consumer products, not the need to cut energy costs for supercomputers or datacenters. The GPUs built to run electronic game graphics have to get by on as little energy as possible because of short battery life. Those same GPUs then started going into HPC systems, both to boost performance and to cut down on energy use.

NVIDIA's Kepler generation, released a couple years ago for the gaming market, is the processor that was really designed to reduce energy consumption as much as possible. The company streamlined the design by eliminating a lot of registers and logic that were not really needed, and increased the number of cores dramatically. With more cores working, the company could cut the clock speed of the processors by almost half. "Frequency has a bigger impact on power efficiency" than almost any other change one can make, says Gupta. "The simple trick is to increase the number of cores and decrease the frequency."

Still, Eurora is noticeably more energy efficient than the Titan, which uses the same generation of NVIDIA GPUs. So why the difference? A lot of it is due to Eurotech's on-board cooling system. That system delivers 50 degree C water to a heat exchanger – flat metal plates with water-flow systems attached. The plates sit directly in contact with the components on a circuit board to uniformly draw away heat. That's why NVIDIA had to provide a special version of its GPUs to Eurotech. They don't have the usual heat sinks attached, but accommodate a plate instead.

Still, given the efficiency of Beacon, there will be an ongoing contest between NVIDIA's GPUs and Intel's Xeon Phi coprocessors for the best performance per watt. The Intel Phi processors are not yet shipping beyond a few test systems such as Beacon, but will be more generally available within a few months. As these technologies move beyond their testing stages and into real production systems, supercomputers and datacenters should see real benefits.

And, of course everyone will be watching to see whose accelerators show up on top of the next Green500 list.