Summer is heating up, and so are our supercomputers. The insatiable drive for more computational performance means servers are becoming ever denser, and correspondingly hotter. Today, a rack of high-end blades can dissipate 30 kilowatts or more. And with the era of coprocessor acceleration upon us, many HPC servers are being fitted with 200-watt GPUs, further adding to the heat load.

This wouldn’t matter so much if we kept our machines in the pool, but air being what it is (a poor conductor of heat), the burden on the cooling infrastructure keeps escalating. Keeping the machinery at a comfortable temperature can represent from a third to a half of a facility’s power consumption. Even with the most recent recommendations from the American Society of Heating, Refrigerating and Air-conditioning Engineers (ASHRAE) to crank up the datacenter thermostat from 77F to 80F, HPC machine rooms are reaching their thermal limits.

That’s why liquid cooling has been such a big part of supercomputing. These machines, especially the proprietary designs, have always been on the leading edge of computational density, and sometimes wouldn’t survive on air flow alone. In fact, since the days of the early Cray systems in the 1970s, a lot of the top-end supercomputers have had water or some other liquid coolant running through the hardware. That’s why the father of supercomputing, Seymour Cray, referred to himself as “an overpaid plumber.”

A couple of recent stories point to a new direction for liquid cooling. Instead of just running coolant through the racks, it’s now being funneled directly onto the hottest components: the processors themselves. IBM’s Aquasar supercomputer, which was recently delivered to the Swiss Federal Institute of Technology Zurich (ETH Zurich), is an example of one such system.

The 6-teraflop Aquasar machine uses customized water-cooled BladeCenter servers that sport both Intel Nehalem CPUs and IBM PowerXCell processors. Water is piped into a heat exchanger that sits right on top of the chips. Because of the intimate contact with the processors, the water does not need to be chilled, and can be as warm as 60C. That’s 140F for those of you keeping score in the USA. The idea is to keep the processors below their critical maximum of 85C (185F).

At ETH Zurich, the heated (waste) water is piped away to help warm the buildings at the facility. IBM claims the carbon footprint of such a system is reduced by as much as 85 percent compared to a conventionally-cooled computer setup.

A more general case involves what Google is doing — or thinking about doing. The company recently filed to patent a server assembly design in which two motherboards sandwich a liquid-cooled heat sink. In this setup, the processors are being cooled via the heat sink, while the other components, like the memory chips, are air cooled. According to a report in Data Center Knowledge:

The design is among a number of Google patents on new cooling techniques for high-density servers that have emerged since the company’s last major disclosure of its data center technology in April 2009. Several of these patents deal with cooling innovations using either liquid cooling or air cooling applied directly on server components.

In 2007, Google filed a patent for a different sort of liquid-cooling arrangement. The “Water Based Data Center” design outlined sea-based computing facility that floats on the water, employs the waves to help generate electricity, and uses the sea water to help provide cooling for the computers. That patent was granted in May 2009.

Perhaps an even more novel method is immersion cooling, in which the whole server is submerged into an inert liquid, such as mineral oil. That too, is not a new concept. Some of the early supercomputing systems, including the Cray-2*, used immersion cooling. A modern version is being offered by Austin, Texas-based Green Revolution Cooling, which claims its horizonal rack design and “GreenDef” oil coolant can manage power densities as high as 100 kilowatts per rack. Bring on the GPUs!

The company is claiming its immersion system uses 95 percent less power than conventional cooling. Some of that can be attributed to the fact that all the internal server fans can be yanked out, which alone should reduce the power draw by 5 to 25 percent. The company recently installed some test units at the Texas Advanced Computing Center (TACC). If the Green Revolution offering pans out as advertised, maybe we’ll see more supers taking the plunge.

*The original post incorrectly specifed Cray-1 as one of the early supercomputers using immersion cooling.  It was the Cray-2 design that introduced this cooling design. Hat tips to readers Richard Lakein and Max  Dechantsreiter for pointing out the gaffe. — Michael