As high-performance computing (HPC) enters the petascale age, the scientific challenges facing researchers have never been greater. Nor has the might of today’s production petascale machines. The recent exponential growth in the power of modern supercomputers has gone hand-in-hand with an increased demand on resources — as machines have gotten bigger and faster, the amount of resources required for their operation has likewise increased.

As a result, HPC centers now face unprecedented power demands from the very machines they rely on to tackle today’s most daunting scientific challenges, from climate change to the modeling of biological processes. However, recent energy-saving innovations at ORNL are setting a new standard for resource-responsible HPC research. The laboratory has taken an all-angles approach, seeking energy savings from a suite of different areas.

ORNL’s leadership system, a Cray XT known as Jaguar, is now the fastest computer in the world for open science with a maximum speed of 1.6 petaflops. With this great power comes great responsibility, especially when it comes to energy consumption. “We take energy utilization very seriously,” said ORNL’s Leadership Computing Facility Project Director Buddy Bland. “The scale of this machine is just phenomenal. There are very few places in the world where this computer could have been built.”

Needless to say, feeding this animal is no small task: simulation at the petascale requires robust power and cooling networks to ensure maximum production from these machines. But now those necessary support networks, and the system itself, have been designed with unprecedented efficiency, responsibly satisfying Jaguar’s energy appetite. These advances make ORNL among the most energy-efficient locations for HPC, enabling groundbreaking research with minimal resource impact.

It all starts with the building. ORNL’s Computational Sciences Building (CSB) was among the first Leadership in Energy and Environmental Design (LEED)-certified computing facilities in the country, meaning that its design satisfies criteria used by the U.S. Green Building Council to measure the efficiency and sustainability of a building.

Take the computer room for example: it’s sealed off from the rest of the building by a vapor barrier to reduce the infiltration of humidity. The air pressure inside the computer room is slightly higher than the surrounding area so air will flow out of the computer room without the air outside flowing in.

Because ORNL is located in an area of the country with high humidity, keeping moisture out of the air is a high priority said Bland, one that the building was designed to tackle as efficiently as possible. Too much moisture in the air can lead to water condensation on equipment, while too little moisture can cause static electricity to build up — both of which can be problematic for a room filled with expensive electronics. Both removing moisture from or adding it to the air uses a lot of power, so keeping the humidity stable is a great tool for reducing energy consumption.

Another computing building on the ORNL campus adjacent to the CSB was recently certified LEED Gold, and Bland points out that the laboratory plans on an equal rating for future HPC facilities. But the innovation doesn’t stop with the building — there is plenty more under the roof.

Jaguar requires huge amounts of chilled water to keep the machine cool. To accomplish this as efficiently as possible, the laboratory uses high-efficiency chillers, which are the first step in a multifaceted, efficient cooling design.

A newly introduced Cray cooling system for Jaguar, dubbed ECOphlex, complements the chillers and the CSB’s efficiency. Using a common refrigerant and a series of heat exchangers, ECOphlex efficiently removes the heat generated by Jaguar to keep the computer room cool. The combination of air- and refrigerant-based cooling is much more efficient than traditional systems, which rely almost solely on air for temperature control. Without ECOphlex, the number of air-based units would not fit into the CSB’s computer room. This high-efficiency cooling system makes Jaguar possible.

ECOphlex also allows ORNL to reduce the amount of chilled water used to cool Jaguar by accommodating a broader inlet temperature range for the cooling water. Considering the fact that thousands of gallons of water per minute are necessary to keep Jaguar cool, a reduction in the volume of necessary chilled water means a proportionate reduction in the energy used to cool it. Simply put, warmer water can mean big energy savings for the laboratory and the taxpayer. Whereas most centers use 0.8 watts of power for cooling per every watt of power used for computing, ORNL enjoys a far more efficient ratio of 0.3 to 1, one of the lowest of all data centers measured.

Another important innovation is one that ORNL has been working on with Cray for several years. Instead of using the more common 208-volt power supply that Jaguar used in the past, the system now runs directly on 480-volt power. This seemingly “minor” change is saving the laboratory $1 million in the cost of copper used in the power cords for the cabinets. Furthermore, keeping the voltage high allows a lower current, which means lower resistance and less power turned into heat as it travels down the wires. The reduction in electrical resistance will reduce energy costs by as much as half a million dollars.

Finally, ORNL gets a little help from history. The power grid for the city of Oak Ridge was designed when the work conducted during the Manhattan Project used one-seventh of all the electricity in the country. The grid was constructed with every protection possible out of the fear that any interruption in supply would drastically set back development. The result: an extremely resilient local power grid.

Because of this grid, said Bland, Oak Ridge doesn’t need huge uninterruptable power supply (UPS) systems, which generally consume lots of electricity. However, the laboratory does have flywheel-based UPSs in case of an emergency. If there is a problem, the flywheel keeps generating power, which is a much more efficient process than conventional UPSs and therefore a greener method of supplying backup power. Because the flywheel-based UPS is mechanical as opposed to battery-operated, it also generates less waste in the long-term as battery replacement is not a concern.

While all of these steps are important, taken together they are greater than the sum of their parts. “There is no silver bullet,” said Bland. By tackling energy efficiency from multiple angles, ORNL is helping to ensure that the groundbreaking research taking place on its petascale machines is conducted as responsibly as possible, setting new standards in both HPC and energy responsibility.