Startup Makes Liquid Cooling an Immersive Experience

By Michael Feldman

August 31, 2010

There’s nothing like a blazing hot summer to focus one’s attention on the best ways to keep cool. That goes for datacenter operators as well, who are equally worried about keeping their servers properly chilled. While there is no shortage of innovative cooling solutions being proffered by various vendors, a new liquid immersion cooling solution from startup Green Revolution Cooling could end up being the best of them all.

The stakes for more efficient datacenter cooling are already high. Power consumption for a traditional air-cooled facility eats up a third to more than a half of the energy cost. Making cooling more efficient leaves more money available for computing, which, after all, is the central purpose of the datacenter. Efficient cooling is an especially important consideration in high performance computing, since this class of users gravitate toward faster and denser (and thus hotter) server configurations. If the setup in the center is not optimal, you end up sacrificing a lot of FLOPS for cooling.

With the increasing density of servers, storage, switches and other equipment, facility managers are taking an extra hard look at liquid cooling. Water-cooled servers have been around for decades, and direct-cooled CPUs are now being offered by a handful of vendors. Submerged liquid cooling, too, has been around since the days of the Cray 2, but this technology may be poised for a big comeback.

Servers Take a Bath

Green Revolution Cooling (GRC), a two year-old company based in Austin, Texas, is offering a general-purpose liquid immersion cooling solution that they introduced at SC09 in Portland last November. It was selected as one of the “Disruptive Technologies of the Year” for the 2009 conference, an award they’ve recaptured for SC10.

In a nutshell, the system consists of a 42U rack enclosure tipped on its back and filled with an inert mineral oil mixture in which you immerse the server hardware. A pump is used to circulate the oil to an external heat exchanger, typically located outside the building.

The big advantage is that, unlike water, the oil formulation is not electrically conductive, but has 1,200 times the heat capacity of air. And since the oil is in direct contact with all the components, it only needs to be cooled down to about 104F (40C) to be effective. (CPUs can operate at 75C and hard drives at 45C.) Unless your datacenter happens to be located in Yuma, Arizona, cooling a liquid to 40C is relatively easy to attain with a simple heat exchanger or cooling tower. The solution is advertised to reduce the cooling energy by 90 percent and cut overall power consumption in the datacenter by up to 45 percent. The pitch is that a single 10kW server rack at 8 cents per kWh will save over $5,000 per year on energy costs alone.

According to Green Revolution co-founder Christiaan Best, basically any piece of datacenter equipment — rackmount server, blade, switch — that adheres to the standard 19-inch form factor can be slid into the GRC enclosure. The only equipment modifications required are the removal of the internal fans (you don’t need air cooling any more) and the sealing of any hard drive units, with an epoxy coating, to make them airtight. Typically this procedure takes a few minutes per server.

Because the GRC enclosure is laid on its back, it does takes up more floor space than a regular vertical rack. But since you no longer need hot aisles, chillers, and CRAC units, there is extra square footage to play with. Also, because there is no need to run cold air beneath the equipment anymore, the raised floor is now superfluous. “Essentially you could run it in a barn,” says Best. “All you need is a level floor.”

If you’re looking for performance, the GRC rack allows you to overclock the processors without worrying about melting the server. An NSF-funded study found that cranking up the clock on an Intel E5520 “Nehalem” CPU inside a GRC-cooled server yielded a 54 percent performance boost on Linpack, while keeping the CPU temperature at 76C. The server cost per gigaflop was reduced by about 50 percent.

It’s not just for overclocking. Theoretically, you could throw almost any sort of artificially dense board — multi-GPUs servers, custom blades with 10 CPUs on the motherboard, etc. — into the oil bath and realize the additional cost benefit of shrinking down your hardware footprint.

One possible roadblock to widespread adoption is the lack of warranty support from the OEMs. Warranties don’t typically allow the customer to take the server apart and dunk it into foreign liquids. According to Best, they’ve been talking with all the major OEMs to get their solution qualified under the original warranties, but currently none have committed to supporting the GRC setup. Since many of the big system vendors have their own liquid cooling solutions they’d like sell, they are likely to be less than enthusiastic to qualify a third-party solution.

In any case, Best says they’ve retained third-party support that will honor the original equipment warranties, so customers can be covered for any mishaps. GRC has logged over a quarter million server hours on their in-house test system and has yet to encounter a failure (with the exception of hard drive mechanical failures). Although there is no data to support it, Best is fairly certain that their solution will extend the life of the servers, given the more stable thermal environment, the lack of vibration from internal fans, and the elimination of oxidation on the electrical contacts.

Looking for a Few Brave Customers

Austin-based Midas Networks, a collocation firm, is the company’s first customer. Midas has purchased four of the GRC racks, and the systems are scheduled to be up and running later this year. Best says they also have a number of other customers in the pipeline, including some with HPC facilities, but no checks are in the bank just yet.

With the exception of Green Revolution itself, the Texas Advanced Computing Center (TACC) has acquired the most experience with the technology. TACC installed a pre-production GRC unit back in April and has been putting the system through its paces for the past five months.

Even in oil-rich Texas, energy is not cheap, so power savings has become a big priority at TACC. “We’re really, really chill-water limited where we are now,” says Dan Stanzione, TACC’s deputy director. According to him, they don’t have the ability to add any more chilled water capacity, but do have plans to expand computing capability over the next several years.

The TACC experiment started with immersing some older 1U servers in the GRC enclosure, and since then they’ve added other equipment including InfiniBand switches, GPU-powered servers, and blades. According to Stanzione, all the hardware has performed flawlessly, with no failures to date. They’ve even overclocked some of the server CPUs by 30 to 40 percent, without incident.

At present they have about 10kW of equipment in the rack, and are using just 250 watts to power the GRC solution. That’s more than a 90 percent reduction when compared to the 3,000 to 4,000 watts they would have consumed with a conventional air-cooled system. Stanzione estimates the total power savings for the whole system (equipment plus cooling) was reduced by 25 to 30 percent. “The overall power consumption has been fantastic,” he says.

The TACC crew is going to continue collecting data with the GRC system for the rest of the year. If everything checks out, Stanzione would like to start putting some production units into the upcoming datacenter buildout. They’re already thinking about loading 30 to 40 kW of compute equipment into a single rack, and GRC cooling would make that level of density quite practical. Further into the future, Stanzione is thinking about the cost savings they could accrue by immersing all 140 racks of the center’s equipment. “I think this has a tremendous amount of potential,” he says.

Barring some unforseen technological breakthrough, datacenter computing is only going to get denser and hotter in the years ahead. And since the cooling capacity of air isn’t going to change, the move to liquid-cooled systems appears all but inevitable. “You may not buy liquid cooling from us,” concludes Best, “but you will buy it from someone.”

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

World Cup is Lame Compared to This Competition

June 18, 2018

So you think World Cup soccer is a big deal? While I’m sure it’s very compelling to watch a bunch of athletes kick a ball around, World Cup misses the boat because it doesn’t include teams putting together their ow Read more…

By Dan Olds

IBM Demonstrates Deep Neural Network Training with Analog Memory Devices

June 18, 2018

From smarter, more personalized apps to seemingly-ubiquitous Google Assistant and Alexa devices, AI adoption is showing no signs of slowing down – and yet, the hardware used for AI is far from perfect. Currently, GPUs Read more…

By Oliver Peckham

Sandia to Take Delivery of World’s Largest Arm System

June 18, 2018

While the enterprise remains circumspect on prospects for Arm servers in the datacenter, the leadership HPC community is taking a bolder, brighter view of the x86 server CPU alternative. Amongst current and planned Arm HPC installations – i.e., the innovative Mont-Blanc project, led by Bull/Atos, the 'Isambard’ Cray XC50 going into the University of Bristol, and commitments from both Japan and France among others -- HPE is announcing that it will be supply the United States National Nuclear Security Administration (NNSA) with a 2.3 petaflops peak Arm-based system, named Astra. Read more…

By Tiffany Trader

HPE Extreme Performance Solutions

HPC and AI Convergence is Accelerating New Levels of Intelligence

Data analytics is the most valuable tool in the digital marketplace – so much so that organizations are employing high performance computing (HPC) capabilities to rapidly collect, share, and analyze endless streams of data. Read more…

IBM Accelerated Insights

Banks Boost Infrastructure to Tackle GDPR

As banks become more digital and data-driven, their IT managers are challenged with fast growing data volumes and lines-of-businesses’ (LoBs’) seemingly limitless appetite for analytics. Read more…

Challenges Face Astroinformatics as It Sorts Through the Stars

June 15, 2018

You might have seen one of those YouTube videos: they begin on Earth, slowly zooming out to the Moon, the Solar System, the Milky Way, beyond – and suddenly, you’re looking at trillions of stars. It’s a lot to take Read more…

By Oliver Peckham

Sandia to Take Delivery of World’s Largest Arm System

June 18, 2018

While the enterprise remains circumspect on prospects for Arm servers in the datacenter, the leadership HPC community is taking a bolder, brighter view of the x86 server CPU alternative. Amongst current and planned Arm HPC installations – i.e., the innovative Mont-Blanc project, led by Bull/Atos, the 'Isambard’ Cray XC50 going into the University of Bristol, and commitments from both Japan and France among others -- HPE is announcing that it will be supply the United States National Nuclear Security Administration (NNSA) with a 2.3 petaflops peak Arm-based system, named Astra. Read more…

By Tiffany Trader

The Machine Learning Hype Cycle and HPC

June 14, 2018

Like many other HPC professionals I’m following the hype cycle around machine learning/deep learning with interest. I subscribe to the view that we’re probably approaching the ‘peak of inflated expectation’ but not quite yet starting the descent into the ‘trough of disillusionment. This still raises the probability that... Read more…

By Dairsie Latimer

Xiaoxiang Zhu Receives the 2018 PRACE Ada Lovelace Award for HPC

June 13, 2018

Xiaoxiang Zhu, who works for the German Aerospace Center (DLR) and Technical University of Munich (TUM), was awarded the 2018 PRACE Ada Lovelace Award for HPC for her outstanding contributions in the field of high performance computing (HPC) in Europe. Read more…

By Elizabeth Leake

U.S Considering Launch of National Quantum Initiative

June 11, 2018

Sometime this month the U.S. House Science Committee will introduce legislation to launch a 10-year National Quantum Initiative, according to a recent report by Read more…

By John Russell

ORNL Summit Supercomputer Is Officially Here

June 8, 2018

Oak Ridge National Laboratory (ORNL) together with IBM and Nvidia celebrated the official unveiling of the Department of Energy (DOE) Summit supercomputer toda Read more…

By Tiffany Trader

Exascale USA – Continuing to Move Forward

June 6, 2018

The end of May 2018, saw several important events that continue to advance the Department of Energy’s (DOE) Exascale Computing Initiative (ECI) for the United Read more…

By Alex R. Larzelere

Exascale for the Rest of Us: Exaflops Systems Capable for Industry

June 6, 2018

Enterprise advanced scale computing – or HPC in the enterprise – is an entity unto itself, situated between (and with characteristics of) conventional enter Read more…

By Doug Black

Fracas in Frankfurt: ISC18 Cluster Competition Teams Unveiled

June 6, 2018

The Student Cluster Competition season heats up with the seventh edition of the ISC Student Cluster Competition, slated to begin on June 25th in Frankfurt, Germ Read more…

By Dan Olds

MLPerf – Will New Machine Learning Benchmark Help Propel AI Forward?

May 2, 2018

Let the AI benchmarking wars begin. Today, a diverse group from academia and industry – Google, Baidu, Intel, AMD, Harvard, and Stanford among them – releas Read more…

By John Russell

How the Cloud Is Falling Short for HPC

March 15, 2018

The last couple of years have seen cloud computing gradually build some legitimacy within the HPC world, but still the HPC industry lies far behind enterprise I Read more…

By Chris Downing

US Plans $1.8 Billion Spend on DOE Exascale Supercomputing

April 11, 2018

On Monday, the United States Department of Energy announced its intention to procure up to three exascale supercomputers at a cost of up to $1.8 billion with th Read more…

By Tiffany Trader

Deep Learning at 15 PFlops Enables Training for Extreme Weather Identification at Scale

March 19, 2018

Petaflop per second deep learning training performance on the NERSC (National Energy Research Scientific Computing Center) Cori supercomputer has given climate Read more…

By Rob Farber

Lenovo Unveils Warm Water Cooled ThinkSystem SD650 in Rampup to LRZ Install

February 22, 2018

This week Lenovo took the wraps off the ThinkSystem SD650 high-density server with third-generation direct water cooling technology developed in tandem with par Read more…

By Tiffany Trader

ORNL Summit Supercomputer Is Officially Here

June 8, 2018

Oak Ridge National Laboratory (ORNL) together with IBM and Nvidia celebrated the official unveiling of the Department of Energy (DOE) Summit supercomputer toda Read more…

By Tiffany Trader

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

HPE Wins $57 Million DoD Supercomputing Contract

February 20, 2018

Hewlett Packard Enterprise (HPE) today revealed details of its massive $57 million HPC contract with the U.S. Department of Defense (DoD). The deal calls for HP Read more…

By Tiffany Trader

Leading Solution Providers

SC17 Booth Video Tours Playlist

Altair @ SC17

Altair

AMD @ SC17

AMD

ASRock Rack @ SC17

ASRock Rack

CEJN @ SC17

CEJN

DDN Storage @ SC17

DDN Storage

Huawei @ SC17

Huawei

IBM @ SC17

IBM

IBM Power Systems @ SC17

IBM Power Systems

Intel @ SC17

Intel

Lenovo @ SC17

Lenovo

Mellanox Technologies @ SC17

Mellanox Technologies

Microsoft @ SC17

Microsoft

Penguin Computing @ SC17

Penguin Computing

Pure Storage @ SC17

Pure Storage

Supericro @ SC17

Supericro

Tyan @ SC17

Tyan

Univa @ SC17

Univa

Hennessy & Patterson: A New Golden Age for Computer Architecture

April 17, 2018

On Monday June 4, 2018, 2017 A.M. Turing Award Winners John L. Hennessy and David A. Patterson will deliver the Turing Lecture at the 45th International Sympo Read more…

By Staff

Google Chases Quantum Supremacy with 72-Qubit Processor

March 7, 2018

Google pulled ahead of the pack this week in the race toward "quantum supremacy," with the introduction of a new 72-qubit quantum processor called Bristlecone. Read more…

By Tiffany Trader

Google I/O 2018: AI Everywhere; TPU 3.0 Delivers 100+ Petaflops but Requires Liquid Cooling

May 9, 2018

All things AI dominated discussion at yesterday’s opening of Google’s I/O 2018 developers meeting covering much of Google's near-term product roadmap. The e Read more…

By John Russell

Nvidia Ups Hardware Game with 16-GPU DGX-2 Server and 18-Port NVSwitch

March 27, 2018

Nvidia unveiled a raft of new products from its annual technology conference in San Jose today, and despite not offering up a new chip architecture, there were still a few surprises in store for HPC hardware aficionados. Read more…

By Tiffany Trader

Pattern Computer – Startup Claims Breakthrough in ‘Pattern Discovery’ Technology

May 23, 2018

If it weren’t for the heavy-hitter technology team behind start-up Pattern Computer, which emerged from stealth today in a live-streamed event from San Franci Read more…

By John Russell

Part One: Deep Dive into 2018 Trends in Life Sciences HPC

March 1, 2018

Life sciences is an interesting lens through which to see HPC. It is perhaps not an obvious choice, given life sciences’ relative newness as a heavy user of H Read more…

By John Russell

Intel Pledges First Commercial Nervana Product ‘Spring Crest’ in 2019

May 24, 2018

At its AI developer conference in San Francisco yesterday, Intel embraced a holistic approach to AI and showed off a broad AI portfolio that includes Xeon processors, Movidius technologies, FPGAs and Intel’s Nervana Neural Network Processors (NNPs), based on the technology it acquired in 2016. Read more…

By Tiffany Trader

Google Charts Two-Dimensional Quantum Course

April 26, 2018

Quantum error correction, essential for achieving universal fault-tolerant quantum computation, is one of the main challenges of the quantum computing field and it’s top of mind for Google’s John Martinis. At a presentation last week at the HPC User Forum in Tucson, Martinis, one of the world's foremost experts in quantum computing, emphasized... Read more…

By Tiffany Trader

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This