Is Liquid Cooling Ready to Go Mainstream?

By Steve Campbell

February 13, 2017

 

Editor’s Note: It’s no secret that heat is a killer of electronics – performance, density, reliability, and energy efficiency all suffer. In this commentary Steve Campbell, co-founder and managing partner at OrionX, a strategy and research firm, contends liquid cooling may be closer than we think to widespread use in HPC and traditional datacenters and no longer just the purview of supercomputers. Many of the benefits are familiar. Campbell argues that improved cooling technology, expanding offerings from vendors, and potentially major energy savings are the drivers of the liquid cooling adoption trend. SC16, he says, may have represented an early tipping point. See if you agree.

Lost in the frenzy of SC16 was a substantial rise in the number of vendors showing server oriented liquid cooling technologies. Three decades ago liquid cooling was pretty much the exclusive realm of the Cray-2 and IBM mainframe class products. That’s changing. We are now seeing an emergence of x86 class server products with exotic plumbing technology ranging from Direct-to-Chip to servers and storage completely immersed in a dielectric fluid.

Most people know that liquid cooling is far more efficient than air-cooling in terms of heat transfer. It is also more economical, reducing the cost of power by as much as 40 percent depending on the installation. Being more efficient with electricity can also reduce carbon footprint and contribute positively to the goals of “greenness” in the data centers, but there are other compelling benefits as well, more on that later.

Most HPC users are familiar with the Top500 but may not be as familiar with the Green500, which ranks the Top500 supercomputers in the world by energy efficiency. The focus of performance-at-any-cost computer operations has led to the emergence of supercomputers that consume vast amounts of electrical power and produce so much heat that large cooling facilities must be constructed to ensure proper performance.

To address this trend, the Green500 list puts a premium on energy-efficient performance for sustainable supercomputing. The most recent Green500, released during SC16, has several systems in the top 10 using liquid cooling.

CoolIT CHx650

US data centers consumed about 70 billion kilowatt-hours of electricity in 2014, about two percent of the country’s total energy consumption, according to a 2014 study conducted by the US Department of Energy in collaboration with researchers from Stanford University, Northwestern University, and Carnegie Mellon University. Liquid cooling can reduce electrical usage by as much as 40 percent, which would take a huge bite out of datacenter energy consumption.

Liquid cooling can also increase server density. The heat generated from the HPC servers rises up, and in a full rack of servers those at the top will experience temperature increases and ultimately shut down. Consequently, you can’t completely populate the rack with servers all the way to the top; instead you need additional racks and extra floor space to get to processing power wanted. Liquid cooling eliminates the need for additional racks, creating higher data center server density using less floor space. This need for processing power and capacity has only been increasing in a race to the top.

Another liquid cooling benefit is higher-speed processing – CPUs and other components can run at higher speed as they are cooled more efficiently. Also, servers require no fans making them operationally silent. The more servers in a datacenter the more fans are required and noise levels increase – until it hits a painful point, sometimes literally. Liquid cooling eliminates fans and thus reduces acoustic noise levels.

Reliability can also be improved as mechanical and thermal fatigue have been reduced in liquid cooling systems as there are no moving parts, no vibrations from fans for example, and the systems are being cooled more efficiently. The elimination of hot spots and thermal stresses will also lead to improved overall reliability, performance and life.

Liquid Cooling Round-up

Following is round up of vendors demonstrating liquid cooled servers at SC16:

Aquila developed Aquarius water cooled server system offered in an Open Compute Platform (OCP) rack, in partnership with Clustered Systems utilizing their cold plate cooling technology. Aquila has also partnered with Houston based TAS Energy to co-develop an edge data center around the Aquarius platform.

Asetek, a provider of hot water, direct-to-chip liquid cooling technology showcased solutions in use worldwide at HPC users with their OEM partners such as Cray, Fujitsu, Format, and Penguin. Liquid cooling solutions for HPE, NVIDIA, Intel, and others were also on display.

Asetek’s direct-to-chip cooling technology is deployed nine installations in the November 2016 Green500 list. The highest ranked is #5 on the list; The University of Regensburg QPACE3 is a joint research project with The University of Wuppertal and Jülich Supercomputing Center. Featuring Asetek liquid cooled Fujitsu PRIMERGY servers, it is one of the first Intel Xeon Phi KNL based HPC clusters in Europe. Ranked #6 on the Green500, Oakforest-PACS is the highest performance supercomputer system in Japan and ranked #6 on the Top500. Fujitsu also deployed HPC clusters with PRIMERGY server nodes at the Joint Center for Advanced High-Performance Computing (JCAHPC) in conjunction with the University of Tokyo and Tsukuba University.

Asetek also announced liquid cooling technology is cooling eight installations in the November 2016 edition of the TOP500 list of the fastest supercomputers in the world.

CoolIT Systems is a leader in direct Contact Liquid Cooling energy efficient liquid cooling solutions for the HPC, Cloud and Enterprise markets. CoolIT’s solutions target racks of high-density servers. The technology can be deployed with any server in any rack, according to CoolIT.

CoolIT has several OEMs including:

  • Hewlett Packard Enterprise Apollo 2000 System
  • NEC Blue Marlin
  • Dell PowerEdge C6320
  • Lenovo NeXtScale product offering

CoolIT Systems have also partnered with STULZ and showcased their Chip-to-Atmosphere concept within a micro datacenter. CoolIT was recently selected by the University of Toronto to provide custom liquid cooling for its new signal processing backend, which will support Canada’s largest radio telescope, the Canadian Hydrogen Intensity Mapping Experiment (CHIME), a joint project between the National Research Council of Canada (NRC) and three major Universities (McGill, Toronto, UBC).

Ebullient has developed a two-phase cooling system for data center servers. Low-pressure fluid, 3M Novec 7000, is pumped through flexible tubing to sealed modules mounted on the processors in each server. The fluid captures heat from the processors and transports it back to a central unit, where it is either rejected outside the facility or reused elsewhere in the facility or in neighboring facilities. Ebullient’s direct-to-chip systems can cool any server, regardless of make or model.

Ebullient is an early stage company founded in 2013 based on technology developed University of Wisconsin. The company raised $2.3 million in January 2016

Green Revolution Cooling’s CarnotJet System is a liquid immersion cooling solution for data center servers. Rack-mounted servers from any OEM vendor can be installed in special racks filled with a dielectric mineral oil. On show at their SC16 booth was the Minimus server, their own design to further cost reduce the server component of the overall system.

In December Green Revolution announced a strategic partnership with Heat Transfer Solutions (HTS), an independent HVAC manufacturers’ representative in North America. As part of the partnership, HTS is making a financial investment in GRC, which will provide growth capital as the company continues to expand its presence in the data center market. In addition, a new CEO was appointed to help grow the company.

LiquidCool Solutions is a technology development firm specializing in cooling electronics by total immersion in their own proprietary dielectric fluid. LiquidCool Solutions was originally founded in 2006 as Hardcore Computing with a focus on workstations, rebranding in 2012 to LiquidCool Solutions and its focus on servers. The company has demonstrated two new liquid submerged servers based on the Clamshell design. The Submerged Cloud Server, a 2U 4-node server designed for Cloud-computing applications and Submerged GPU Server, is a 2U dual node server designed for HPC applications that can be equipped with four GPU cards or four Xeon Phi boards.

LiquidMips showcased a server-cooling concept, a single processor chip immersed in 3M Fluorinert. It’s a long way from being a commercially viable product but represents another company entering the immersive cooling market.

Inspur Systems Inc., part of Inspur Group, showed two types of cooling solutions at SC16, a phase changing cooling solution with ultra-high thermal capacity, and a direct contact liquid cooling solution which allows users to maximize performance and lower operating expenses.

Allied Control specializes in 2-phase immersion cooling solutions for HPC applications. Having built the world’s largest 40MW immersion cooled data center with 252kW per single rack resulting in 34.7kW/sqm or 3.2kW/sqft incl. white space, Allied Control offers performance centric solutions for ultra-high density HPC applications. Allied Control utilizes the 3M Novec dielectric fluid.

The BitFury Group (Bitcoin mining giant) acquired Allied Control in 2015. In January 2017 BitFury Group announced a deal Credit China Fintech Holdings to set up a joint venture that will focus on promoting the technology in China. As part of the deal, Credit China Fintech will invest $30 million in BitFury and the setting up of the joint venture that will sell BitFury’s bitcoin mining equipment.

ExaScaler Inc. is specialized in submersion liquid cooling technology. ExaScaler, and its sister company PEZY Computing, unveiled ZettaScaler-1.8, the first Super Computer with a performance density of 1.5 PetaFLOPS/m. The ZettaScaler-1.8 is an advanced prototype of the ZettaScaler-2.0 due to be released in 2017 with a performance density three times higher than the ZettaScaler-1.8. ExaScaler immersion liquid cooling using 3M Fluorinert cools ZettaScaler-1.8 Super Computer.

Fujitsu demonstrated a new form of data center, which included cloud-based servers, storage, network switch and center facilities, by combining the liquid immersion cooling technology for supercomputers developed by ExaScaler Inc. with Fujitsu’s know-how on general-purpose computers. Fujitsu is able to capitalize on three decades of liquid cooling expertise with mainframes, to supercomputers to Intel x86.

This new style of data center uses liquid immersion cooling technology that completely immerses IT systems such as servers, storage, and networking equipment in liquid coolant in order to cool the devices.

The liquid immersion cooling technology uses 3M’s Fluorinert, an inert fluid that provides high heat-transfer efficiency and insulation as a coolant. IT devices, including servers and storage, are totally submerged in a dedicated reservoir tank filled with liquid Fluorinert, and the heat generated from the devices is processed by circulating the cooled liquid through the devices. This improves the efficiency of the entire cooling system, thereby significantly reducing power consumption. A further benefit of the immersed cooling is that it provides protection from harsh environmental elements, such as corrosion, contamination, and pollution.

3M’s HPC offers solutions using 3M Engineered Fluids such as Novec or Fluorinert. Perhaps the winner at SC16 for immersed cooling is 3M as most of the vendors mentioned here use 3M Engineering Fluids. 3M fluids also featured in some of the networking products at the event. Fully immersed systems can improve energy efficiency, allow for significantly greater computing density, and help minimize thermal limitations during design.

Huawei announced a next-generation FusionServer X6000 HPC server that uses a liquid cooling solution featuring a skive fin micro-channel heat sink for CPU heat dissipation and processing technology where water flows through memory modules. This modular board design and 50ºC warm water cooling offers high energy-efficiency and reduces total cost of ownership (TCO).

Other vendors

HPE Apollo

HPE and Dell both introduced liquid cooling server products in 2016. Though they do not have the lineage of Fujitsu they nevertheless recognize the values liquid cooling delivers to the datacenter.

HPE’s entrance is the Apollo family of high-density servers. These rack-based solutions include compute, storage, networking, power and cooling. Target users are high-performance computing workloads and big data analytics. The top of the server lineup is the Apollo 8000 uses a warm water-cooling system whereas other members of the Apollo family of servers integrate the CoolIT Systems Closed-Loop DCLC (Direct Contact Liquid Cooling).

Dell, like HPE, does not have the decades of liquid cooling expertise of Fujitsu. Dell took the covers of the Dell Triton water cooling system in mid 2016. Dell’s Extreme Scale Infrastructure team built Triton as a proof of concept for eBay, leveraging Dell’s rack-scale infrastructure. The liquid-cooled cold plates directly contact the CPUs and incorporates liquid to air heat exchanges to cool the airborne heat generated by the large number of densely packed processor nodes.

Can we add liquid cooling to existing servers?

Good question, and the answer is no you really cannot. Adopting liquid cooling only makes sense on new server deployments. That is not to say it is impossible, but there are lots of modifications needed to make water cooling, like direct-to-chip, or fully immersed work, big maybe and not really recommended. An existing server has cooling fans that need to be disabled and CPU cooling towers removed and so on. You also need to add plumbing to your existing rack, which can be a pain.

There is no question that a prospective user needs to consider the impact and requirements on existing datacenter infrastructure, the physical bricks, mortar, plumbing, etc. For users considering water-cooled solutions you will need to plumb water to the server rack. If you are in a new datacenter that is one level of effort but if your datacenter is a large closet in an older building, like 43 percent of North American datacenter/server rooms, it may be a lot more difficult and expensive.

If you are considering a fully immersed solution, such as Fujitsu, no plumbing is required; all you need to do is hook up to chiller. It may be easier and less expensive than water-cooling. As a completely sealed unit it is conceivable that liquid immersed cooling solutions can be deployed almost anywhere, no datacenter required.

Most vendors covered in this market are small emerging technology companies. Asetek’s data center revenue was $1.8 million in the third quarter and $3.6 million in the first nine months of 2016, compared with $0.5 million and $1.0 million in the third quarter and first nine months of 2015, respectively. Asetek is forecasting significant data center revenue growth in 2016 from $1.9M in 2015.

CoolIT reported 2014 revenue of $27 million for all product categories. It is worth noting that Asetek and CoolIT data center revenues are less than 10% of total company revenue. The remaining 90% is workstation and PC liquid cooling solutions. Ebullient, Liquid MIPS, LiquidCooling, Green Revolution and Aquila have very few customers and probably below $10M annual revenues.

The obvious question is since most of the vendors are small and very early stage – is there truly a market for liquid cooled servers? Industry analysts believe there is and forecast the market to grow from about $110 million in 2015 to almost $960 million in 2020, an additional $850 million of incremental revenue in just five years.

With healthy future growth prospects, we’ve started to see larger players enter this market such as Fujitsu. In addition, the HPC system vendors are all OEMing liquid cooling technology solutions to solve the big system cooling issues in the datacenters. With the huge increase in data being generated, artificial intelligence and other applications need to mine this data. Consequently, more and more server power is required and new innovative cooling solutions needed making liquid cooling a practical and feasible solution.

As a side note, more and more government RFPs are asking for liquid cooling solutions. Solutions such as the one from Fujitsu can make the crossover from HPC to commercial datacenter a reality.

Could 2017 the breakout year for liquid cooling, move from innovator to early adopter?

The Supercomputing Conference is frequently a window into the future. At SC16, there were over a dozen companies demonstrating server liquid cooling solutions, with technologies ranging from Direct-to-Chip to Liquid Immersive Cooling where servers and storage are fully immersed in dielectric fluid.

Today the majority of providers are early stage or startup companies with a notable exception. Fujitsu, a global IT powerhouse brought over thirty years of liquid cooling experience and demonstrated an immersive cooling solution that had Intel-based servers, storage and network switches fully immersed in Fluorinert.

We will see cooling technology move from the confines of high-end supercomputers to a nice niche in the enterprise datacenter for such workloads as big data analytics, AI and high frequency trading.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Live and in Color, Meet the European Student Cluster Teams

November 21, 2017

The SC17 Student Cluster Competition welcomed two teams from Europe, the German team of FAU/TUC and Team Poland, the pride of Warsaw. Let's get to know them better through the miracle of video..... Team FAU/TUC is a c Read more…

By Dan Olds

SC17 Student Cluster Kick Off – Guts, Glory, Grep

November 21, 2017

The SC17 Student Cluster Competition started with a well-orchestrated kick-off emceed by Stephen Harrell, the competition chair. It began with a welcome from SC17 chair Bernd Mohr, where he lauded the competition for Read more…

By Dan Olds

Activist Investor Starboard Buys 10.7% Stake in Mellanox; Sale Possible?

November 20, 2017

Starboard Value has reportedly taken a 10.7 percent stake in interconnect specialist Mellanox Technologies, and according to the Wall Street Journal, has urged the company “to improve its margins and stock and explore Read more…

By John Russell

HPE Extreme Performance Solutions

Harness Scalable Petabyte Storage with HPE Apollo 4510 and HPE StoreEver

As a growing number of connected devices challenges IT departments to rapidly collect, manage, and store troves of data, organizations must adopt a new generation of IT to help them operate quickly and intelligently. Read more…

Installation of Sierra Supercomputer Steams Along at LLNL

November 20, 2017

Sierra, the 125 petaflops (peak) machine based on IBM’s Power9 chip being built at Lawrence Livermore National Laboratory, sometimes takes a back seat to Summit, the ~200 petaflops system being built at Oak Ridge Natio Read more…

By John Russell

Live and in Color, Meet the European Student Cluster Teams

November 21, 2017

The SC17 Student Cluster Competition welcomed two teams from Europe, the German team of FAU/TUC and Team Poland, the pride of Warsaw. Let's get to know them bet Read more…

By Dan Olds

SC17 Student Cluster Kick Off – Guts, Glory, Grep

November 21, 2017

The SC17 Student Cluster Competition started with a well-orchestrated kick-off emceed by Stephen Harrell, the competition chair. It began with a welcome from Read more…

By Dan Olds

SC Bids Farewell to Denver, Heads to Dallas for 30th

November 17, 2017

After a jam-packed four-day expo and intensive six-day technical program, SC17 has wrapped up another successful event that brought together nearly 13,000 visit Read more…

By Tiffany Trader

SC17 Keynote – HPC Powers SKA Efforts to Peer Deep into the Cosmos

November 17, 2017

This week’s SC17 keynote – Life, the Universe and Computing: The Story of the SKA Telescope – was a powerful pitch for the potential of Big Science projects that also showcased the foundational role of high performance computing in modern science. It was also visually stunning. Read more…

By John Russell

How Cities Use HPC at the Edge to Get Smarter

November 17, 2017

Cities are sensoring up, collecting vast troves of data that they’re running through predictive models and using the insights to solve problems that, in some Read more…

By Doug Black

Student Cluster LINPACK Record Shattered! More LINs Packed Than Ever before!

November 16, 2017

Nanyang Technological University, the pride of Singapore, utterly destroyed the Student Cluster Competition LINPACK record by posting a score of 51.77 TFlop/s a Read more…

By Dan Olds

Hyperion Market Update: ‘Decent’ Growth Led by HPE; AI Transparency a Risk Issue

November 15, 2017

The HPC market update from Hyperion Research (formerly IDC) at the annual SC conference is a business and social “must,” and this year’s presentation at S Read more…

By Doug Black

Nvidia Focuses Its Cloud Containers on HPC Applications

November 14, 2017

Having migrated its top-of-the-line datacenter GPU to the largest cloud vendors, Nvidia is touting its Volta architecture for a range of scientific computing ta Read more…

By George Leopold

US Coalesces Plans for First Exascale Supercomputer: Aurora in 2021

September 27, 2017

At the Advanced Scientific Computing Advisory Committee (ASCAC) meeting, in Arlington, Va., yesterday (Sept. 26), it was revealed that the "Aurora" supercompute Read more…

By Tiffany Trader

NERSC Scales Scientific Deep Learning to 15 Petaflops

August 28, 2017

A collaborative effort between Intel, NERSC and Stanford has delivered the first 15-petaflops deep learning software running on HPC platforms and is, according Read more…

By Rob Farber

Oracle Layoffs Reportedly Hit SPARC and Solaris Hard

September 7, 2017

Oracle’s latest layoffs have many wondering if this is the end of the line for the SPARC processor and Solaris OS development. As reported by multiple sources Read more…

By John Russell

AMD Showcases Growing Portfolio of EPYC and Radeon-based Systems at SC17

November 13, 2017

AMD’s charge back into HPC and the datacenter is on full display at SC17. Having launched the EPYC processor line in June along with its MI25 GPU the focus he Read more…

By John Russell

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Google Releases Deeplearn.js to Further Democratize Machine Learning

August 17, 2017

Spreading the use of machine learning tools is one of the goals of Google’s PAIR (People + AI Research) initiative, which was introduced in early July. Last w Read more…

By John Russell

GlobalFoundries Puts Wind in AMD’s Sails with 12nm FinFET

September 24, 2017

From its annual tech conference last week (Sept. 20), where GlobalFoundries welcomed more than 600 semiconductor professionals (reaching the Santa Clara venue Read more…

By Tiffany Trader

Amazon Debuts New AMD-based GPU Instances for Graphics Acceleration

September 12, 2017

Last week Amazon Web Services (AWS) streaming service, AppStream 2.0, introduced a new GPU instance called Graphics Design intended to accelerate graphics. The Read more…

By John Russell

Leading Solution Providers

EU Funds 20 Million Euro ARM+FPGA Exascale Project

September 7, 2017

At the Barcelona Supercomputer Centre on Wednesday (Sept. 6), 16 partners gathered to launch the EuroEXA project, which invests €20 million over three-and-a-half years into exascale-focused research and development. Led by the Horizon 2020 program, EuroEXA picks up the banner of a triad of partner projects — ExaNeSt, EcoScale and ExaNoDe — building on their work... Read more…

By Tiffany Trader

Delays, Smoke, Records & Markets – A Candid Conversation with Cray CEO Peter Ungaro

October 5, 2017

Earlier this month, Tom Tabor, publisher of HPCwire and I had a very personal conversation with Cray CEO Peter Ungaro. Cray has been on something of a Cinderell Read more…

By Tiffany Trader & Tom Tabor

Cray Moves to Acquire the Seagate ClusterStor Line

July 28, 2017

This week Cray announced that it is picking up Seagate's ClusterStor HPC storage array business for an undisclosed sum. "In short we're effectively transitioning the bulk of the ClusterStor product line to Cray," said CEO Peter Ungaro. Read more…

By Tiffany Trader

Reinders: “AVX-512 May Be a Hidden Gem” in Intel Xeon Scalable Processors

June 29, 2017

Imagine if we could use vector processing on something other than just floating point problems.  Today, GPUs and CPUs work tirelessly to accelerate algorithms Read more…

By James Reinders

Intel Launches Software Tools to Ease FPGA Programming

September 5, 2017

Field Programmable Gate Arrays (FPGAs) have a reputation for being difficult to program, requiring expertise in specialty languages, like Verilog or VHDL. Easin Read more…

By Tiffany Trader

HPC Chips – A Veritable Smorgasbord?

October 10, 2017

For the first time since AMD's ill-fated launch of Bulldozer the answer to the question, 'Which CPU will be in my next HPC system?' doesn't have to be 'Whichever variety of Intel Xeon E5 they are selling when we procure'. Read more…

By Dairsie Latimer

Flipping the Flops and Reading the Top500 Tea Leaves

November 13, 2017

The 50th edition of the Top500 list, the biannual publication of the world’s fastest supercomputers based on public Linpack benchmarking results, was released Read more…

By Tiffany Trader

IBM Advances Web-based Quantum Programming

September 5, 2017

IBM Research is pairing its Jupyter-based Data Science Experience notebook environment with its cloud-based quantum computer, IBM Q, in hopes of encouraging a new class of entrepreneurial user to solve intractable problems that even exceed the capabilities of the best AI systems. Read more…

By Alex Woodie

Share This