Liquid Cooling Makes a Splash in Enterprise Data Centers

August 12, 2019

Direct liquid cooling is now gaining momentum in mainstream data centers — for many good reasons.

As servers get faster, they generate more heat, and that increases the need for innovative approaches to cooling that go beyond the limitations of conventional air-cooled approaches. One of these innovative approaches that is gaining a lot of traction is direct liquid cooling.

Direct liquid cooling, or DLC, transfers heat away from processors and other heat-generating system components via contact with a liquid-cooled heat sink rather than using just air in the heat-exchange process. In one approach to DLC, used by the company CoolIT, a metal plate, or a cold plate, functions as a heat sink. This plate is placed in direct contact with the processor casing surface. Liquid flows through sealed micro-channels in the plate to capture heat from the processor, and then flows on to a heat exchanger, which dissipates the heat. The cooled liquid coming out of the heat exchanger travels back through the system, and the cycle repeats itself.

While it has been around for decades for various use cases — including supercomputing systems and desktop systems with accelerators — direct liquid cooling is gaining traction in enterprise data centers and service provider environments. The title of a paper by 451 Research pretty much says it all: “The Tide is Turning Toward Liquid Cooling in Datacenters.”[1]

Market research validates this view of the growing momentum for liquid cooling. A report by the research firm Global Market Insights projects that the data center liquid cooling market will jump to $2.5 billion by 2025, up from $750 million in 2018.[2]

A thirst for throughput

So why is direct liquid cooling gaining traction? One reason is an unquenchable thirst for higher throughput for artificial intelligence and high performance computing. To meet to this demand, new generations of technology bring faster Intel® processors and more dense racks, and that equates to more heat with the need for more efficient approaches to cooling.

Over the years, as Moore’s Law played out and the capacity and performance of processors grew, system builders continued to keep things cool with air-based approaches. But now we have reached the point where processor capacities and wattages are starting to bump up against the limits of conventional approaches to cooling. The more we push for higher performance in systems, the more wattage we require in processors, and that equates to more heat. Liquid cooling is one of the ways to deal with that heat in an efficient and cost-effective manner.

Let’s talk watts. Air cooling has worked for systems with CPUs that top out around 150 watts, which is what we have had for the past decade or so. But today, we are seeing more systems with CPUs that run at 200W and above and accelerators that run at 300W, with plans for even higher-wattage processors. These higher-wattage processors will be increasingly important for many computationally intensive artificial intelligence workloads that use hardware accelerators, including machine learning, deep learning and applications like high-frequency training and fraud detection.

Collectively, these trends are causing people to take a closer look at liquid cooling, which is a more efficient way to cool processors. It’s also a more efficient way to cool high-density storage and systems deployed at the network edge, according to 451 Research.

“DLC will largely take off because the existing tech­nology — air cooling — will hit its limitations and inflict pain on major buyers of infrastructure in the form of lost performance or higher costs (or both),” the firm says in its report on the tide turning to DLC. What’s more, the data centers that we build today will likely still be in service in the late 2020s and early 2030s, so they have to be built with future cooling needs in mind. “It is for these reasons that 451 Research views DLC as a must-have in the datacenter operator’s toolset if it wants to keep pace with infrastructure requirements.”1

Liquid cooling in a new supercomputer

Among the organizations that have opted for liquid cooling is the Texas Advanced Computing Center (TACC), with new supercomputer Frontera. The primary system, supplied by Dell EMC, will have more than 8,000 dual-socket Intel® Xeon® nodes with 205W processors. Frontera will also include a small accelerated subsystem, to support molecular dynamics and machine learning applications.[3]

Frontera, which debuted at No. 5 on the latest TOP500 list,[4] leverages DLC-cooled Dell EMC PowerEdge C6420 servers and Dell EMC Isilon unstructured data storage in combination with 2nd-generation Intel® Xeon® Scalable Platinum processors, Intel® Optane™ DC Persistent Memory and a high-performance Mellanox HDR 200Gb/s InfiniBand interconnect.

For the system’s 448,448 cores, the racks leverage DLC technology from CoolIT, a Dell EMC partner. CoolIT’s Direct Contact Liquid Cooling technology uses the exceptional thermal conductivity of liquid to provide dense, concentrated cooling to targeted surface areas. The technology uses warm or room-temperature water to cool CPUs, eliminating the need for chilled water and reducing the cost of cooling by as much as 56 percent over traditional cooling methods.[5]

Dell EMC now offers DCLC technology from CoolIT in the PowerEdge C6420 server, a density-optimized workhorse for artificial intelligence and high performance computing workloads. Liquid cooling allows the server to support higher wattage processors for increased performance and energy efficiency.

Key takeaways

Direct liquid cooling is coming to a data center near you as one of the keys to enabling the use of higher-wattage processors and more powerful computing systems.  As 451 Research notes, “DLC is quickly becoming a relevant consideration for enterprises and service providers, moving beyond the classic supercomputing use case.”1

To learn more

For a closer look at liquid cooling in HPC servers, see the Dell EMC-CoolIT solution brief and watch the Dell EMC video featuring CoolIT Systems CEO/CTO Geoff Lyon.


[1] 451 Research, “The Tide is Turning Toward Liquid Cooling in Datacenters,” July 2018.

[2] Global Market Insights, Inc., “Data Center Liquid Cooling Market worth over $2.5bn by 2025,” February 04, 2019.

[3] HPCwire, “TACC’s ‘Frontera’ Supercomputer Expands Horizon for Extreme-Scale Science,” August 29, 2018.

[4] TOP500, “Frontera – Dell C6420, Xeon Platinum 8280 28C 2.7GHz, Mellanox InfiniBand HDR,” June 2019.

[5] Dell EMC solutions brief, “Improve Power Efficiency with Dell EMC PowerEdge and CoolIT Systems Rack DCLC,” 2017.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

IBM Unveils Latest Achievements in AI Hardware

December 13, 2019

“The increased capabilities of contemporary AI models provide unprecedented recognition accuracy, but often at the expense of larger computational and energetic effort,” IBM Research wrote in a blog post. “Therefor Read more…

By Oliver Peckham

Focused on ‘Silicon TAM,’ Intel Puts Gary Patton, Former GlobalFoundries CTO, in Charge of Design Enablement

December 12, 2019

Change within Intel’s upper management – and to its company mission – has continued as a published report has disclosed that chip technology heavyweight Gary Patton, GlobalFoundries’ CTO and R&D SVP as well a Read more…

By Doug Black

Quantum Bits: Rigetti Debuts New Gates, D-Wave Cuts NEC Deal, AWS Jumps into the Quantum Pool

December 12, 2019

There’s been flurry of significant news in the quantum computing world. Yesterday, Rigetti introduced a new family of gates that reduces circuit depth required on some problems and D-Wave struck a deal with NEC to coll Read more…

By John Russell

How Formula 1 Used Cloud HPC to Build the Next Generation of Racing

December 12, 2019

Formula 1, Rob Smedley explained, is maybe the biggest racing spectacle in the world, with five hundred million fans tuning in for every race. Smedley, a chief engineer with Formula 1’s performance engineering and anal Read more…

By Oliver Peckham

RPI Powers Up ‘AiMOS’ AI Supercomputer

December 11, 2019

Designed to push the frontiers of computing chip and systems performance optimized for AI workloads, an 8 petaflops (Linpack) IBM Power9-based supercomputer has been unveiled in upstate New York that will be used by IBM Read more…

By Doug Black

AWS Solution Channel

Making High Performance Computing Affordable and Accessible for Small and Medium Businesses with HPC on AWS

High performance computing (HPC) brings a powerful set of tools to a broad range of industries, helping to drive innovation and boost revenue in finance, genomics, oil and gas extraction, and other fields. Read more…

IBM Accelerated Insights

GPU Scheduling and Resource Accounting: The Key to an Efficient AI Data Center

[Connect with LSF users and learn new skills in the IBM Spectrum LSF User Community!]

GPUs are the new CPUs

GPUs have become a staple technology in modern HPC and AI data centers. Read more…

At SC19: Developing a Digital Twin

December 11, 2019

In the not too distant future, we can expect to see our skies filled with unmanned aerial vehicles (UAVs) delivering packages, maybe even people, from location to location. In such a world, there will also be a digital twin for each UAV in the fleet: a virtual model that will follow the UAV through its existence, evolving with time. Read more…

By Aaron Dubrow

Focused on ‘Silicon TAM,’ Intel Puts Gary Patton, Former GlobalFoundries CTO, in Charge of Design Enablement

December 12, 2019

Change within Intel’s upper management – and to its company mission – has continued as a published report has disclosed that chip technology heavyweight G Read more…

By Doug Black

Quantum Bits: Rigetti Debuts New Gates, D-Wave Cuts NEC Deal, AWS Jumps into the Quantum Pool

December 12, 2019

There’s been flurry of significant news in the quantum computing world. Yesterday, Rigetti introduced a new family of gates that reduces circuit depth require Read more…

By John Russell

RPI Powers Up ‘AiMOS’ AI Supercomputer

December 11, 2019

Designed to push the frontiers of computing chip and systems performance optimized for AI workloads, an 8 petaflops (Linpack) IBM Power9-based supercomputer has Read more…

By Doug Black

At SC19: Developing a Digital Twin

December 11, 2019

In the not too distant future, we can expect to see our skies filled with unmanned aerial vehicles (UAVs) delivering packages, maybe even people, from location to location. In such a world, there will also be a digital twin for each UAV in the fleet: a virtual model that will follow the UAV through its existence, evolving with time. Read more…

By Aaron Dubrow

Intel’s Jim Clarke on its New Cryo-controller and why Intel isn’t Late to the Quantum Party

December 9, 2019

Intel today introduced the ‘first-of-its-kind’ cryo-controller chip for quantum computing and previewed a cryo-prober tool for characterizing quantum proces Read more…

By John Russell

On the Spack Track @SC19

December 5, 2019

At the annual supercomputing conference, SC19 in Denver, Colorado, there were Spack events each day of the conference. As a reflection of its grassroots heritage, nine sessions were planned by more than a dozen thought leaders from seven organizations, including three U.S. national Department of Energy (DOE) laboratories and Sylabs... Read more…

By Elizabeth Leake

Intel’s New Hyderabad Design Center Targets Exascale Era Technologies

December 3, 2019

Intel's Raja Koduri was in India this week to help launch a new 300,000 square foot design and engineering center in Hyderabad, which will focus on advanced com Read more…

By Tiffany Trader

AWS Debuts 7nm 2nd-Gen Graviton Arm Processor

December 3, 2019

The “x86 Big Bang,” in which market dominance of the venerable Intel CPU has exploded into fragments of processor options suited to varying workloads, has n Read more…

By Doug Black

Using AI to Solve One of the Most Prevailing Problems in CFD

October 17, 2019

How can artificial intelligence (AI) and high-performance computing (HPC) solve mesh generation, one of the most commonly referenced problems in computational engineering? A new study has set out to answer this question and create an industry-first AI-mesh application... Read more…

By James Sharpe

D-Wave’s Path to 5000 Qubits; Google’s Quantum Supremacy Claim

September 24, 2019

On the heels of IBM’s quantum news last week come two more quantum items. D-Wave Systems today announced the name of its forthcoming 5000-qubit system, Advantage (yes the name choice isn’t serendipity), at its user conference being held this week in Newport, RI. Read more…

By John Russell

DARPA Looks to Propel Parallelism

September 4, 2019

As Moore’s law runs out of steam, new programming approaches are being pursued with the goal of greater hardware performance with less coding. The Defense Advanced Projects Research Agency is launching a new programming effort aimed at leveraging the benefits of massive distributed parallelism with less sweat. Read more…

By George Leopold

Ayar Labs to Demo Photonics Chiplet in FPGA Package at Hot Chips

August 19, 2019

Silicon startup Ayar Labs continues to gain momentum with its DARPA-backed optical chiplet technology that puts advanced electronics and optics on the same chip Read more…

By Tiffany Trader

SC19: IBM Changes Its HPC-AI Game Plan

November 25, 2019

It’s probably fair to say IBM is known for big bets. Summit supercomputer – a big win. Red Hat acquisition – looking like a big win. OpenPOWER and Power processors – jury’s out? At SC19, long-time IBMer Dave Turek sketched out a different kind of bet for Big Blue – a small ball strategy, if you’ll forgive the baseball analogy... Read more…

By John Russell

Cray, Fujitsu Both Bringing Fujitsu A64FX-based Supercomputers to Market in 2020

November 12, 2019

The number of top-tier HPC systems makers has shrunk due to a steady march of M&A activity, but there is increased diversity and choice of processing compon Read more…

By Tiffany Trader

Crystal Ball Gazing: IBM’s Vision for the Future of Computing

October 14, 2019

Dario Gil, IBM’s relatively new director of research, painted a intriguing portrait of the future of computing along with a rough idea of how IBM thinks we’ Read more…

By John Russell

Kubernetes, Containers and HPC

September 19, 2019

Software containers and Kubernetes are important tools for building, deploying, running and managing modern enterprise applications at scale and delivering enterprise software faster and more reliably to the end user — while using resources more efficiently and reducing costs. Read more…

By Daniel Gruber, Burak Yenier and Wolfgang Gentzsch, UberCloud

Leading Solution Providers

SC 2019 Virtual Booth Video Tour

AMD
AMD
CEJN
CJEN
DDN
DDN
MELLANOX
MELLANOX
ONE STOP SYSTEMS
ONE STOP SYSTEMS
PANASAS
PANASAS
SIX NINES IT
SIX NINES IT
VERNE GLOBAL
VERNE GLOBAL
WEKAIO
WEKAIO

Intel Debuts New GPU – Ponte Vecchio – and Outlines Aspirations for oneAPI

November 17, 2019

Intel today revealed a few more details about its forthcoming Xe line of GPUs – the top SKU is named Ponte Vecchio and will be used in Aurora, the first plann Read more…

By John Russell

Dell Ramps Up HPC Testing of AMD Rome Processors

October 21, 2019

Dell Technologies is wading deeper into the AMD-based systems market with a growing evaluation program for the latest Epyc (Rome) microprocessors from AMD. In a Read more…

By John Russell

SC19: Welcome to Denver

November 17, 2019

A significant swath of the HPC community has come to Denver for SC19, which began today (Sunday) with a rich technical program. As is customary, the ribbon cutt Read more…

By Tiffany Trader

When Dense Matrix Representations Beat Sparse

September 9, 2019

In our world filled with unintended consequences, it turns out that saving memory space to help deal with GPU limitations, knowing it introduces performance pen Read more…

By James Reinders

Cerebras to Supply DOE with Wafer-Scale AI Supercomputing Technology

September 17, 2019

Cerebras Systems, which debuted its wafer-scale AI silicon at Hot Chips last month, has entered into a multi-year partnership with Argonne National Laboratory and Lawrence Livermore National Laboratory as part of a larger collaboration with the U.S. Department of Energy... Read more…

By Tiffany Trader

With the Help of HPC, Astronomers Prepare to Deflect a Real Asteroid

September 26, 2019

For years, NASA has been running simulations of asteroid impacts to understand the risks (and likelihoods) of asteroids colliding with Earth. Now, NASA and the European Space Agency (ESA) are preparing for the next, crucial step in planetary defense against asteroid impacts: physically deflecting a real asteroid. Read more…

By Oliver Peckham

Jensen Huang’s SC19 – Fast Cars, a Strong Arm, and Aiming for the Cloud(s)

November 20, 2019

We’ve come to expect Nvidia CEO Jensen Huang’s annual SC keynote to contain stunning graphics and lively bravado (with plenty of examples) in support of GPU Read more…

By John Russell

IBM Opens Quantum Computing Center; Announces 53-Qubit Machine

September 19, 2019

Gauging progress in quantum computing is a tricky thing. IBM yesterday announced the opening of the IBM Quantum Computing Center in New York, with five 20-qubit Read more…

By John Russell

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This