Sept. 19 — The roar can be deafening. Cooling fans and power supplies whoosh and whine from rows and rows of supercomputers at the main data center of the Texas Advanced Computing Center in Austin. The power bill at TACC can reach over a million dollars a year to keep the machines humming. But there’s a stranger in town that might change how data centers power their systems.
A new kind of advanced computing system called Hikari (Japanese for the word “light”) came online at TACC late August, 2016. What’s new is that Hikari runs on solar power and high voltage direct current, or HVDC.
Hikari is a microgrid that supports a supercomputer, a first for the U.S. By day solar panels that shade a TACC parking lot provide nearly all of Hikari’s power, up to 208 kilowatts. At night it switches back to conventional AC power from the utility grid.
“The Hikari project is a Japan and Texas collaboration project, and it aims to demonstrate the potential of the HVDC system,” said Toshihiro Hayashi, assistant manager in the engineering divisions of NTT FACILITIES, INC., Japan.
Engineers of the Hikari HVDC power feeding system predict it will save 15 percent compared to conventional systems. “The 380 volt design reduces the number of power conversions when compared to AC voltage systems,” said James Stark, director of Engineering and Construction at the Electronic Environments Corporation (EEC), a Division of NTT FACILITIES, INC..
“What’s interesting about that,” Stark added, “is the computers themselves – the supercomputer, the blade servers, cooling units, and lighting – are really all designed to run on DC voltage. By supplying 380 volts DC to Hikari instead of having an AC supply with conversion steps, it just makes a lot more sense. That’s really the largest technical innovation.”
Data centers in the U.S. consumed an estimated 70 billion kilowatt hours in 2014, which represents about 1.8 percent of total U.S. electricity consumption. That’s according to a June 2016 Department of Energy report by the Lawrence Berkeley National Laboratory.
High Voltage Direct Current also allows for ease in connection to renewable energy, solar for Hikari but potentially other sources like wind and hydrogen fuel cells. “That’s really one of our main focuses, trying to make data centers more sustainable so that we’re reducing overall power consumption within an industry that has traditionally consumed a lot of power,” James Stark said.
With sustainability comes reliability. This held true for another NTT FACILITIES, INC. project called the Sendai Microgrid on the fateful day of the Tohoku earthquake, March 11, 2011. The Sendai Microgrid was equipped with gas engines, a fuel cell, and a photovoltaic array. It continued to supply power and heat to facilities at Tohoku Fukushi University for hospital patients and elderly nursing home residents despite days of blackouts from the catastrophic damage to the district’s energy supply system caused by the earthquake.
“This microgrid power supply system activated very well after the earthquake,” Hayashi said. “It showed that it’s a very highly reliable energy system.”
Another NTT FACILITIES project leading to Hikari is the Tsukuba Research and Development Center, a HVDC microgrid that generates 70 kilowatts for a modular data center. “We have various experiences in Japan of having this HVDC system,” Hayashi said.
These projects, including Hikari, are supported by NEDO, the New Energy and Industrial Technology Development Organization, a public agency of Japan. The Hikari project partners are NEDO, NTT FACILITIES, INC., and the University of Texas at Austin through the Texas Advanced Computing Center.
The collaboration started with a visit to TACC by engineers at NTT FACILITIES, INC. in early 2014, fresh off the heels of basic studies they’d done for installing HVDC at U.S. sites. TACC also shares a strong interest in developing new technologies, including energy savings.
“We’re very interested in UT Austin’s motto,” said Hayashi, “which is ‘what starts here changes the world.’ We very much agree with this motto.”
Hayashi’s team worked with TACC to develop feasibility studies of the Hikari HVDC project from December 2014 to May 2015. This effort led to a Memorandum of Understanding between the State of Texas and NEDO in August of 2015.
NTT FACILITIES, INC. worked with EEC to build out Hikari, which completed the system installation late August 2016. “If there wasn’t such partnership, we wouldn’t have launched this project. I would like to express my gratitude to NEDO for establishing the partnership,” Hayashi said.
The Hikari supercomputer cluster consists of 432 Hewlett Packard Enterprise (HPE) Apollo 8000 XL730f servers coupled with HPE DL380 and DL360 nodes that are interconnected with a first-of-its-kind Mellanox End-to-End EDR InfinBand at 100 gigabytes per second. Over 10,000 cores from “Haswell” Xeon processors will deliver more than 400 teraflops.