Eurotech Hive Takes The Sting Out Of Density

By Timothy Prickett Morgan

November 21, 2014

Back at the International Supercomputing Conference in June, supercomputer maker Eurotech dropped some hints about its future water-cooled Aurora systems that would employ a mix of ARM processors and Nvidia Tesla GPU accelerators in a dense form. At the SC14 conference this week, these machines have now been officially launched as the Aurora Hive systems, and it turns out that the systems will also allow customers to build massively parallel machines based on Intel Xeon processors and Xeon Phi coprocessors.

The Hive systems use a modular enclosure that that is based on a cubic shape rather than a hexagonal one, but the concept of densely stacking compute elements while isolating them from each other, as a beehive does, holds true. The system crams up to 128 nodes (which are called bricks) into a single rack – 64 nodes in the front and another 64 nodes in the back, which is something you can do when you use water cooling on the components of the nodes because you do not have to worry about airflow from cold to hot aisles through each rack.eurotech-aurora-hive-cross-section

The Hive system makes use of a second generation of direct hot water cooling from the Aurora line, which Fabio Gallo, Eurotech HPC business unit managing director, tells HPCwire can cool a system with 50 degree Celsius (122 degrees Fahrenheit) inlet water temperature. The new water cooling is lighter and more compact, allowing for more compute and cooling to be crammed into the same space. The water distribution system is built right into the Aurora Hive rack, and there are dripless connectors for inlet cold (relatively speaking) and outlet hot water coming off each node. Being able to take the heat away quickly and efficiently is vital because a fully configured Hive rack draws 166 kilowatts of juice.

“You can free cool this machine nearly anywhere on earth,” says Gallo. By Eurotech’s math, customers using the Aurora Hive should be able to attain a power usage effectiveness of 1.05, which is about as good as the hyperscale datacenter operators are getting. (PUE, as this metric is abbreviated, is the ratio of the power consumed by a datacenter divided by the power consumed by the compute, storage, and network components of the datacenter. Getting as close as possible to 1 is the goal.)

eurotech-hive-block-exposedThe Hive nodes are 3U high, and you can put them into a rack four across and sixteen high. (Each node is 130 mm high by 105 mm deep by 325 mm deep.) Each node has a system board that includes risers for a compute module and five coprocessor modules; this system board also includes a PCI-Express 3.0 switch from PLX Technology (now part of Avago Technologies) that links the compute and coprocessor elements to each other. The PCI-Express switch also has hooks out to network adapters, in this case a two-port FDR InfiniBand adapter from Mellanox Technologies. All of the PCI-Express slots have the full bandwidth of an x16 slot, which means Nvidia Tesla GPU and Intel Xeon Phi coprocessors can find a place.

Eurotech’s first Hive system will have a CPU compute element that is based on Intel’s “Haswell” Xeon E3-1200 v3 processors. This family of chips has four cores and clock speeds that range from 3.1 GHz to 3.7 GHz in standard versions. The Intel E3-1200 v3 compute node has 32 GB of memory welded onto it for low clearance and also has a 256 GB half-height 1.8-inch solid state disk drive. You can use any E3-1200 v3 chip that has a thermal design point of 84 watts or lower.

The compute brick allows for up to four coprocessors to be fitted with cold plates for sucking the heat off their components and linked to each one of the cores over the PCI-Express switch and into the PCI-Express controllers on the E3-1200 processors. Gallo tells HPCwire that it will ship the Xeon E3-1200 plus Xeon Phi configuration in a few weeks to initial customers, and that a few months after that the combination of the Xeon E3 processor and Nvidia’s Tesla K40 coprocessor will be supported. The Xeon Phi 7120X is rated at 1.2 teraflops doing double precision floating point math, while the Tesla K40 card has a base performance of 1.43 teraflops that can rise to 1.66 teraflops with GPU Boost overclocking turned on. That works out to 614 teraflops per rack with Xeon Phis and 732 teraflops per rack with the Tesla K40s (not counting the extra performance from GPU Boost).

eurotech-hive-rack_openBack in June at ISC, Eurotech was talking up the Hive system (which did not yet have that name) by saying that it would be delivering a variant of the system that would marry a 64-bit ARM processor from Applied Micro with Tesla GPU coprocessors, and you might have gotten the impression that this would come out first. While Applied Micro is shipping its “Storm” X-Gene 1 chip now, it is readying the much-better “Shadowcat” X-Gene 2 processor, which has been sampling since August. This chip will support the RDMA over Converged Ethernet (RoCE) protocol over its integrated Ethernet network interface cards, simplifying the components that go into an ARM server node. The X-Gene 1 and 2 chips have two 10 Gb/sec Ethernet ports on the die, and these can be hooked eight into adapter ports. That, in theory, leaves more room for other peripherals in the complex. The plan is to ship the X-Gene 2 as the ARM option for the CPU side of the hybrid node, along with the Tesla K40 cards as coprocessors, sometime around the second quarter of 2015.

Incidentally, Eurotech is able to get its hands on a modified Tesla K40 card with its thermal plates modified so it fits into the super-skinny Hive module. The new Tesla K80 coprocessor card, announced this week at SC14, will be a bit tricky to add to the Aurora Hive system, explains Gallo, because this dual-GPU card has some of its power connectors across the top of the card. This does not work with the very tight tolerances in the Hive module, which are necessitated by the thermal conduction plates. With the Tesla K80 offering a base 1.87 teraflops of double precision math with a GPU Boost of up to 2.91 teraflops, you can bet some customers will want this. Gallo says that there is enough thermal capacity to pull the heat off this 300 watt part, if the connectors can be sorted. Being able to double the flops in the box is a pretty strong motivator to solve this engineering problem.

Generally speaking, the X86 processor option plus either the Xeon Phi or Tesla GPU accelerators draws about 1,500 watts per node, which works out to around 5 gigaflops per watt. The top machines on the Green500 ranking of supercomputers are in the range of 4 gigaflops per watt.

Gallo is tight lipped about what other processing components it might add to the Aurora Hive system, but obviously next year’s “Knights Landing” Xeon Phi, which will be sold as a standalone processor as well as a PCI-Express coprocessor, will slide right into this system. At 3 teraflops of double-precision floating point performance, and with the ability to put in five cards, this will be a radical increase in the math capabilities. And for dense-packed, CPU only workloads that used low-speed Ethernet, Eurotech could make Hive bricks that are just based on Xeon E3 or various ARM processors which sport their own networking on the chip. If you take out the network card, that leaves room for six CPU-only compute cards per module, or 768 processors per rack. Another option would be to add cards that have flash drives with the high-speed, low-latency NVM Express protocol linking into that PCI-Express switch. You could also swap out some of the flash drives and put in GPU cards for visualization to do visualization in the same nodes where the data is stored. Eurotech has lots of options with the Aurora Hive architecture, and that is so by design.

But initially at least, Eurotech is going after the workloads that have been accelerated. “There are markets where accelerated application have become the norm instead of an exotic thing,” says Gallo. “Geosciences, particularly reverse time migration reservoir analysis, is a good example. In general, signal processing will be interesting on this system, as well be machine learning, analytics, and some computer-aided engineering tools that have been modified for accelerators.”

The Aurora Hive comes preconfigured with the CentOS 6.X variant of Linux and support from Eurotech for this distribution, but customers can deploy other Linux operating systems on the machine as needed. Scientific Linux, Red Hat Enterprise Linux, SUSE Linux Enterprise Server, and Canonical Ubuntu Server are all supported. The Aurora software stack includes support for Intel Cluster Studio, Nvidia CUDA, MPSS, and the GCC compilers as well as the Intel MPI, Open MPI, and MVAPICH2 communication libraries.

Pricing for the Aurora Hive system was not available, and the question is what kind of premium can Eurotech charge for density and hot water cooling. The combination of the two should allow Eurotech to command a premium for its systems over plain vanilla clusters based on rack or blade servers, but it is a question as to how much. The market will decide.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industry updates delivered to you every week!

Nvidia’s New Blackwell GPU Can Train AI Models with Trillions of Parameters

March 18, 2024

Nvidia's latest and fastest GPU, code-named Blackwell, is here and will underpin the company's AI plans this year. The chip offers performance improvements from its predecessors, including the red-hot H100 and A100 GPUs. Read more…

Nvidia Showcases Quantum Cloud, Expanding Quantum Portfolio at GTC24

March 18, 2024

Nvidia’s barrage of quantum news at GTC24 this week includes new products, signature collaborations, and a new Nvidia Quantum Cloud for quantum developers. While Nvidia may not spring to mind when thinking of the quant Read more…

2024 Winter Classic: Meet the HPE Mentors

March 18, 2024

The latest installment of the 2024 Winter Classic Studio Update Show features our interview with the HPE mentor team who introduced our student teams to the joys (and potential sorrows) of the HPL (LINPACK) and accompany Read more…

Houston We Have a Solution: Addressing the HPC and Tech Talent Gap

March 15, 2024

Generations of Houstonian teachers, counselors, and parents have either worked in the aerospace industry or know people who do - the prospect of entering the field was normalized for boys in 1969 when the Apollo 11 missi Read more…

Apple Buys DarwinAI Deepening its AI Push According to Report

March 14, 2024

Apple has purchased Canadian AI startup DarwinAI according to a Bloomberg report today. Apparently the deal was done early this year but still hasn’t been publicly announced according to the report. Apple is preparing Read more…

Survey of Rapid Training Methods for Neural Networks

March 14, 2024

Artificial neural networks are computing systems with interconnected layers that process and learn from data. During training, neural networks utilize optimization algorithms to iteratively refine their parameters until Read more…

Nvidia’s New Blackwell GPU Can Train AI Models with Trillions of Parameters

March 18, 2024

Nvidia's latest and fastest GPU, code-named Blackwell, is here and will underpin the company's AI plans this year. The chip offers performance improvements from Read more…

Nvidia Showcases Quantum Cloud, Expanding Quantum Portfolio at GTC24

March 18, 2024

Nvidia’s barrage of quantum news at GTC24 this week includes new products, signature collaborations, and a new Nvidia Quantum Cloud for quantum developers. Wh Read more…

Houston We Have a Solution: Addressing the HPC and Tech Talent Gap

March 15, 2024

Generations of Houstonian teachers, counselors, and parents have either worked in the aerospace industry or know people who do - the prospect of entering the fi Read more…

Survey of Rapid Training Methods for Neural Networks

March 14, 2024

Artificial neural networks are computing systems with interconnected layers that process and learn from data. During training, neural networks utilize optimizat Read more…

PASQAL Issues Roadmap to 10,000 Qubits in 2026 and Fault Tolerance in 2028

March 13, 2024

Paris-based PASQAL, a developer of neutral atom-based quantum computers, yesterday issued a roadmap for delivering systems with 10,000 physical qubits in 2026 a Read more…

India Is an AI Powerhouse Waiting to Happen, but Challenges Await

March 12, 2024

The Indian government is pushing full speed ahead to make the country an attractive technology base, especially in the hot fields of AI and semiconductors, but Read more…

Charles Tahan Exits National Quantum Coordination Office

March 12, 2024

(March 1, 2024) My first official day at the White House Office of Science and Technology Policy (OSTP) was June 15, 2020, during the depths of the COVID-19 loc Read more…

AI Bias In the Spotlight On International Women’s Day

March 11, 2024

What impact does AI bias have on women and girls? What can people do to increase female participation in the AI field? These are some of the questions the tech Read more…

Alibaba Shuts Down its Quantum Computing Effort

November 30, 2023

In case you missed it, China’s e-commerce giant Alibaba has shut down its quantum computing research effort. It’s not entirely clear what drove the change. Read more…

Nvidia H100: Are 550,000 GPUs Enough for This Year?

August 17, 2023

The GPU Squeeze continues to place a premium on Nvidia H100 GPUs. In a recent Financial Times article, Nvidia reports that it expects to ship 550,000 of its lat Read more…

Analyst Panel Says Take the Quantum Computing Plunge Now…

November 27, 2023

Should you start exploring quantum computing? Yes, said a panel of analysts convened at Tabor Communications HPC and AI on Wall Street conference earlier this y Read more…

DoD Takes a Long View of Quantum Computing

December 19, 2023

Given the large sums tied to expensive weapon systems – think $100-million-plus per F-35 fighter – it’s easy to forget the U.S. Department of Defense is a Read more…

Shutterstock 1285747942

AMD’s Horsepower-packed MI300X GPU Beats Nvidia’s Upcoming H200

December 7, 2023

AMD and Nvidia are locked in an AI performance battle – much like the gaming GPU performance clash the companies have waged for decades. AMD has claimed it Read more…

Synopsys Eats Ansys: Does HPC Get Indigestion?

February 8, 2024

Recently, it was announced that Synopsys is buying HPC tool developer Ansys. Started in Pittsburgh, Pa., in 1970 as Swanson Analysis Systems, Inc. (SASI) by John Swanson (and eventually renamed), Ansys serves the CAE (Computer Aided Engineering)/multiphysics engineering simulation market. Read more…

Intel’s Server and PC Chip Development Will Blur After 2025

January 15, 2024

Intel's dealing with much more than chip rivals breathing down its neck; it is simultaneously integrating a bevy of new technologies such as chiplets, artificia Read more…

Baidu Exits Quantum, Closely Following Alibaba’s Earlier Move

January 5, 2024

Reuters reported this week that Baidu, China’s giant e-commerce and services provider, is exiting the quantum computing development arena. Reuters reported � Read more…

Leading Solution Providers

Contributors

Choosing the Right GPU for LLM Inference and Training

December 11, 2023

Accelerating the training and inference processes of deep learning models is crucial for unleashing their true potential and NVIDIA GPUs have emerged as a game- Read more…

Training of 1-Trillion Parameter Scientific AI Begins

November 13, 2023

A US national lab has started training a massive AI brain that could ultimately become the must-have computing resource for scientific researchers. Argonne N Read more…

Shutterstock 1179408610

Google Addresses the Mysteries of Its Hypercomputer 

December 28, 2023

When Google launched its Hypercomputer earlier this month (December 2023), the first reaction was, "Say what?" It turns out that the Hypercomputer is Google's t Read more…

Comparing NVIDIA A100 and NVIDIA L40S: Which GPU is Ideal for AI and Graphics-Intensive Workloads?

October 30, 2023

With long lead times for the NVIDIA H100 and A100 GPUs, many organizations are looking at the new NVIDIA L40S GPU, which it’s a new GPU optimized for AI and g Read more…

AMD MI3000A

How AMD May Get Across the CUDA Moat

October 5, 2023

When discussing GenAI, the term "GPU" almost always enters the conversation and the topic often moves toward performance and access. Interestingly, the word "GPU" is assumed to mean "Nvidia" products. (As an aside, the popular Nvidia hardware used in GenAI are not technically... Read more…

Shutterstock 1606064203

Meta’s Zuckerberg Puts Its AI Future in the Hands of 600,000 GPUs

January 25, 2024

In under two minutes, Meta's CEO, Mark Zuckerberg, laid out the company's AI plans, which included a plan to build an artificial intelligence system with the eq Read more…

Google Introduces ‘Hypercomputer’ to Its AI Infrastructure

December 11, 2023

Google ran out of monikers to describe its new AI system released on December 7. Supercomputer perhaps wasn't an apt description, so it settled on Hypercomputer Read more…

China Is All In on a RISC-V Future

January 8, 2024

The state of RISC-V in China was discussed in a recent report released by the Jamestown Foundation, a Washington, D.C.-based think tank. The report, entitled "E Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire