Full Details Uncovered on Chinese Top Supercomputer

By Nicole Hemsoth

June 2, 2013

At the end of May, an international group of high performance computing researchers gathered at the International HPC Forum in Changsha, China. One of the talks detailed the specs for the new Tianhe-2 system, which as we reported last week, is expected to rather dramatically top the Top500 list of the world’s fastest supercomputers.

China, supercomputer, Tianhe-2, Tianhe, Tianhe2
Artist’s rendering of the system as it will look once finally implemented at its final destination.

As noted previously, the system will be housed at the National Supercomputer Center in Guangzhou and has been aimed at providing an open platform for research and education and to provide a high performance computing service for southern China.

Dr. Jack Dongarra from the University of Tennessee and Oak Ridge National Lab, one of the founders of the Top500, was on hand for the event in China and shared a draft document that offers deep detail on the full scope of the Tianhe-2, which will, barring any completely unexpected surprises, far surpass the Cray-built Titan.

The 16,000-node Inspur-built Tianhe-2 is based on Ivy Bridge (32,000 sockets) and 48,000 Xeon Phi boards, meaning a total of 3,120,000 cores. Each of the nodes sports 2 Ivy Bridge sockets and 3 Phi boards.

According to Dongarra, there are some new and notable LINPACK results:

I was sent results showing a run of HPL benchmark using 14,336 nodes, that run was made using 50 GB of the memory of each node and achieved 30.65 petaflops out of a theoretical peak of 49.19 petaflops, or an efficiency of 62.3% of theoretical peak performance taking a little over 5 hours to complete.The fastest result shown was using 90% of the machine. They are expecting to make improvements and increase the number of nodes used in the test.

This certainly seems to confirm that this will indeed be the top system on this June’s list. But let’s take a closer look at some architectural elements to put those numbers in context…

Interestingly, each of the Phi boards have 57 cores instead of 61. This is because they were early in the production cycle at the time and yield was an issue. Still each of the 57 cores can boast 4 threads of execution and each thread can hit 4 flops per cycle. By Dongarra’s estimate, the 1.1 GHz cycle time produces a theoretical peak of 1.003 teraflops for each Phi element.

Each of the nodes is laden with 64 GB of memory, each of the Phi elements come with 8 GB of memory for a total of 88 GB of memory per node for a total of full system memory at 1.404 petabytes. There is not a lot of detail about the storage infrastructure, but there is a global shared parallel storage system sporting 12.4 petabytes.

According to Dongarra, there are “2 nodes per board, 16 boards per frame, 4 frames per rack, and 125 racks make up the system.” He says that the compute board has two compute nodes and is composed of two halves—the CPM and APM. The CPM portion of the board contains the 4 Ivy Bridge processors, memory and 1 Xeon Phi board while the CPM half contains the 5 Xeon Phi boards.

compute, node, xeon, tianhe-2, tianhe2, china, supercomputer

There are also 5 horizontal blind push-pull connections on the edge; connections from the Ivy Bridges to each of the coprocessors are made via PCI-E 2, which has 16 lanes and are 10 Gbps each. Dongarra points out that the actual design and implementation of the board is for PCI-E 3.0 but the Phi only supports PCI0E 2. There is also a PCI-E connection to the NIC.

We already knew that this was a system from the Chinese IT company, Inspur. According to Dongarra, “Inspur contributed to the manufacturing of the printed circuit boards and is also contributing to the system installation and testing.” At this point, the system is still being assembled and tested at the National University of Defense Technology before being installed at its permanent home.

As we know from the original Tianhe-1A system, NUDT has been hard at work on their own interconnects. On the TH-2, they are using their TH Express-2 interconnect network, which taps a fat tree topology with 13 switches, each with 576 ports at the top level.

As Dongarra notes, “This is an optoelectronics hybrid transport technology and runs a proprietary network. The interconnect uses their own chip set. The high radix router ASIC called NRC has a 90 nm feature size with a 17.16×17.16 mm die and 2577 pins.”

He says that “the throughput of a single NRC is 2.56 Tbps. The network interface ASIC called NIC has the same feature size and package as the NIC, the die size is 10.76×10.76 mm, 675 pins and uses PCI-E G2 16X. A broadcast operation via MPI was running at 6.36 GB/s and the latency measured with 64K of data within 12,000 nodes is about 85us.

Dongarra says that the 720 square meter footprint means a rather confined space and isn’t optimally laid out. However, this is just temporary since when it arrives in its permanent home in Guangzhou it will be laid out more efficiently, as seen in the artist’s rendering of the system at the top of the article.

The peak power consumption under load for the system is 17.6 MWs, but this is just for the processors, memory and interconnect network. When the closely-coupled chilled water with customized liquid water cooling unit operations are added in, the total consumption is 24 MWs. Dongarra says that it has a high cooling capacity of 80 KW and when installed at its home site, it will use city water as its source. Power load is monitored by a series of lights on the cabinet doors.

For far more details about these and other aspects of the Tianhe-2 system, check out Dr. Dongarra’s extensive report…

http://www.netlib.org/utk/people/JackDongarra/PAPERS/tianhe-2-dongarra-report.pdf

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industry updates delivered to you every week!

Anders Jensen Discusses EuroHPC Ahead of SC24

November 12, 2024

Thanks to increased funding from the European Union and stepped-up collaboration between the Commission-supported EuroHPC initiative and EU member states Europe has become one of the global leaders in HPC and related AI Read more…

Boffo HPC Conference Sessions: Leicester 2024

November 11, 2024

Leicester was the center of the HPC universe on October 15-16 as the HPC/AI Advisory Council and DiRAC hosted their sixth annual UK conference.The theme this year was “Democratising HPC and AI Opportunities” was co Read more…

Top500 Wild Cards Could Add Thrills to Supercomputing 2024 Show

November 11, 2024

The fantastic Supercomputing 2024 show is coming back to Atlanta this year. If last year was any indication, there will be plenty of coffee flowing and lots of discussions around power efficiency, storage, and the future Read more…

The Growing E-Waste Footprint of GenAI

November 7, 2024

The rapid advancement of digital technologies has led to the proliferation of electronic devices and systems, resulting in an alarming increase in electronic waste (e-waste).  GenAI, in particular, requires substantial Read more…

OSI Open AI Definition Stops Short of Requiring Open Data for LLMs

November 6, 2024

The movement toward open source AI made progress today when the Open Source Initiative released the first Open Source AI Definition (OSAID). While the OSAID provides one step forward, the lack of requirements around open Read more…

Microsoft Azure & AMD Solution Channel

Join Microsoft Azure and AMD at SC24

Atlanta, Georgia is the place to be this fall as the high-performance computing (HPC) community convenes for Supercomputing 2024. SC24 will bring together an unparalleled mix of scientists, engineers, researchers, educators, programmers, and developers for a week of learning and sharing. Read more…

D-Wave Readies 4,400-plus-qubit Advantage2 System for Use

November 6, 2024

Quantum computing pioneer D-Wave today announced it had completed calibration and benchmarking the latest latest version of its Advantage2 quantum processor, a 4,400-plus-qubit device. D-Wave said that compared with the Read more…

Anders Jensen Discusses EuroHPC Ahead of SC24

November 12, 2024

Thanks to increased funding from the European Union and stepped-up collaboration between the Commission-supported EuroHPC initiative and EU member states Europe Read more…

Top500 Wild Cards Could Add Thrills to Supercomputing 2024 Show

November 11, 2024

The fantastic Supercomputing 2024 show is coming back to Atlanta this year. If last year was any indication, there will be plenty of coffee flowing and lots of Read more…

OSI Open AI Definition Stops Short of Requiring Open Data for LLMs

November 6, 2024

The movement toward open source AI made progress today when the Open Source Initiative released the first Open Source AI Definition (OSAID). While the OSAID pro Read more…

Bill Gropp on ‘Different Approaches to AI’

November 6, 2024

Around this same time last year, I expounded on what the “Future of AI” may entail. A lot has happened in the 12 months since then, including new approaches Read more…

Shutterstock 1179408610

Google Cloud Sporting a New Look in HPC and AI Hardware

November 5, 2024

It's raining hardware at Google Cloud, with the company making major upgrades in advance of bringing Nvidia's Blackwell GPUs into its fold next year. The upg Read more…

Go (Mountain) West, Quantum Workers! CU, CUbit, and Elevate Quantum Issue Workforce Roadmap

November 5, 2024

Last week the University of Colorado (Boulder), the CUbit Quantum Initiative, and the Elevate Quantum consortium released workforce roadmap for educating and bu Read more…

Collaboration Speeds Complex Chemical Modeling

November 4, 2024

A recent collaboration among researchers from HUN-REN Wigner Research Centre for Physics in Hungary and the Department of Energy's Pacific Northwest National La Read more…

High-Performance Storage for AI and Analytics Panel

October 31, 2024

When storage is mentioned in an AI or Big Data analytics context, it is assumed to be a high-performance system. In practice, it may not be, and the user eventu Read more…

Sorry, but nothing matches what you're looking for. Please try again with some different keywords.

Leading Solution Providers

Contributors

CORNELL I-WAY DEMONSTRATION PITS PARASITE AGAINST VICTIM

October 6, 1995

Ithaca, NY --Visitors to this year's Supercomputing '95 (SC'95) conference will witness a life-and-death struggle between parasite and victim, using virtual Read more…

SGI POWERS VIRTUAL OPERATING ROOM USED IN SURGEON TRAINING

October 6, 1995

Surgery simulations to date have largely been created through the development of dedicated applications requiring considerable programming and computer graphi Read more…

U.S. Will Relax Export Restrictions on Supercomputers

October 6, 1995

New York, NY -- U.S. President Bill Clinton has announced that he will definitely relax restrictions on exports of high-performance computers, giving a boost Read more…

Dutch HPC Center Will Have 20 GFlop, 76-Node SP2 Online by 1996

October 6, 1995

Amsterdam, the Netherlands -- SARA, (Stichting Academisch Rekencentrum Amsterdam), Academic Computing Services of Amsterdam recently announced that it has pur Read more…

Cray Delivers J916 Compact Supercomputer to Solvay Chemical

October 6, 1995

Eagan, Minn. -- Cray Research Inc. has delivered a Cray J916 low-cost compact supercomputer and Cray's UniChem client/server computational chemistry software Read more…

NEC Laboratory Reviews First Year of Cooperative Projects

October 6, 1995

Sankt Augustin, Germany -- NEC C&C (Computers and Communication) Research Laboratory at the GMD Technopark has wrapped up its first year of operation. Read more…

Sun and Sybase Say SQL Server 11 Benchmarks at 4544.60 tpmC

October 6, 1995

Mountain View, Calif. -- Sun Microsystems, Inc. and Sybase, Inc. recently announced the first benchmark results for SQL Server 11. The result represents a n Read more…

New Study Says Parallel Processing Market Will Reach $14B in 1999

October 6, 1995

Mountain View, Calif. -- A study by the Palo Alto Management Group (PAMG) indicates the market for parallel processing systems will increase at more than 4 Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire