NASA Pushes Long-Range Climate Model Limits with SGI

By Nicole Hemsoth

November 17, 2014

The Discover system at NASA’s Center for Climate Simulation was designed with scalability and flexibility in mind, starting with its original nodes in 2006 and consistently evolving and growing with new hardware to handle the computational and data demands of complex long-term climate models.

In total, with the individual scalable units, the supercomputer is a combination of mostly IBM racks with some Dell and SuperMicro in the mix, culminate in just over a petaflop at peak across the over 43,000 cores.

But for this center, it’s out with the old and in with the new for around one-quarter to one-third of the system each year. The spare nodes are shipped off to be used in other research projects, leaving Dan Duffy, High Performance Computing Lead for NASA and his team with a revolving chain of decisions. For this round of system upgrades, they moved away from their IBM-heavy approach and looked to SGI’s Rackable line for memory, power, and CPU balance.

In the latest upgrade round, the team removed 1200 nodes from Discover or about 14,000 cores—the equivalent of roughly 150 teraflops of compute that was installed in 2010. Now, in the same footprint and power envelope, the new 18 racks they are receiving from SGI offers more performance (over a petaflop peak) and 30,000 cores for 1000 nodes. This upgrade represents almost an 8x performance boost in the same space when compared to what was available four years ago.

The key to these efficiencies are inside SGI’s Rackable cluster line, which for NCCS means 1.9 petaflops of capability powered by the Xeon E5-2696 v3 processors fed with a non-blocking fat tree FDR interconnect from Mellanox. Duffy said that while it was tempting to hold off and wait for EDR, especially since these applications are between 30-40% message passing, they needed the system up and running right away.

Further, Duffy added that if they were going to delay to have EDR in the machine, there are some key technologies coming in the future that are also worth the wait. While he and his team have been experimenting with both GPUs and Xeon Phis for some time, there’s still a great deal of code work to be done. He said that they’re keenly interested in where Knight’s Landing will take their workloads and are taking a particular shine to 64-bit ARM, if for no other reason than what it will mean for the competitive processor ecosystem.

“A couple of years ago we explored the Phi and made good progress but our results show that its performance is the same we can get in a node for this application. With GPUs, we upgraded to the NVIDIA K40, but the numbers are low. Our goal is to keep developing the applications to exploit manycore and hybrid core architectures. We’re interested in getting our hands on Knights Landing and comparing that to Haswell and eventually Broadwell. In a separate project we do a climate model on Power systems as well.”

When one takes into account the workload across this scalable architecture at NASA, it’s clear how important the CPU to data to storage balance is. The main application for complex long-range climate modeling is called GEOS-5, which integrates multiple models using the Earth System Modeling Framework—itself a beast in terms of computational requirements and complexity. Duffy says that to date, teams have been able to scaled past 30,000 cores. While right now they’re just running this at NCCS on roughly 8,000 cores, the plan is to take to its full heights with the SGI system. “It’s no small task,” laughed Duffy. “We are simulating the climate for a long amount of time at very high resolution. On top of that, there’s a 2 year run of this to look at the next generation of earth observing instruments to validate the work happening with that. That 2-year simulation is at a 7 km resolution and has generated 5PB of data.

This means that aside from raw compute and the critical FDR interconnect backbone, memory and storage are other considerations. In addition to the benchmarks that NASA uses for these applications to test everything from message passing to generalized performance, the memory footprint needed strong consideration. Duffy said that while he certainly isn’t at liberty to endorse any particular vendor, the SGI benchmark results delivered across all of these areas for this particular application area, opening new doors for the company, which just rolled out yet another line of HPC systems targeted at other workloads and featuring next generation Xeon E5-2600 v3 processors as well as a new UV system for data-intensive workloads that is based on the Xeon E7-8800 v2 processors, the capability to snap in GPUs and Xeon Phis, and most interestingly, the seventh generation of their NUMAlink technology which is arguably the true differentiating factor for SGI at the high end. We’ll be talking shortly with Dr. Goh about these new systems and their path forward with customers like NASA, who according to Duffy, are looking beyond standard rack systems and into a future that could include emerging technologies they’re watching on the processor, memory (specifically the HP Memristor, he says), and storage sides.

Computational story aside, at the end of the day, what Duffy and his systems are providing, are a glimpse into detailed, long-range climate narratives that will offer a first glimpse into how the planet’s climate has evolved from Paleolithic times until the present.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

At GTC: Nvidia Expands Scope of Its AI and Datacenter Ecosystem

March 19, 2019

In the high-stakes race to provide the AI life-cycle solution of choice, three of the biggest horses in the field are IBM, Intel and Nvidia. While the latter is only a fraction of the size of its two bigger rivals, and h Read more…

By Doug Black

AWS to Offer Nvidia’s T4 GPUs for AI Inferencing

March 19, 2019

The AI inference market is booming, prompting well-known hyperscaler and Nvidia partner Amazon Web Services to offer a new cloud instance that addresses the growing cost of scaling inference. The new “G4” instances... Read more…

By George Leopold

Nvidia Debuts Clara AI Toolkit with Pre-Trained Models for Radiology Use

March 19, 2019

AI’s push into healthcare got a boost yesterday with Nvidia’s release of the Clara Deploy AI toolkit which includes 13 pre-trained models for use in radiology. Clara, you may recall, is Nvidia’s biomedical platform Read more…

By John Russell

HPE Extreme Performance Solutions

HPE and Intel® Omni-Path Architecture: How to Power a Cloud

Learn how HPE and Intel® Omni-Path Architecture provide critical infrastructure for leading Nordic HPC provider’s HPCFLOW cloud service.

powercloud_blog.jpgFor decades, HPE has been at the forefront of high-performance computing, and we’ve powered some of the fastest and most robust supercomputers in the world. Read more…

IBM Accelerated Insights

The Spark That Ignited A New World of Real-Time Analytics

High Performance Computing has always been about Big Data. It’s not uncommon for research datasets to contain millions of files and many terabytes, even petabytes of data, or more. Read more…

DARPA, NSF Seek Real-Time ML Processor

March 18, 2019

A new U.S. research initiative seeks to develop a processor capable of real-time learning while operating with the “efficiency of the human brain.” The National Science Foundation (NSF) and the Defense Advanced Research Projects Agency jointly announced a “Real Time Machine Learning” project on March 15 soliciting industry proposals for “foundational breakthroughs” in hardware required to “build systems that respond and adapt in real time.” Read more…

By George Leopold

At GTC: Nvidia Expands Scope of Its AI and Datacenter Ecosystem

March 19, 2019

In the high-stakes race to provide the AI life-cycle solution of choice, three of the biggest horses in the field are IBM, Intel and Nvidia. While the latter is Read more…

By Doug Black

Nvidia Debuts Clara AI Toolkit with Pre-Trained Models for Radiology Use

March 19, 2019

AI’s push into healthcare got a boost yesterday with Nvidia’s release of the Clara Deploy AI toolkit which includes 13 pre-trained models for use in radiolo Read more…

By John Russell

It’s Official: Aurora on Track to Be First U.S. Exascale Computer in 2021

March 18, 2019

The U.S. Department of Energy along with Intel and Cray confirmed today that an Intel/Cray supercomputer, "Aurora," capable of sustained performance of one exaf Read more…

By Tiffany Trader

Why Nvidia Bought Mellanox: ‘Future Datacenters Will Be…Like High Performance Computers’

March 14, 2019

“Future datacenters of all kinds will be built like high performance computers,” said Nvidia CEO Jensen Huang during a phone briefing on Monday after Nvidia revealed scooping up the high performance networking company Mellanox for $6.9 billion. Read more…

By Tiffany Trader

Oil and Gas Supercloud Clears Out Remaining Knights Landing Inventory: All 38,000 Wafers

March 13, 2019

The McCloud HPC service being built by Australia’s DownUnder GeoSolutions (DUG) outside Houston is set to become the largest oil and gas cloud in the world th Read more…

By Tiffany Trader

Quick Take: Trump’s 2020 Budget Spares DoE-funded HPC but Slams NSF and NIH

March 12, 2019

U.S. President Donald Trump’s 2020 budget request, released yesterday, proposes deep cuts in many science programs but seems to spare HPC funding by the Depar Read more…

By John Russell

Nvidia Wins Mellanox Stakes for $6.9 Billion

March 11, 2019

The long-rumored acquisition of Mellanox came to fruition this morning with GPU chipmaker Nvidia’s announcement that it has purchased the high-performance net Read more…

By Doug Black

Optalysys Rolls Commercial Optical Processor

March 7, 2019

Optalysys, Ltd., a U.K. company seeking to advance it optical co-processor technology, moved a step closer this week with the unveiling of what it claims is th Read more…

By George Leopold

Quantum Computing Will Never Work

November 27, 2018

Amid the gush of money and enthusiastic predictions being thrown at quantum computing comes a proposed cold shower in the form of an essay by physicist Mikhail Read more…

By John Russell

The Case Against ‘The Case Against Quantum Computing’

January 9, 2019

It’s not easy to be a physicist. Richard Feynman (basically the Jimi Hendrix of physicists) once said: “The first principle is that you must not fool yourse Read more…

By Ben Criger

ClusterVision in Bankruptcy, Fate Uncertain

February 13, 2019

ClusterVision, European HPC specialists that have built and installed over 20 Top500-ranked systems in their nearly 17-year history, appear to be in the midst o Read more…

By Tiffany Trader

Intel Reportedly in $6B Bid for Mellanox

January 30, 2019

The latest rumors and reports around an acquisition of Mellanox focus on Intel, which has reportedly offered a $6 billion bid for the high performance interconn Read more…

By Doug Black

Looking for Light Reading? NSF-backed ‘Comic Books’ Tackle Quantum Computing

January 28, 2019

Still baffled by quantum computing? How about turning to comic books (graphic novels for the well-read among you) for some clarity and a little humor on QC. The Read more…

By John Russell

Why Nvidia Bought Mellanox: ‘Future Datacenters Will Be…Like High Performance Computers’

March 14, 2019

“Future datacenters of all kinds will be built like high performance computers,” said Nvidia CEO Jensen Huang during a phone briefing on Monday after Nvidia revealed scooping up the high performance networking company Mellanox for $6.9 billion. Read more…

By Tiffany Trader

Contract Signed for New Finnish Supercomputer

December 13, 2018

After the official contract signing yesterday, configuration details were made public for the new BullSequana system that the Finnish IT Center for Science (CSC Read more…

By Tiffany Trader

Deep500: ETH Researchers Introduce New Deep Learning Benchmark for HPC

February 5, 2019

ETH researchers have developed a new deep learning benchmarking environment – Deep500 – they say is “the first distributed and reproducible benchmarking s Read more…

By John Russell

Leading Solution Providers

SC 18 Virtual Booth Video Tour

Advania @ SC18 AMD @ SC18
ASRock Rack @ SC18
DDN Storage @ SC18
HPE @ SC18
IBM @ SC18
Lenovo @ SC18 Mellanox Technologies @ SC18
NVIDIA @ SC18
One Stop Systems @ SC18
Oracle @ SC18 Panasas @ SC18
Supermicro @ SC18 SUSE @ SC18 TYAN @ SC18
Verne Global @ SC18

IBM Quantum Update: Q System One Launch, New Collaborators, and QC Center Plans

January 10, 2019

IBM made three significant quantum computing announcements at CES this week. One was introduction of IBM Q System One; it’s really the integration of IBM’s Read more…

By John Russell

IBM Bets $2B Seeking 1000X AI Hardware Performance Boost

February 7, 2019

For now, AI systems are mostly machine learning-based and “narrow” – powerful as they are by today's standards, they're limited to performing a few, narro Read more…

By Doug Black

The Deep500 – Researchers Tackle an HPC Benchmark for Deep Learning

January 7, 2019

How do you know if an HPC system, particularly a larger-scale system, is well-suited for deep learning workloads? Today, that’s not an easy question to answer Read more…

By John Russell

HPC Reflections and (Mostly Hopeful) Predictions

December 19, 2018

So much ‘spaghetti’ gets tossed on walls by the technology community (vendors and researchers) to see what sticks that it is often difficult to peer through Read more…

By John Russell

Arm Unveils Neoverse N1 Platform with up to 128-Cores

February 20, 2019

Following on its Neoverse roadmap announcement last October, Arm today revealed its next-gen Neoverse microarchitecture with compute and throughput-optimized si Read more…

By Tiffany Trader

Move Over Lustre & Spectrum Scale – Here Comes BeeGFS?

November 26, 2018

Is BeeGFS – the parallel file system with European roots – on a path to compete with Lustre and Spectrum Scale worldwide in HPC environments? Frank Herold Read more…

By John Russell

France to Deploy AI-Focused Supercomputer: Jean Zay

January 22, 2019

HPE announced today that it won the contract to build a supercomputer that will drive France’s AI and HPC efforts. The computer will be part of GENCI, the Fre Read more…

By Tiffany Trader

Microsoft to Buy Mellanox?

December 20, 2018

Networking equipment powerhouse Mellanox could be an acquisition target by Microsoft, according to a published report in an Israeli financial publication. Microsoft has reportedly gone so far as to engage Goldman Sachs to handle negotiations with Mellanox. Read more…

By Doug Black

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This