Heading into SC16 CENATE Flexes its Growing Muscle

By John Russell

November 8, 2016

In September, the Center for Advanced Technology Evaluation (CENATE) at Pacific Northwest National Laboratory (PNNL) took possession of NVIDIA’s DGX-1 GPU-based (Pascal 100) supercomputer. More on what they are doing with it later. Soon, IBM will deliver its Con Tutto memory technology. Data Vortex’s advanced switch technology is already in-house, along with products (and ideas) from a handful of other technology heavyweights. Now entering its second year, CENATE already has some potent equipment and ambitious ideas.

“We have established the lofty goal for us to even design some neuromorphic technologies that are doing machine learning natively and not as you can do machine learning for example on a GPU in which you sort of come in from behind and map machine workload to the architecture of the GPU,” says Adolfy Hoisie, PNNL’s chief scientist for computing and CENATE’s principal investigator and director. All things (time and money) being available, “We would like those neuromorphic systems chips, whatever, to actually cast them in silicon.”

This is perhaps getting ahead of the story. Launched last fall and funded by the Department of Energy (DOE) Office of Advanced Scientific Computing Research (ASCR), CENATE is envisioned as a proving ground for advanced technology that is making its way into market – the DGX-1 and Data Vortex platforms are good examples – and as a lens for keeping a selective eye on technologies further out. A core goal is to assess these technologies for DOE workloads and to influence emerging architecture including those headed into leadership class systems.

Adolfy Hoisie, CENATE’s director and PNNL’s chief scientist for computing
Adolfy Hoisie, CENATE’s director and PNNL’s chief scientist for computing

In CENATE taxonomy, the center has what Hoisie labels as “four thrusts:”

  • Enablement of Tests Beds. “Test beds don’t have a uniform definitions of what they are because in CENATE even the notion of a technology pipeline, technology maturity pipeline [varies]. We are going to tackle technologies that range from very early concepts or blueprints all the way to pre-productions machines.”
  • Extensive Instrumentation and Measurement. “For a national laboratory, we have unique, instrumentation and measurement capability. This applies not only to measuring time, which is performance, but also to measurement of power and thermal effects, and we are contemplating venturing into reliability as well. At this time our lab has the capability to measure performance, power and thermal at very high resolutions and very high frequency for dynamic measurements. We measure both static and dynamic these technologies.”
  • Technology Evaluation. “Not all these technologies apply equally well to all the application workloads that ASCR may be interested in. What evaluation means is to determine what applications map well onto various architectures that are being contemplated and studied within CENATE then go the extra step in trying to guide future development of those activities within the applications.”
  • Modeling and Simulation. “[This] is a big forte of ours. It is a very exciting capability because if we had machines from vendor X from vendor Y and we have the capability to measure things, to validate the initial models, our modeling and simulation [expertise] allows us to ‘place’ this system into the future not just doing simple statistical extrapolations but really guiding the architecture to maximize the positive impact on the applications.”


Work on any technology may cross all CENATE domains and evaluation of NVIDIA’s DGX-1, CENATE’s latest addition, is a good example.

“Firstly we are interested in ways to accelerate computation. We are interested in PASCAL as the next generation of GPU. We’ll see what we learn through measurements on Pascal, running benchmarks, and pushing it forward with what-if scenarios, [looking for] when Pascal goes from where it is right now to double precision, to increased memory, to possibly modification in the SIMD characteristics and so forth,” says Hoisie.

Part of the interest, not surprisingly, stems from DOE’s forthcoming Summit supercomputer, to be based at Oak Ridge Leadership Computing Facility. Summit will be based on IBM Power technology and NVIDIA GPUS. Ian Buck, vice president of accelerated computing, NVIDIA, says the DGX-1, like Summit, is based on a strong node architecture versus weak node – where you have “a single node with maybe one or two cores, some memory, and the you replicated that en masse, hundreds of thousands, and you relied on the network for kind of MPI scalability to achieve performance at scale.”

The large infrastructure requirements (cabling, power, networks) inherently limited the weak node approach and complicated programming, he argues. “The vision we have been pursuing for exascale and in general for supercomputing is building these strong nodes systems like DGX-1 where we put a lot of horsepower into a single node and minimize the number of nodes you have to scale up to,” says Ian Buck, vice president of accelerated computing, NVIDIA.

As on the OLCF website: “Summit will deliver more than five times the computational performance of Titan’s 18,688 nodes, using only approximately 3,400 nodes when it arrives in 2017. Like Titan, Summit will have a hybrid architecture, and each node will contain multiple IBM POWER9 CPUs and NVIDIA Volta GPUs all connected together with NVIDIA’s high-speed NVLink. Each node will have over half a terabyte of coherent memory (high bandwidth memory + DDR4) addressable by all CPUs and GPUs plus 800GB of non-volatile RAM that can be used as a burst buffer or as extended memory. To provide a high rate of I/O throughput, the nodes will be connected in a non-blocking fat-tree using a dual-rail Mellanox EDR InfiniBand interconnect.”

NVIDIA Tesla P100 frontThe DGX-1 is a mini-version of sorts, with eight P100 and NVlink interconnect. Buck says NVIDIA is looking to CENATE for “insight into scalability with a strong node architecture and to help us define where the bottlenecks are in power and these workloads so we can better optimize.” There’s no shortage of questions. “Maybe we should be focusing more on 32-bit floating and not 64-bit floating point for some of these workloads. Also programmability is a challenge. Hoisie and PNNL are users of PGI and OpenACC and may have ideas how can we improve programmability and compiler ability.”

Power consumption, of course, is major worry in the race to exascale and an area where CENATE and its unique measurement abilities can help address. “We are in the single gigaflop per watt era now with these strong nodes supercomputers,” says Buck. “We should get into the double digit category relatively soon and the goals for exascale we’ve got to get upwards of 25 Gflops per watt,” says Buck who is interested to see what new ideas CENATE might offer.

Besides learning more about overall acceleration, Hoisie says understanding and assessing the DGX-1’s machine learning capability is an important objective.

“We have developed here, not part of CENATE but part of PNNL, very important scalable algorithms for the machine learning. Those libraries exist today and they are open source and we want to assess frankly them on the DGX-1. NVIDIA represents this as a machine learning box, and we believe that there is some truth in that, but you know as researchers in CENATE we are going to ask the question, ‘OK let’s quantify that. What does it mean for a box to be a machine learning box?,’ he says.

“Part of that is looking at how it compares to others, how does DGX-1 compare to Knights Landing [systems] for example. Again these DGX-1 and KNL systems are offerings you can go and buy now, or they are very close to being in that stage (the sweet spot for CENATE). So we are going to look not only at where DXX-1 may go in the future, where Pascal may go in the future, and what would that do to our workloads, but also how machine learning performs today on the DGX-1 and how it compares with other top-of-the-line systems,” Hoisie says. “We are also looking at new algorithms for machine learning.”

Hoisie expects CENATE’s modeling and simulation capabilities will be valuable in this portion of the DGX-1 work. “If we measure something on a current machine and we validate and calibrate a model, we get enough confidence to predict what is the performance of a different algorithm is, sometimes drastically different sometimes only marginally different.”

Yet a third project with the DGX-1 is work by distinguished PNNL researcher Ruby Leung and her team with portions of their climate modeling code. In particular, says Hoisie, they are look at portion of code already running on GPUs and are benchmarking the code’s performance on DGX-1: “We are going to be able to say the DGX-1 is this much faster or it is not or whatever the case may be and look at what needs to be done to improve the performance as the specs of the machine allows.” Leung is the Chief Scientist of Department of Energy Accelerated Climate Modeling for Energy (ACME).

There’s obviously lots going on inside CENATE. One important mission element is diffusing the knowledge it gleans into the broader HPC community. Given all of the IP involved and the center’s capacity to identify shortcomings as well as strengths, sharing information is tricky.

“We would like the researchers from all of the national laboratories and from academia to have the opportunity to access these resources and we are very much committed to that; however it’s not a simple exercise. Vendors are legitimately concerned about crosspollination of ideas. We at the national laboratory are equipped to deal with that but academia is less so,” says Hoisie. As a general rule all of the national labs have NDAs “with all the vendors that are in the HPC orbit.”

For good or ill, Hoisie says, “If we saw something wrong, we wouldn’t go and publish a paper on that attempting to talk down that product; instead we would point it out for the benefit of the vendor and for ourselves the ways in which that particular architecture or architectural issue can be improved.”

Part of effort to share learnings will occur at regular CENATE workshops. “We are planning to have the first CENATE symposium in early spring next year. We want it to have enough information [available] so we can discuss ideas with the users, [we] want to have systems all set up, machines on the floor that people can access, both locally and remotely, and then we are going to organize that. It is something very important for us to do.” A frequency of the meetings hasn’t set been determined.

Clearly CENATE does have lofty goals and its list of projects is growing. That said, Hoisie is quick to emphasize CENATE has “no interest in hording technology” wanting instead to use the resources it has in a focused way the produces more than just incrementally advanced. On early stage CENATE project is development of a scalability test capability that combines optical and electronic technology for the network.

“Imagine that you have, for example, an InfiniBand network that connects a cluster of nodes – these are yet to be determined – and the network is also comprised of optical technology that allows you to basically re-cable the machine in seconds or less without literally having to move any cable. This allows us to create enclaves within the system that isolates jobs from the rest of the activity on the system, which is a problem that plagues say many of these very large scale machine in which the nodes are dedicated to a job but not a network path.

The design of this system for this project is far along, says Hoisie, and the project is now in the procurement process. “It’s generated a lot of interest within the vendor community – Penguin Computing, Cray, DDN, Mellanox, and optical switch vendor Calient. These vendors are all so interested in this concept and its potential for future commercial uses that in some cases, for example Mellanox, is donating the entire Infiniband gear for it.” That’s an indication of CENATE’s growing success,” says Hoisie.

As has been the case in the past few years, the national labs do not have individual SC booths, but are represented in the DOE both. CENATE will have a presence there and Hoisie expects several participating vendors, perhaps NVIDIA for example, to also have CENATE materials or demos at their booths.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Geospatial Data Research Leverages GPUs

August 17, 2017

MapD Technologies, the GPU-accelerated database specialist, said it is working with university researchers on leveraging graphics processors to advance geospatial analytics. The San Francisco-based company is collabor Read more…

By George Leopold

Intel, NERSC and University Partners Launch New Big Data Center

August 17, 2017

A collaboration between the Department of Energy’s National Energy Research Scientific Computing Center (NERSC), Intel and five Intel Parallel Computing Centers (IPCCs) has resulted in a new Big Data Center (BDC) that Read more…

By Linda Barney

Google Releases Deeplearn.js to Further Democratize Machine Learning

August 17, 2017

Spreading the use of machine learning tools is one of the goals of Google’s PAIR (People + AI Research) initiative, which was introduced in early July. Last week the cloud giant released deeplearn.js as part of that in Read more…

By John Russell

HPE Extreme Performance Solutions

Leveraging Deep Learning for Fraud Detection

Advancements in computing technologies and the expanding use of e-commerce platforms have dramatically increased the risk of fraud for financial services companies and their customers. Read more…

Spoiler Alert: Glimpse Next Week’s Solar Eclipse Via Simulation from TACC, SDSC, and NASA

August 17, 2017

Can’t wait to see next week’s solar eclipse? You can at least catch glimpses of what scientists expect it will look like. A team from Predictive Science Inc. (PSI), based in San Diego, working with Stampede2 at the Read more…

By John Russell

Microsoft Bolsters Azure With Cloud HPC Deal

August 15, 2017

Microsoft has acquired cloud computing software vendor Cycle Computing in a move designed to bring orchestration tools along with high-end computing access capabilities to the cloud. Terms of the acquisition were not disclosed. Read more…

By George Leopold

HPE Ships Supercomputer to Space Station, Final Destination Mars

August 14, 2017

With a manned mission to Mars on the horizon, the demand for space-based supercomputing is at hand. Today HPE and NASA sent the first off-the-shelf HPC system i Read more…

By Tiffany Trader

AMD EPYC Video Takes Aim at Intel’s Broadwell

August 14, 2017

Let the benchmarking begin. Last week, AMD posted a YouTube video in which one of its EPYC-based systems outperformed a ‘comparable’ Intel Broadwell-based s Read more…

By John Russell

Deep Learning Thrives in Cancer Moonshot

August 8, 2017

The U.S. War on Cancer, certainly a worthy cause, is a collection of programs stretching back more than 40 years and abiding under many banners. The latest is t Read more…

By John Russell

IBM Raises the Bar for Distributed Deep Learning

August 8, 2017

IBM is announcing today an enhancement to its PowerAI software platform aimed at facilitating the practical scaling of AI models on today’s fastest GPUs. Scal Read more…

By Tiffany Trader

IBM Storage Breakthrough Paves Way for 330TB Tape Cartridges

August 3, 2017

IBM announced yesterday a new record for magnetic tape storage that it says will keep tape storage density on a Moore's law-like path far into the next decade. Read more…

By Tiffany Trader

AMD Stuffs a Petaflops of Machine Intelligence into 20-Node Rack

August 1, 2017

With its Radeon “Vega” Instinct datacenter GPUs and EPYC “Naples” server chips entering the market this summer, AMD has positioned itself for a two-head Read more…

By Tiffany Trader

Cray Moves to Acquire the Seagate ClusterStor Line

July 28, 2017

This week Cray announced that it is picking up Seagate's ClusterStor HPC storage array business for an undisclosed sum. "In short we're effectively transitioning the bulk of the ClusterStor product line to Cray," said CEO Peter Ungaro. Read more…

By Tiffany Trader

How ‘Knights Mill’ Gets Its Deep Learning Flops

June 22, 2017

Intel, the subject of much speculation regarding the delayed, rewritten or potentially canceled “Aurora” contract (the Argonne Lab part of the CORAL “ Read more…

By Tiffany Trader

Nvidia’s Mammoth Volta GPU Aims High for AI, HPC

May 10, 2017

At Nvidia's GPU Technology Conference (GTC17) in San Jose, Calif., this morning, CEO Jensen Huang announced the company's much-anticipated Volta architecture a Read more…

By Tiffany Trader

Reinders: “AVX-512 May Be a Hidden Gem” in Intel Xeon Scalable Processors

June 29, 2017

Imagine if we could use vector processing on something other than just floating point problems.  Today, GPUs and CPUs work tirelessly to accelerate algorithms Read more…

By James Reinders

Quantum Bits: D-Wave and VW; Google Quantum Lab; IBM Expands Access

March 21, 2017

For a technology that’s usually characterized as far off and in a distant galaxy, quantum computing has been steadily picking up steam. Just how close real-wo Read more…

By John Russell

Russian Researchers Claim First Quantum-Safe Blockchain

May 25, 2017

The Russian Quantum Center today announced it has overcome the threat of quantum cryptography by creating the first quantum-safe blockchain, securing cryptocurrencies like Bitcoin, along with classified government communications and other sensitive digital transfers. Read more…

By Doug Black

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

HPC Compiler Company PathScale Seeks Life Raft

March 23, 2017

HPCwire has learned that HPC compiler company PathScale has fallen on difficult times and is asking the community for help or actively seeking a buyer for its a Read more…

By Tiffany Trader

Groq This: New AI Chips to Give GPUs a Run for Deep Learning Money

April 24, 2017

CPUs and GPUs, move over. Thanks to recent revelations surrounding Google’s new Tensor Processing Unit (TPU), the computing world appears to be on the cusp of Read more…

By Alex Woodie

Leading Solution Providers

Trump Budget Targets NIH, DOE, and EPA; No Mention of NSF

March 16, 2017

President Trump’s proposed U.S. fiscal 2018 budget issued today sharply cuts science spending while bolstering military spending as he promised during the cam Read more…

By John Russell

CPU-based Visualization Positions for Exascale Supercomputing

March 16, 2017

In this contributed perspective piece, Intel’s Jim Jeffers makes the case that CPU-based visualization is now widely adopted and as such is no longer a contrarian view, but is rather an exascale requirement. Read more…

By Jim Jeffers, Principal Engineer and Engineering Leader, Intel

Google Debuts TPU v2 and will Add to Google Cloud

May 25, 2017

Not long after stirring attention in the deep learning/AI community by revealing the details of its Tensor Processing Unit (TPU), Google last week announced the Read more…

By John Russell

Six Exascale PathForward Vendors Selected; DoE Providing $258M

June 15, 2017

The much-anticipated PathForward awards for hardware R&D in support of the Exascale Computing Project were announced today with six vendors selected – AMD Read more…

By John Russell

Top500 Results: Latest List Trends and What’s in Store

June 19, 2017

Greetings from Frankfurt and the 2017 International Supercomputing Conference where the latest Top500 list has just been revealed. Although there were no major Read more…

By Tiffany Trader

IBM Clears Path to 5nm with Silicon Nanosheets

June 5, 2017

Two years since announcing the industry’s first 7nm node test chip, IBM and its research alliance partners GlobalFoundries and Samsung have developed a proces Read more…

By Tiffany Trader

MIT Mathematician Spins Up 220,000-Core Google Compute Cluster

April 21, 2017

On Thursday, Google announced that MIT math professor and computational number theorist Andrew V. Sutherland had set a record for the largest Google Compute Engine (GCE) job. Sutherland ran the massive mathematics workload on 220,000 GCE cores using preemptible virtual machine instances. Read more…

By Tiffany Trader

Messina Update: The US Path to Exascale in 16 Slides

April 26, 2017

Paul Messina, director of the U.S. Exascale Computing Project, provided a wide-ranging review of ECP’s evolving plans last week at the HPC User Forum. Read more…

By John Russell

  • arrow
  • Click Here for More Headlines
  • arrow
Share This