IEEE Conference Keynoters Lay Out Path to Exascale Computing

By Aaron Dubrow

October 5, 2011

The challenges of exascale computing were the main focus of the three keynote addresses at the IEEE Cluster 2011 conference hosted in Austin, Texas from September 26 through 30. The speakers, renowned leaders in cluster computing, described the obstacles and opportunities involved in building systems one thousand times more powerful than today’s petascale supercomputers. Speaking from the perspective of the software developer (Thomas Sterling), the cluster designer (Liu GuangMing) and the chip architect (Charles Moore), each presented their thoughts on what is needed to reach exascale.

Thomas Sterling, Indiana University, Center for Research in Extreme Scale Technologies (CREST)

With a confidence born from long experience, Thomas Sterling, father of Beowulf, industry veteran, and associate director of the Center for Research in Extreme Scale Technologies (CREST) at Indiana University, kicked off the conference on Tuesday with a keynote on the need for a new paradigm in programming that will be adaptive, intelligent, asynchronous and able to get significantly better performance than today’s execution model.

Before jumping into an explanation of the new programming model, Sterling presented an eccentric history of cluster computing from the MIT Whirlwind project in the 1950s to Norbert Weiner’s cybernetic systems through the Beowulf era, where commodity PCs were first harnessed together to build a powerful cluster, to today’s petaflop mega-machines, one million times faster than the first Beowulf cluster.

Throughout the various phases of supercomputing innovation, several different programming paradigms have emerged, Sterling explained, from serial execution to vector processing to SIMD, to today’s dominant model, which uses MPI (Message Passing Interface) to communicate among many cores.

“Clusters will go through another metamorphosis,” Sterling predicted, adding, “commodity clusters will survive paradigm shifts.”

Current trends suggest the trajectory for computing speed is leveling. Sterling identified a number of problems that may prevent technologists from developing large systems. Power and reliability will be challenging, but Sterling sees the programming model as the biggest obstacle.

In the synchronous model represented by MPI, calculations need to be performed in a specific order, and with precision, to minimize latency, a dance that is difficult to keep up with. Only a handful of codes can run on the hundreds of thousands of cores that are available on today’s large supercomputers. Exascale computers, which Sterling said he hopes to see by the end of the decade, will likely have millions of cores.  At this level of core count, the component reliability and synchronization costs cannot accommodate the usual data-parallel computing approach.

“We must manage asynchrony to allow computing to be self-adaptive,” he said.

As an analogy, he pointed to the difference between a guided missile and a cannon. MPI represents an uncontrolled, ballistic, brute force method to solve problems. The new paradigm, or “experimental execution model” presented by Sterling, is exemplified by his own project, the ParalleX Research Group.

“ParalleX is an abstract test bed to explore the synthesis of ideas for current and extreme scale applications,” Sterling said. “We want to bring strong scaled applications back into the cluster world.”

His software employs micro-checkpointing: ephemeral detection and correction on the fly, and introspection (a kind of machine learning) closing the loop, as in cybernetics, to constantly adjust like the guided missile. It also manages asynchrony by “constraint-based synchronization.”

“You don’t want to tell the program when to do the tasks,” Sterling said. “You want to tell the program the conditions under which the task can be done. This allows the program to decide on its own when to undertake a given task.”

He pointed to initial performance gains for porting the adaptive mesh refinement algorithm for astrophysics to work on ParalleX execution. Results showed an improvement in performance of two to three times by changing the underlying context from MPI to ParalleX.

Some of these same goals are being pursued in a few significant, but not particularly well-known programming experiments, according to Sterling. In addition to ParalleX, he discussed examples from the StarsS project at the Barcelona Supercomputing Center, which employ a new model for data flow executions, and the SWift Adaptive Runtime Machine (SWARM) by ET International.

These execution models may not yet provide optimal computing, Sterling admitted, but the solutions being developed are needed for the community to advance.

“Cluster computing is going through a phase transition,” he asserted. “It will take leadership in this new paradigm shift and it will be the medium where a new paradigm is manifested. “

The tools are open source and XPI, the API for the execution environment, is in alpha testing and available to friendly users. It will be released soon to the general public.

Liu GuangMing, Director, National Supercomputer Center, Tianjin, China

Liu GuangMing, the designer of Tianhe-1A — China’s most powerful supercomputer and the second most powerful in the world — began his Wednesday keynote with an overview of the system deployed at the National Supercomputer Center in Tianjin, China.  He followed with an analysis of the barriers that designers face in building an exascale system.

Built from 143,336 Intel CPU processors, 7168 NVIDIA GPUs, and 2048 Galaxy FT-1000 eight-core processors designed by Liu himself, Tianhe-1A has a peak performance of 2.56 petaflops. The hybrid cluster is comprised largely of commodity parts; however, a few of the components, including the interconnects and FT chips, are proprietary.

“To get to the petascale, you can choose a traditional design or a new design,” Liu said. “We have been looking for a new way to design and implement a petaflop supercomputer.”

When it was deployed in 2010, many in the HPC world questioned Tianhe-1A’s ability to run scientific applications efficiently. Liu described a broad range of problems that used thousands to hundreds of thousands of processors with great efficiency, from seismic imaging for petroleum exploration to decoding the genome of the E. coli bacteria that sickened thousands in Germany. These results were delivered and put to bed some of the questions about Tianhe-1A’s usability.

After describing the technological and scientific successes of Tianhe-1A, Liu transitioned to a discussion of the problems associated with future exascale systems. He divided the problems into five categories: power, memory, communication, reliability, and application scalability, and quantified each problem with mathematical models.

Literally.

Transforming each of the main challenges into equations, he described how the models depict the obstacles facing continued speedups. The goal of this endeavor was to “build a synthesized speedup model and define quantitatively the ‘walls’,” Liu said.

He went on to suggest potential ways over each wall, sometimes through concerted effort by the HPC community, sometimes through emerging innovations.

Liu also showed enthusiasm for untested, emerging technologies such as optical or wireless interconnects, nanoelectronics and quantum and DNA computing, all of which he expects to play a role in the evolution of new systems. He pointed to the high-speed 3D interconnects associated with the Cray XT5 and Fujitsu K computer systems as examples of current technologies that he believes are on the right path to reaching the exascale.

Liu also gave examples of instances where the community must do a better job of optimizing applications for larger systems. Speaking about computer memory, he classified six types of data access that must be considered when speeding-up and scaling-up applications to tens of thousands of cores.

“Traditional optimization techniques usually consider only some of these characteristics,” Liu said. “We must consider all six characteristics and create a harmonious optimization algorithm.”

This holistic, deep thinking about the interrelationship of various levels of computation were the main message of Liu’s presentation. He repeatedly returned to graphs that showed the impact of various processes, from memory access and communication, to power consumption and cost, on the overall time and efficiency of computation.

“To reach the exascale, we must research solutions at all system levels,” Liu concluded.

Charles Moore, Corporate Fellow and the Technology Group CTO, Advanced Micro Devices

Reaching exascale was the subtext of Charles Moore’s Thursday keynote at IEEE Cluster 2011, but AMD’s emerging line of accelerated processing units (APUs) was the real subject of his talk.

APUs are a class of chip that Moore believes will power future exascale systems. According to Moore, exascale systems will achieve their massive speedup by using both CPUs and GPUs or other accelerators.

“We are approaching what we at AMD call the heterogeneous systems era,” Moore said. That alone is not groundbreaking; what is important is the fact but for AMD, these cores will all be located on the same chip.

Among the chips discussed by Moore were the “Brazos” E-series Fusion APU, which contains dual cores, dual GPUs, and a video accelerator on a single chip. It achieves 90 gigaflops of single-precision performance using just 18W TDP. “Desna,” Brazos’ little cousin, runs on only 6W, and is suitable for passively cooled designs like tablets. “Llano,” AMD’s higher-end chip, will have four CPU cores, advanced GPUs, and will offer 500 gigaflops of compute power per node.

One advantage of AMD’s new line is that you “can use this chip for graphics or as a compute offload or both at the same time,” Moore said.

The powerful chips that Moore prophesied won’t quite take us to the exascale, but they will get us most of the way, he said. For exascale, an overhaul of the memory architecture and programming models is needed.

Moore alluded to 3D stacked memory being developed by AMD as a possible technological solution to memory access problems. He also described the new AMD Fusion system architecture, where the goal is “making the GPU a first class citizen in the system architecture.”

The Fusion system architecture itself is “agnostic for CPU and GPU.”  “We’ll add other accelerators to this frame in the future,” Moore said. “It’s not just about GPUs, it’s about heterogeneous computing in general.”

Openness was a common theme in the last part of Moore’s talk where he described AMD’s long-standing dedication to open source software and standards. He discussed emerging standards including HyperShare, the Open Compute Project, and the Common Communication Interface, which he believes will play key roles in getting to exascale.

“Open standards are the basis for large ecosystems,” he said. “If you look over time, open standards always win.”

Looking beyond the next-generation of chips, Moore described the potential for an “awesome exascale-class” 10-teraflop x86 APU computing node feasible in the 2018 timeframe.

“We intend to make the unprecedented processing capability of the APU as accessible to programmers as the CPU is today.”

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Q&A with Google’s Bill Magro, an HPCwire Person to Watch in 2021

June 11, 2021

Last Fall Bill Magro joined Google as CTO of HPC, a newly created position, after two decades at Intel, where he was responsible for the company's HPC strategy. This interview was conducted by email at the beginning of A Read more…

A Carbon Crisis Looms Over Supercomputing. How Do We Stop It?

June 11, 2021

Supercomputing is extraordinarily power-hungry, with many of the top systems measuring their peak demand in the megawatts due to powerful processors and their correspondingly powerful cooling systems. As a result, these Read more…

Honeywell Quantum and Cambridge Quantum Plan to Merge; More to Follow?

June 10, 2021

Earlier this week, Honeywell announced plans to merge its quantum computing business, Honeywell Quantum Solutions (HQS), which focuses on trapped ion hardware, with the U.K.-based Cambridge Quantum Computing (CQC), which Read more…

ISC21 Keynoter Xiaoxiang Zhu to Deliver a Bird’s-Eye View of a Changing World

June 10, 2021

ISC High Performance 2021 – once again virtual due to the ongoing pandemic – is swiftly approaching. In contrast to last year’s conference, which canceled its in-person component with a couple months’ notice, ISC Read more…

Xilinx Expands Versal Chip Family With 7 New Versal AI Edge Chips

June 10, 2021

FPGA chip vendor Xilinx has been busy over the last several years cranking out its Versal AI Core, Versal Premium and Versal Prime chip families to fill customer compute needs in the cloud, datacenters, networks and more. Now Xilinx is expanding its reach to the booming edge... Read more…

AWS Solution Channel

Building highly-available HPC infrastructure on AWS

Reminder: You can learn a lot from AWS HPC engineers by subscribing to the HPC Tech Short YouTube channel, and following the AWS HPC Blog channel. Read more…

Space Weather Prediction Gets a Supercomputing Boost

June 9, 2021

Solar winds are a hot topic in the HPC world right now, with supercomputer-powered research spanning from the Princeton Plasma Physics Laboratory (which used Oak Ridge’s Titan system) to University College London (which used resources from the DiRAC HPC facility). One of the larger... Read more…

A Carbon Crisis Looms Over Supercomputing. How Do We Stop It?

June 11, 2021

Supercomputing is extraordinarily power-hungry, with many of the top systems measuring their peak demand in the megawatts due to powerful processors and their c Read more…

Honeywell Quantum and Cambridge Quantum Plan to Merge; More to Follow?

June 10, 2021

Earlier this week, Honeywell announced plans to merge its quantum computing business, Honeywell Quantum Solutions (HQS), which focuses on trapped ion hardware, Read more…

ISC21 Keynoter Xiaoxiang Zhu to Deliver a Bird’s-Eye View of a Changing World

June 10, 2021

ISC High Performance 2021 – once again virtual due to the ongoing pandemic – is swiftly approaching. In contrast to last year’s conference, which canceled Read more…

Xilinx Expands Versal Chip Family With 7 New Versal AI Edge Chips

June 10, 2021

FPGA chip vendor Xilinx has been busy over the last several years cranking out its Versal AI Core, Versal Premium and Versal Prime chip families to fill customer compute needs in the cloud, datacenters, networks and more. Now Xilinx is expanding its reach to the booming edge... Read more…

What is Thermodynamic Computing and Could It Become Important?

June 3, 2021

What, exactly, is thermodynamic computing? (Yes, we know everything obeys thermodynamic laws.) A trio of researchers from Microsoft, UC San Diego, and Georgia Tech have written an interesting viewpoint in the June issue... Read more…

AMD Introduces 3D Chiplets, Demos Vertical Cache on Zen 3 CPUs

June 2, 2021

At Computex 2021, held virtually this week, AMD showcased a new 3D chiplet architecture that will be used for future high-performance computing products set to Read more…

Nvidia Expands Its Certified Server Models, Unveils DGX SuperPod Subscriptions

June 2, 2021

Nvidia is busy this week at the virtual Computex 2021 Taipei technology show, announcing an expansion of its nascent Nvidia-certified server program, a range of Read more…

Using HPC Cloud, Researchers Investigate the COVID-19 Lab Leak Hypothesis

May 27, 2021

At the end of 2019, strange pneumonia cases started cropping up in Wuhan, China. As Wuhan (then China, then the world) scrambled to contain what would, of cours Read more…

AMD Chipmaker TSMC to Use AMD Chips for Chipmaking

May 8, 2021

TSMC has tapped AMD to support its major manufacturing and R&D workloads. AMD will provide its Epyc Rome 7702P CPUs – with 64 cores operating at a base cl Read more…

Intel Launches 10nm ‘Ice Lake’ Datacenter CPU with Up to 40 Cores

April 6, 2021

The wait is over. Today Intel officially launched its 10nm datacenter CPU, the third-generation Intel Xeon Scalable processor, codenamed Ice Lake. With up to 40 Read more…

Berkeley Lab Debuts Perlmutter, World’s Fastest AI Supercomputer

May 27, 2021

A ribbon-cutting ceremony held virtually at Berkeley Lab's National Energy Research Scientific Computing Center (NERSC) today marked the official launch of Perlmutter – aka NERSC-9 – the GPU-accelerated supercomputer built by HPE in partnership with Nvidia and AMD. Read more…

CERN Is Betting Big on Exascale

April 1, 2021

The European Organization for Nuclear Research (CERN) involves 23 countries, 15,000 researchers, billions of dollars a year, and the biggest machine in the worl Read more…

Google Launches TPU v4 AI Chips

May 20, 2021

Google CEO Sundar Pichai spoke for only one minute and 42 seconds about the company’s latest TPU v4 Tensor Processing Units during his keynote at the Google I Read more…

Iran Gains HPC Capabilities with Launch of ‘Simorgh’ Supercomputer

May 18, 2021

Iran is said to be developing domestic supercomputing technology to advance the processing of scientific, economic, political and military data, and to strengthen the nation’s position in the age of AI and big data. On Sunday, Iran unveiled the Simorgh supercomputer, which will deliver.... Read more…

HPE Launches Storage Line Loaded with IBM’s Spectrum Scale File System

April 6, 2021

HPE today launched a new family of storage solutions bundled with IBM’s Spectrum Scale Erasure Code Edition parallel file system (description below) and featu Read more…

Quantum Computer Start-up IonQ Plans IPO via SPAC

March 8, 2021

IonQ, a Maryland-based quantum computing start-up working with ion trap technology, plans to go public via a Special Purpose Acquisition Company (SPAC) merger a Read more…

Leading Solution Providers

Contributors

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from Julialang.org. I Read more…

AMD Launches Epyc ‘Milan’ with 19 SKUs for HPC, Enterprise and Hyperscale

March 15, 2021

At a virtual launch event held today (Monday), AMD revealed its third-generation Epyc “Milan” CPU lineup: a set of 19 SKUs -- including the flagship 64-core, 280-watt 7763 part --  aimed at HPC, enterprise and cloud workloads. Notably, the third-gen Epyc Milan chips achieve 19 percent... Read more…

Can Deep Learning Replace Numerical Weather Prediction?

March 3, 2021

Numerical weather prediction (NWP) is a mainstay of supercomputing. Some of the first applications of the first supercomputers dealt with climate modeling, and Read more…

Livermore’s El Capitan Supercomputer to Debut HPE ‘Rabbit’ Near Node Local Storage

February 18, 2021

A near node local storage innovation called Rabbit factored heavily into Lawrence Livermore National Laboratory’s decision to select Cray’s proposal for its CORAL-2 machine, the lab’s first exascale-class supercomputer, El Capitan. Details of this new storage technology were revealed... Read more…

GTC21: Nvidia Launches cuQuantum; Dips a Toe in Quantum Computing

April 13, 2021

Yesterday Nvidia officially dipped a toe into quantum computing with the launch of cuQuantum SDK, a development platform for simulating quantum circuits on GPU-accelerated systems. As Nvidia CEO Jensen Huang emphasized in his keynote, Nvidia doesn’t plan to build... Read more…

Microsoft to Provide World’s Most Powerful Weather & Climate Supercomputer for UK’s Met Office

April 22, 2021

More than 14 months ago, the UK government announced plans to invest £1.2 billion ($1.56 billion) into weather and climate supercomputing, including procuremen Read more…

African Supercomputing Center Inaugurates ‘Toubkal,’ Most Powerful Supercomputer on the Continent

February 25, 2021

Historically, Africa hasn’t exactly been synonymous with supercomputing. There are only a handful of supercomputers on the continent, with few ranking on the Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire