Compilers and More: Expose, Express, Exploit

By Michael Wolfe

March 28, 2011

Part 2: Programming at Exascale

In my previous column, I introduced six levels of parallelism that we’ll have in exascale systems:

  • Node level
  • Socket level
  • Core level
  • Vector level
  • Instruction level
  • Pipeline level

As we move towards exascale, we want to take advantage of all of these. We need to expose parallelism at all the levels, either explicitly in the program, or implicitly within the compiler. We need to express this parallelism, in a language and in the generated code. And we need to exploit the parallelism, efficiently and effectively, at runtime on the target machine. Performance is important; in fact, performance is key. The only reason we are looking at parallelism is for higher performance. This is a point I have made in the past, and I’ll say it again: the only reason to program in parallel is for higher performance. (Someone at SC10 responded that another reason to use parallelism is for redundancy, for error detection. I should have replied that redundancy is good, but you do the redundant computations in parallel to get the benefits of redundant computation without performance degradation.)

Exposing Parallelism

Someone has to find or create the parallelism in the program. At tera and petascale, we’ve mostly focused on data parallelism: large datasets, where the program can operate on separate data in parallel. At exascale, we’re likely to want even more parallelism, such as running coupled models in parallel. We’ll do the structural mechanics model concurrently with the heat transfer and deformation physics models. Since these models are coupled, it’s much easier to separate the problems and iterate through them one at a time; it will be a challenge to structure the code so they can run in parallel with each other. Parallelism at this high a level has to be exposed in the application design.

Within each algorithm, there is additional parallelism. This is where we get most of the parallel execution today, from individual algorithms. Parallelism at this level has finer granularity, and often is naturally dynamic. Some parallelism we get for free, meaning the compiler can find vector and instruction-level parallelism without changing the program. To be honest, this is a misrepresentation. The compiler can only find parallelism that already exists in the program. There are always ways to write the program to defeat the compiler; this isn’t a failing of the compiler, it’s compiler abuse.

Exposing parallelism, at any level, is a creative process. You choose how to organize your data such that it can be processed in parallel, or not. You choose the algorithms, solvers, conditioners, etc., that can use more parallelism, or that use less. You make these decisions in your application design. You might make different decisions, use different algorithms, depending on the target machine (laptop vs. cluster vs. exascale); this makes your job more challenging. However, since these decisions affect not only the parallelism and performance, but the accuracy and quality of the results, they have to be made by an expert, by a human.

Expressing Parallelism

Most of the discussion about parallel programming falls into this heading: how do I write my parallel program? Will I use MPI? Do I use some programming framework which is itself built on top of MPI? Do I use OpenMP directives? Do I use Cilk language extensions, or Unified Parallel C? Should I write in a consistent sequential language and use a compiler or other tool to find the parallelism? Should I use a truly higher order language, like SISAL or Haskell? Can I design a domain-specific language to make the programming more natural? Maybe I can build a C++ class library that will support my data structures directly?

There is a big step between designing an algorithm and expressing that algorithm in some language. Writing a parallel message-passing solver for large systems (such as the High Performance Linpack benchmark) is much more work that simply calling the LAPACK DGESV routine. You have to worry about data distribution, communication, and load balancing. Your answers to these will affect what algorithms you use locally and globally, and may be affected by your tolerance for numerical accuracy.

Since performance is key, we should focus on those aspects of the program that lead to high performance, or that might degrade performance. Given a parallel program, the performance keys are high locality and little synchronization. This has different meanings at different levels of parallelism. At the lowest levels of parallelism, locality is implemented in registers; the compiler manages register allocation to minimize memory accesses. We expect synchronization at the lower levels to be handled in hardware, almost for free, but we want the compiler to find enough instruction-level parallelism to exploit the functional units efficiently.

Between cores or sockets, locality is implemented in cache memory. We want the application to be organized to take advantage of the improved performance of caches for spatially and temporally local memory references. Modern cache memory hierarchies are large (12MB is not uncommon), but large parallel datasets can be huge (6GB is often considered small). Cache memory locality is typically optimized by loop tiling, usually manually in the program, but sometimes by the compiler. Cache memory locality for parallel cores can be hard to manage since some levels of the cache are shared between cores. This implies that we want those cores to share the data at that level in the cache, to avoid the cores interfering with each other. Cores on processors in separate sockets don’t share cache, so we’d like those cores to not share data, at least not very often, so the caches don’t thrash. Synchronization between cores that share memory is typically done through the memory, using locks or semaphores. It can be challenging to get these primitives correct and inserted in a way that gives high performance. Transactional memory has been proposed as a mechanism to simplify parallel programming. It treats shared data structure updates as atomic transactions, the same way a database implementation treats database updates. An implementation allows updates to separate areas of the shared data structure to proceed speculatively in parallel without explicit locks, although under the hood, as with databases, the transaction commit does use locks and incurs additional cost to verify that the transaction will behave as if it were truly atomic.

Any Single Program-Multiple Data (SPMD) program, whether MPI, Unified Parallel C, using coarrays in Fortran, or other method, requires the user to manage data locality explicitly in the program. In an MPI program, the programmer specifies what data to allocate on each node (MPI rank), and when to send or receive data from other nodes. In UPC or coarrays in Fortran, the programmer uses an additional index (or indices), the UPC shared array dimension or coarray image codimensions, to determine on which thread or image (i.e., node) the data resides. Using MPI, the program uses explicit messages; messages can have additional costs, such as implicit data copies and buffers on the sender or receiver side. However, messages also carry synchronization information, “the data is ready, and here’s the data,” along with the data. With an implicit or global address space model, the program must include separate synchronization primitives, as with a shared memory model. Neither is perfect, and both can be prone to errors.

If accelerators become common, then programs have to explicitly or implicitly be partitioned into CPU and accelerator parts. Current accelerators, such as GPUs, use a separate address space and physical memory. I hear predictions that on-chip accelerators, such as promised by the AMD Fusion devices, will solve this problem. I doubt this. Accelerators are designed for high-bandwidth memory access to support large data structures, whereas CPUs are designed for low latency operations. The CPU is supported by a deep cache hierarchy, which won’t help the accelerator, and current accelerators use a different memory implementation than a CPU. Today’s AMD Llano and Intel’s Sandy Bridge combine a relatively low-performance GPU on the CPU chip, designed to replace the integrated graphics chip that appears on many motherboards. These GPUs share the physical memory with the CPU, as those integrated graphics chips do, though not the same virtual address space. Such a solution incurs a performance penalty for not having dedicated graphics memory. Another approach would be to integrate the virtual address space of CPU and accelerator, while still maintaining the separate latency-oriented memory structure for data accessed from the CPU and bandwidth-oriented memory for accelerator data. Such an approach is used on Convey hybrid computers, with hardware to manage coherence between the CPU and accelerator, though again with performance penalties when CPU or accelerator accesses the other memory. The ultimate goal of full performance, coherent memory access across CPU and accelerator will be difficult to achieve.

The job of expressing parallelism is much more difficult today than it should be. This is largely because we have to include locality (data distribution) and synchronization (messages or explicit synchronization), and we have to express different levels of parallelism explicitly. If we want to target a multi-node, multi-core, accelerator based system, we might need to express message passing between the nodes, shared memory parallelism across cores within a node concurrently with accelerator parallelism on that node, and still leave enough vector parallelism to get the peak performance on each core. While some of this task is creative, much of it is mechanical. Choosing whether to distribute data by rows, columns, or panels can be a creative task, much as choosing a sparse matrix layout is creative. Inserting message primitives or optimizing synchronization placement is largely mechanical, and our programming mechanism should be able to handle this.

Exploiting Parallelism

The final step is to take the parallelism we’ve exposed and expressed and to map it to the target machine. With current MPI programs, this is a simple mechanical task, since all the work was done by the programmer. We should demand more from our implementation. We should want more flexibility across many dimensions, including scalability, dynamic parallelism, composability, load balancing, as well as productivity. Let’s take each in turn.

Scalability: The usual discussion of scalability is to be able to write programs that scale up to massive amounts of parallelism. In the exascale world, we need to have the right algorithms with enough parallelism to give us that level of performance; making those choices is a creative task, and we can’t expect to automate that. However, we should demand that the same program can be scaled down to run efficiently on our (smaller) terascale systems, our clusters, our workstations, and even our laptops. Automatically scaling down should be easier than automatically scaling up; in fact, it should be mostly mechanical, choosing which levels of parallelism to scale back or scalarize entirely. If the parallelism is expressed opaquely in the program (as with an MPI library), such decisions must be made by the programmer; we should design better programming strategies.

Dynamic parallelism: Most of the current large-scale parallel programming models are static: MPI mostly requires the processes to be created at program startup, though MPI-2 does add some weak support for dynamic process creation. Coarray Fortran and UPC similarly start a static number of images or threads. High Performance Fortran suffered from the same weakness. Shared memory models are often more dynamic; OpenMP can dynamically create nested parallel regions, and dynamically create tasks within each region, though the number of actual threads for each region is fixed. Cilk allows spawning of parallel strands, though the parallelism is exploited at runtime by some number of threads or processors managed by the runtime. The more recent High Productivity Programming Languages X10, Chapel and Fortress have more support for dynamic parallelism, and some traction among small groups of users, so perhaps there’s hope.

Composability: However I express my basic operations, the implementation must be able to efficiently compose these for the target processor. For instance, if I should write in Fortran:

    r = sum( x(:) * y(:) )

the definition is that the array product x(:)*y(:) be computed and stored in a temporary array, possibly dynamically allocated, then the elements of the temporary array are summed. However, the implementation shouldn’t have to create the temporary array; if this is being executed in scalar mode, the implementation should be just as efficient as the corresponding loop:

    r = 0     do i = . . .      r = r + x(i)*y(i)     enddo

If this is executed in parallel, each thread or process should be able to accumulate its partial sum efficiently in scalar mode, then the partial sums combined appropriately, with only one temporary scalar per thread. Some approaches to HPC are weak with respect to composability, as I’ll point out in the next column.

Load Balancing: Researchers in the past have developed implementations of parallel systems that monitor the performance across the cores and nodes, and redistribute the work and even the data to balance the load and improve the performance. With low-level parallel programming, work and data distribution is opaque to the runtime system, but there’s no reason we can’t change that. This requires certain characteristics of the program to be exposed to the implementation, such as the data and work distribution, so it can be measured and managed.

There’s a great deal of work in runtime optimization, ranging from systems that vary runtime parameters such as tile sizes to tune for cache locality, all the way to managed languages that compile the hot routines to native code and even specializing the code for runtime values. At least some of these are ripe for industrialization in the HPC space.

Productivity: In the past, productivity in HPC focused on how close the application could get to the peak performance of the computer. This is still important; no one would argue that we should use these 100-million-dollar mega-computers inefficiently. However, even at that price, the cost of the 1,000+ scientists and engineers that will use the machine approaches or exceeds the cost of the hardware. So, however measured, we want to make productive use of the machine, and make productive use of our time.

Many argue that we want our scientists (astronomers, chemists, physicists, biologists) to be able to directly program these machines with the same productivity as when they use Matlab or Mathematica. I’m going out on a limb here; I don’t see these astronomers polishing their own mirrors for large, multi-million dollar telescopes. I don’t see physicists winding wires around superconducting magnets for the colliders, or biologists constructing scanning, tunneling electron microscopes. Of course, early astronomers really did grind lenses and polish mirrors, but now they specify the design and hire an expert to do that for them. Why should we not expect the same protocol for high end computing? Why should scientists not hire an HPC expert to build and tune the program? Yes, software is fundamentally different than a piece of hardware, but the goals are not just the specification and implementation of an algorithm, but the expression of that algorithm that provides enough performance to change the nature of science that we can perform. If that’s not enough motivation to invest in expert programmers, what is?

So what are the characteristics of a programming method for the coming exascale systems? How different will it be from what we have today? That is the topic of my third, and hopefully final, column on exascale programming.

About the Author

Michael Wolfe has developed compilers for over 30 years in both academia and industry, and is now a senior compiler engineer at The Portland Group, Inc. (www.pgroup.com), a wholly-owned subsidiary of STMicroelectronics, Inc. The opinions stated here are those of the author, and do not represent opinions of The Portland Group, Inc. or STMicroelectronics, Inc.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Scientists Conduct First Quantum Simulation of Atomic Nucleus

May 23, 2018

OAK RIDGE, Tenn., May 23, 2018—Scientists at the Department of Energy’s Oak Ridge National Laboratory are the first to successfully simulate an atomic nucleus using a quantum computer. The results, published in Ph Read more…

By Rachel Harken, ORNL

Pattern Computer – Startup Claims Breakthrough in ‘Pattern Discovery’ Technology

May 23, 2018

If it weren’t for the heavy-hitter technology team behind start-up Pattern Computer, which emerged from stealth today in a live-streamed event from San Francisco, one would be tempted to dismiss its claims of inventing Read more…

By John Russell

Intel, Micro Debut Quad-Level Cell NAND Flash

May 22, 2018

Chipmakers continue to gear designs toward AI and other demanding cloud workloads that take advantage of datacenter flash storage capacity. To that end, memory specialist Micron Technology Inc. began shipping compact sol Read more…

By George Leopold

HPE Extreme Performance Solutions

HPC and AI Convergence is Accelerating New Levels of Intelligence

Data analytics is the most valuable tool in the digital marketplace – so much so that organizations are employing high performance computing (HPC) capabilities to rapidly collect, share, and analyze endless streams of data. Read more…

IBM Accelerated Insights

Mastering the Big Data Challenge in Cognitive Healthcare

Patrick Chain, genomics researcher at Los Alamos National Laboratory, posed a question in a recent blog: What if a nurse could swipe a patient’s saliva and run a quick genetic test to determine if the patient’s sore throat was caused by a cold virus or a bacterial infection? Read more…

Japan Meteorological Agency Takes Delivery of Pair of Crays

May 21, 2018

Cray has supplied two identical Cray XC50 supercomputers to the Japan Meteorological Agency (JMA) in northwestern Tokyo. Boasting more than 18 petaflops combined peak computing capacity, the new systems will extend the a Read more…

By Tiffany Trader

Pattern Computer – Startup Claims Breakthrough in ‘Pattern Discovery’ Technology

May 23, 2018

If it weren’t for the heavy-hitter technology team behind start-up Pattern Computer, which emerged from stealth today in a live-streamed event from San Franci Read more…

By John Russell

Japan Meteorological Agency Takes Delivery of Pair of Crays

May 21, 2018

Cray has supplied two identical Cray XC50 supercomputers to the Japan Meteorological Agency (JMA) in northwestern Tokyo. Boasting more than 18 petaflops combine Read more…

By Tiffany Trader

ASC18: Final Results Revealed & Wrapped Up

May 17, 2018

It was an exciting week at ASC18 in Nanyang, China. The student teams braved extreme heat, extremely difficult applications, and extreme competition in order to cross the cluster competition finish line. The gala awards ceremony took place on Wednesday. The auditorium was packed with student teams, various dignitaries, the media, and other interested parties. So what happened? Read more…

By Dan Olds

Spring Meetings Underscore Quantum Computing’s Rise

May 17, 2018

The month of April 2018 saw four very important and interesting meetings to discuss the state of quantum computing technologies, their potential impacts, and th Read more…

By Alex R. Larzelere

Quantum Network Hub Opens in Japan

May 17, 2018

Following on the launch of its Q Commercial quantum network last December with 12 industrial and academic partners, the official Japanese hub at Keio University is now open to facilitate the exploration of quantum applications important to science and business. The news comes a week after IBM announced that North Carolina State University was the first U.S. university to join its Q Network. Read more…

By Tiffany Trader

Democratizing HPC: OSC Releases Version 1.3 of OnDemand

May 16, 2018

Making HPC resources readily available and easier to use for scientists who may have less HPC expertise is an ongoing challenge. Open OnDemand is a project by t Read more…

By John Russell

PRACE 2017 Annual Report: Exascale Aspirations; Industry Collaboration; HPC Training

May 15, 2018

The Partnership for Advanced Computing in Europe (PRACE) today released its annual report showcasing 2017 activities and providing a glimpse into thinking about Read more…

By John Russell

US Forms AI Brain Trust

May 11, 2018

Amid calls for a U.S. strategy for promoting AI development, the Trump administration is forming a senior-level panel to help coordinate government and industry research efforts. The Select Committee on Artificial Intelligence was announced Thursday (May 10) during a White House summit organized by the Office of Science and Technology Policy (OSTP). Read more…

By George Leopold

MLPerf – Will New Machine Learning Benchmark Help Propel AI Forward?

May 2, 2018

Let the AI benchmarking wars begin. Today, a diverse group from academia and industry – Google, Baidu, Intel, AMD, Harvard, and Stanford among them – releas Read more…

By John Russell

How the Cloud Is Falling Short for HPC

March 15, 2018

The last couple of years have seen cloud computing gradually build some legitimacy within the HPC world, but still the HPC industry lies far behind enterprise I Read more…

By Chris Downing

Russian Nuclear Engineers Caught Cryptomining on Lab Supercomputer

February 12, 2018

Nuclear scientists working at the All-Russian Research Institute of Experimental Physics (RFNC-VNIIEF) have been arrested for using lab supercomputing resources to mine crypto-currency, according to a report in Russia’s Interfax News Agency. Read more…

By Tiffany Trader

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Deep Learning at 15 PFlops Enables Training for Extreme Weather Identification at Scale

March 19, 2018

Petaflop per second deep learning training performance on the NERSC (National Energy Research Scientific Computing Center) Cori supercomputer has given climate Read more…

By Rob Farber

AI Cloud Competition Heats Up: Google’s TPUs, Amazon Building AI Chip

February 12, 2018

Competition in the white hot AI (and public cloud) market pits Google against Amazon this week, with Google offering AI hardware on its cloud platform intended Read more…

By Doug Black

US Plans $1.8 Billion Spend on DOE Exascale Supercomputing

April 11, 2018

On Monday, the United States Department of Energy announced its intention to procure up to three exascale supercomputers at a cost of up to $1.8 billion with th Read more…

By Tiffany Trader

Lenovo Unveils Warm Water Cooled ThinkSystem SD650 in Rampup to LRZ Install

February 22, 2018

This week Lenovo took the wraps off the ThinkSystem SD650 high-density server with third-generation direct water cooling technology developed in tandem with par Read more…

By Tiffany Trader

Leading Solution Providers

HPC and AI – Two Communities Same Future

January 25, 2018

According to Al Gara (Intel Fellow, Data Center Group), high performance computing and artificial intelligence will increasingly intertwine as we transition to Read more…

By Rob Farber

Google Chases Quantum Supremacy with 72-Qubit Processor

March 7, 2018

Google pulled ahead of the pack this week in the race toward "quantum supremacy," with the introduction of a new 72-qubit quantum processor called Bristlecone. Read more…

By Tiffany Trader

HPE Wins $57 Million DoD Supercomputing Contract

February 20, 2018

Hewlett Packard Enterprise (HPE) today revealed details of its massive $57 million HPC contract with the U.S. Department of Defense (DoD). The deal calls for HP Read more…

By Tiffany Trader

CFO Steps down in Executive Shuffle at Supermicro

January 31, 2018

Supermicro yesterday announced senior management shuffling including prominent departures, the completion of an audit linked to its delayed Nasdaq filings, and Read more…

By John Russell

Deep Learning Portends ‘Sea Change’ for Oil and Gas Sector

February 1, 2018

The billowing compute and data demands that spurred the oil and gas industry to be the largest commercial users of high-performance computing are now propelling Read more…

By Tiffany Trader

Nvidia Ups Hardware Game with 16-GPU DGX-2 Server and 18-Port NVSwitch

March 27, 2018

Nvidia unveiled a raft of new products from its annual technology conference in San Jose today, and despite not offering up a new chip architecture, there were still a few surprises in store for HPC hardware aficionados. Read more…

By Tiffany Trader

Hennessy & Patterson: A New Golden Age for Computer Architecture

April 17, 2018

On Monday June 4, 2018, 2017 A.M. Turing Award Winners John L. Hennessy and David A. Patterson will deliver the Turing Lecture at the 45th International Sympo Read more…

By Staff

Part One: Deep Dive into 2018 Trends in Life Sciences HPC

March 1, 2018

Life sciences is an interesting lens through which to see HPC. It is perhaps not an obvious choice, given life sciences’ relative newness as a heavy user of H Read more…

By John Russell

  • arrow
  • Click Here for More Headlines
  • arrow
Share This