Compilers and More: Expose, Express, Exploit

By Michael Wolfe

March 28, 2011

Part 2: Programming at Exascale

In my previous column, I introduced six levels of parallelism that we’ll have in exascale systems:

  • Node level
  • Socket level
  • Core level
  • Vector level
  • Instruction level
  • Pipeline level

As we move towards exascale, we want to take advantage of all of these. We need to expose parallelism at all the levels, either explicitly in the program, or implicitly within the compiler. We need to express this parallelism, in a language and in the generated code. And we need to exploit the parallelism, efficiently and effectively, at runtime on the target machine. Performance is important; in fact, performance is key. The only reason we are looking at parallelism is for higher performance. This is a point I have made in the past, and I’ll say it again: the only reason to program in parallel is for higher performance. (Someone at SC10 responded that another reason to use parallelism is for redundancy, for error detection. I should have replied that redundancy is good, but you do the redundant computations in parallel to get the benefits of redundant computation without performance degradation.)

Exposing Parallelism

Someone has to find or create the parallelism in the program. At tera and petascale, we’ve mostly focused on data parallelism: large datasets, where the program can operate on separate data in parallel. At exascale, we’re likely to want even more parallelism, such as running coupled models in parallel. We’ll do the structural mechanics model concurrently with the heat transfer and deformation physics models. Since these models are coupled, it’s much easier to separate the problems and iterate through them one at a time; it will be a challenge to structure the code so they can run in parallel with each other. Parallelism at this high a level has to be exposed in the application design.

Within each algorithm, there is additional parallelism. This is where we get most of the parallel execution today, from individual algorithms. Parallelism at this level has finer granularity, and often is naturally dynamic. Some parallelism we get for free, meaning the compiler can find vector and instruction-level parallelism without changing the program. To be honest, this is a misrepresentation. The compiler can only find parallelism that already exists in the program. There are always ways to write the program to defeat the compiler; this isn’t a failing of the compiler, it’s compiler abuse.

Exposing parallelism, at any level, is a creative process. You choose how to organize your data such that it can be processed in parallel, or not. You choose the algorithms, solvers, conditioners, etc., that can use more parallelism, or that use less. You make these decisions in your application design. You might make different decisions, use different algorithms, depending on the target machine (laptop vs. cluster vs. exascale); this makes your job more challenging. However, since these decisions affect not only the parallelism and performance, but the accuracy and quality of the results, they have to be made by an expert, by a human.

Expressing Parallelism

Most of the discussion about parallel programming falls into this heading: how do I write my parallel program? Will I use MPI? Do I use some programming framework which is itself built on top of MPI? Do I use OpenMP directives? Do I use Cilk language extensions, or Unified Parallel C? Should I write in a consistent sequential language and use a compiler or other tool to find the parallelism? Should I use a truly higher order language, like SISAL or Haskell? Can I design a domain-specific language to make the programming more natural? Maybe I can build a C++ class library that will support my data structures directly?

There is a big step between designing an algorithm and expressing that algorithm in some language. Writing a parallel message-passing solver for large systems (such as the High Performance Linpack benchmark) is much more work that simply calling the LAPACK DGESV routine. You have to worry about data distribution, communication, and load balancing. Your answers to these will affect what algorithms you use locally and globally, and may be affected by your tolerance for numerical accuracy.

Since performance is key, we should focus on those aspects of the program that lead to high performance, or that might degrade performance. Given a parallel program, the performance keys are high locality and little synchronization. This has different meanings at different levels of parallelism. At the lowest levels of parallelism, locality is implemented in registers; the compiler manages register allocation to minimize memory accesses. We expect synchronization at the lower levels to be handled in hardware, almost for free, but we want the compiler to find enough instruction-level parallelism to exploit the functional units efficiently.

Between cores or sockets, locality is implemented in cache memory. We want the application to be organized to take advantage of the improved performance of caches for spatially and temporally local memory references. Modern cache memory hierarchies are large (12MB is not uncommon), but large parallel datasets can be huge (6GB is often considered small). Cache memory locality is typically optimized by loop tiling, usually manually in the program, but sometimes by the compiler. Cache memory locality for parallel cores can be hard to manage since some levels of the cache are shared between cores. This implies that we want those cores to share the data at that level in the cache, to avoid the cores interfering with each other. Cores on processors in separate sockets don’t share cache, so we’d like those cores to not share data, at least not very often, so the caches don’t thrash. Synchronization between cores that share memory is typically done through the memory, using locks or semaphores. It can be challenging to get these primitives correct and inserted in a way that gives high performance. Transactional memory has been proposed as a mechanism to simplify parallel programming. It treats shared data structure updates as atomic transactions, the same way a database implementation treats database updates. An implementation allows updates to separate areas of the shared data structure to proceed speculatively in parallel without explicit locks, although under the hood, as with databases, the transaction commit does use locks and incurs additional cost to verify that the transaction will behave as if it were truly atomic.

Any Single Program-Multiple Data (SPMD) program, whether MPI, Unified Parallel C, using coarrays in Fortran, or other method, requires the user to manage data locality explicitly in the program. In an MPI program, the programmer specifies what data to allocate on each node (MPI rank), and when to send or receive data from other nodes. In UPC or coarrays in Fortran, the programmer uses an additional index (or indices), the UPC shared array dimension or coarray image codimensions, to determine on which thread or image (i.e., node) the data resides. Using MPI, the program uses explicit messages; messages can have additional costs, such as implicit data copies and buffers on the sender or receiver side. However, messages also carry synchronization information, “the data is ready, and here’s the data,” along with the data. With an implicit or global address space model, the program must include separate synchronization primitives, as with a shared memory model. Neither is perfect, and both can be prone to errors.

If accelerators become common, then programs have to explicitly or implicitly be partitioned into CPU and accelerator parts. Current accelerators, such as GPUs, use a separate address space and physical memory. I hear predictions that on-chip accelerators, such as promised by the AMD Fusion devices, will solve this problem. I doubt this. Accelerators are designed for high-bandwidth memory access to support large data structures, whereas CPUs are designed for low latency operations. The CPU is supported by a deep cache hierarchy, which won’t help the accelerator, and current accelerators use a different memory implementation than a CPU. Today’s AMD Llano and Intel’s Sandy Bridge combine a relatively low-performance GPU on the CPU chip, designed to replace the integrated graphics chip that appears on many motherboards. These GPUs share the physical memory with the CPU, as those integrated graphics chips do, though not the same virtual address space. Such a solution incurs a performance penalty for not having dedicated graphics memory. Another approach would be to integrate the virtual address space of CPU and accelerator, while still maintaining the separate latency-oriented memory structure for data accessed from the CPU and bandwidth-oriented memory for accelerator data. Such an approach is used on Convey hybrid computers, with hardware to manage coherence between the CPU and accelerator, though again with performance penalties when CPU or accelerator accesses the other memory. The ultimate goal of full performance, coherent memory access across CPU and accelerator will be difficult to achieve.

The job of expressing parallelism is much more difficult today than it should be. This is largely because we have to include locality (data distribution) and synchronization (messages or explicit synchronization), and we have to express different levels of parallelism explicitly. If we want to target a multi-node, multi-core, accelerator based system, we might need to express message passing between the nodes, shared memory parallelism across cores within a node concurrently with accelerator parallelism on that node, and still leave enough vector parallelism to get the peak performance on each core. While some of this task is creative, much of it is mechanical. Choosing whether to distribute data by rows, columns, or panels can be a creative task, much as choosing a sparse matrix layout is creative. Inserting message primitives or optimizing synchronization placement is largely mechanical, and our programming mechanism should be able to handle this.

Exploiting Parallelism

The final step is to take the parallelism we’ve exposed and expressed and to map it to the target machine. With current MPI programs, this is a simple mechanical task, since all the work was done by the programmer. We should demand more from our implementation. We should want more flexibility across many dimensions, including scalability, dynamic parallelism, composability, load balancing, as well as productivity. Let’s take each in turn.

Scalability: The usual discussion of scalability is to be able to write programs that scale up to massive amounts of parallelism. In the exascale world, we need to have the right algorithms with enough parallelism to give us that level of performance; making those choices is a creative task, and we can’t expect to automate that. However, we should demand that the same program can be scaled down to run efficiently on our (smaller) terascale systems, our clusters, our workstations, and even our laptops. Automatically scaling down should be easier than automatically scaling up; in fact, it should be mostly mechanical, choosing which levels of parallelism to scale back or scalarize entirely. If the parallelism is expressed opaquely in the program (as with an MPI library), such decisions must be made by the programmer; we should design better programming strategies.

Dynamic parallelism: Most of the current large-scale parallel programming models are static: MPI mostly requires the processes to be created at program startup, though MPI-2 does add some weak support for dynamic process creation. Coarray Fortran and UPC similarly start a static number of images or threads. High Performance Fortran suffered from the same weakness. Shared memory models are often more dynamic; OpenMP can dynamically create nested parallel regions, and dynamically create tasks within each region, though the number of actual threads for each region is fixed. Cilk allows spawning of parallel strands, though the parallelism is exploited at runtime by some number of threads or processors managed by the runtime. The more recent High Productivity Programming Languages X10, Chapel and Fortress have more support for dynamic parallelism, and some traction among small groups of users, so perhaps there’s hope.

Composability: However I express my basic operations, the implementation must be able to efficiently compose these for the target processor. For instance, if I should write in Fortran:

    r = sum( x(:) * y(:) )

the definition is that the array product x(:)*y(:) be computed and stored in a temporary array, possibly dynamically allocated, then the elements of the temporary array are summed. However, the implementation shouldn’t have to create the temporary array; if this is being executed in scalar mode, the implementation should be just as efficient as the corresponding loop:

    r = 0     do i = . . .      r = r + x(i)*y(i)     enddo

If this is executed in parallel, each thread or process should be able to accumulate its partial sum efficiently in scalar mode, then the partial sums combined appropriately, with only one temporary scalar per thread. Some approaches to HPC are weak with respect to composability, as I’ll point out in the next column.

Load Balancing: Researchers in the past have developed implementations of parallel systems that monitor the performance across the cores and nodes, and redistribute the work and even the data to balance the load and improve the performance. With low-level parallel programming, work and data distribution is opaque to the runtime system, but there’s no reason we can’t change that. This requires certain characteristics of the program to be exposed to the implementation, such as the data and work distribution, so it can be measured and managed.

There’s a great deal of work in runtime optimization, ranging from systems that vary runtime parameters such as tile sizes to tune for cache locality, all the way to managed languages that compile the hot routines to native code and even specializing the code for runtime values. At least some of these are ripe for industrialization in the HPC space.

Productivity: In the past, productivity in HPC focused on how close the application could get to the peak performance of the computer. This is still important; no one would argue that we should use these 100-million-dollar mega-computers inefficiently. However, even at that price, the cost of the 1,000+ scientists and engineers that will use the machine approaches or exceeds the cost of the hardware. So, however measured, we want to make productive use of the machine, and make productive use of our time.

Many argue that we want our scientists (astronomers, chemists, physicists, biologists) to be able to directly program these machines with the same productivity as when they use Matlab or Mathematica. I’m going out on a limb here; I don’t see these astronomers polishing their own mirrors for large, multi-million dollar telescopes. I don’t see physicists winding wires around superconducting magnets for the colliders, or biologists constructing scanning, tunneling electron microscopes. Of course, early astronomers really did grind lenses and polish mirrors, but now they specify the design and hire an expert to do that for them. Why should we not expect the same protocol for high end computing? Why should scientists not hire an HPC expert to build and tune the program? Yes, software is fundamentally different than a piece of hardware, but the goals are not just the specification and implementation of an algorithm, but the expression of that algorithm that provides enough performance to change the nature of science that we can perform. If that’s not enough motivation to invest in expert programmers, what is?

So what are the characteristics of a programming method for the coming exascale systems? How different will it be from what we have today? That is the topic of my third, and hopefully final, column on exascale programming.

About the Author

Michael Wolfe has developed compilers for over 30 years in both academia and industry, and is now a senior compiler engineer at The Portland Group, Inc. (www.pgroup.com), a wholly-owned subsidiary of STMicroelectronics, Inc. The opinions stated here are those of the author, and do not represent opinions of The Portland Group, Inc. or STMicroelectronics, Inc.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

SC21 Was Unlike Any Other — Was That a Good Thing?

December 3, 2021

For a long time, the promised in-person SC21 seemed like an impossible fever dream, the assurances of a prominent physical component persisting across years of canceled conferences, including two virtual ISCs and the virtual SC20. With the advent of the Delta variant, Covid surges in St. Louis and contention over vaccine requirements... Read more…

The Green500’s Crystal Anniversary Sees MN-3 Crystallize Its Winning Streak

December 2, 2021

“This is the 30th Green500,” said Wu Feng, custodian of the Green500 list, at the list’s SC21 birds-of-a-feather session. “You could say 15 years of Green500, which makes it, I guess, the crystal anniversary.” Indeed, HPCwire marked the 15th anniversary of the Green500 – which ranks supercomputers by flops-per-watt, rather than just by flops – earlier this year with... Read more…

AWS Arm-based Graviton3 Instances Now in Preview

December 1, 2021

Three years after unveiling the first generation of its AWS Graviton chip-powered instances in 2018, Amazon Web Services announced that the third generation of the processors – the AWS Graviton3 – will power all-new Amazon Elastic Compute 2 (EC2) C7g instances that are now available in preview. Debuting at the AWS re:Invent 2021... Read more…

Nvidia Dominates Latest MLPerf Results but Competitors Start Speaking Up

December 1, 2021

MLCommons today released its fifth round of MLPerf training benchmark results with Nvidia GPUs again dominating. That said, a few other AI accelerator companies participated and, one of them, Graphcore, even held a separ Read more…

HPC Career Notes: December 2021 Edition

December 1, 2021

In this monthly feature, we’ll keep you up-to-date on the latest career developments for individuals in the high-performance computing community. Whether it’s a promotion, new company hire, or even an accolade, we’ Read more…

AWS Solution Channel

Running a 3.2M vCPU HPC Workload on AWS with YellowDog

Historically, advances in fields such as meteorology, healthcare, and engineering, were achieved through large investments in on-premises computing infrastructure. Upfront capital investment and operational complexity have been the accepted norm of large-scale HPC research. Read more…

At SC21, Experts Ask: Can Fast HPC Be Green?

November 30, 2021

HPC is entering a new era: exascale is (somewhat) officially here, but Moore’s law is ending. Power consumption and other sustainability concerns loom over the enormous systems and chips of this new epoch, for both cost and compliance reasons. Reconciling the need to continue the supercomputer scale-up while reducing HPC’s environmental impacts... Read more…

SC21 Was Unlike Any Other — Was That a Good Thing?

December 3, 2021

For a long time, the promised in-person SC21 seemed like an impossible fever dream, the assurances of a prominent physical component persisting across years of canceled conferences, including two virtual ISCs and the virtual SC20. With the advent of the Delta variant, Covid surges in St. Louis and contention over vaccine requirements... Read more…

The Green500’s Crystal Anniversary Sees MN-3 Crystallize Its Winning Streak

December 2, 2021

“This is the 30th Green500,” said Wu Feng, custodian of the Green500 list, at the list’s SC21 birds-of-a-feather session. “You could say 15 years of Green500, which makes it, I guess, the crystal anniversary.” Indeed, HPCwire marked the 15th anniversary of the Green500 – which ranks supercomputers by flops-per-watt, rather than just by flops – earlier this year with... Read more…

Nvidia Dominates Latest MLPerf Results but Competitors Start Speaking Up

December 1, 2021

MLCommons today released its fifth round of MLPerf training benchmark results with Nvidia GPUs again dominating. That said, a few other AI accelerator companies Read more…

At SC21, Experts Ask: Can Fast HPC Be Green?

November 30, 2021

HPC is entering a new era: exascale is (somewhat) officially here, but Moore’s law is ending. Power consumption and other sustainability concerns loom over the enormous systems and chips of this new epoch, for both cost and compliance reasons. Reconciling the need to continue the supercomputer scale-up while reducing HPC’s environmental impacts... Read more…

Raja Koduri and Satoshi Matsuoka Discuss the Future of HPC at SC21

November 29, 2021

HPCwire's Managing Editor sits down with Intel's Raja Koduri and Riken's Satoshi Matsuoka in St. Louis for an off-the-cuff conversation about their SC21 experience, what comes after exascale and why they are collaborating. Koduri, senior vice president and general manager of Intel's accelerated computing systems and graphics (AXG) group, leads the team... Read more…

Jack Dongarra on SC21, the Top500 and His Retirement Plans

November 29, 2021

HPCwire's Managing Editor sits down with Jack Dongarra, Top500 co-founder and Distinguished Professor at the University of Tennessee, during SC21 in St. Louis to discuss the 2021 Top500 list, the outlook for global exascale computing, and what exactly is going on in that Viking helmet photo. Read more…

SC21: Larry Smarr on The Rise of Supernetwork Data Intensive Computing

November 26, 2021

Larry Smarr, founding director of Calit2 (now Distinguished Professor Emeritus at the University of California San Diego) and the first director of NCSA, is one of the seminal figures in the U.S. supercomputing community. What began as a personal drive, shared by others, to spur the creation of supercomputers in the U.S. for scientific use, later expanded into a... Read more…

Three Chinese Exascale Systems Detailed at SC21: Two Operational and One Delayed

November 24, 2021

Details about two previously rumored Chinese exascale systems came to light during last week’s SC21 proceedings. Asked about these systems during the Top500 media briefing on Monday, Nov. 15, list author and co-founder Jack Dongarra indicated he was aware of some very impressive results, but withheld comment when asked directly if he had... Read more…

IonQ Is First Quantum Startup to Go Public; Will It be First to Deliver Profits?

November 3, 2021

On October 1 of this year, IonQ became the first pure-play quantum computing start-up to go public. At this writing, the stock (NYSE: IONQ) was around $15 and its market capitalization was roughly $2.89 billion. Co-founder and chief scientist Chris Monroe says it was fun to have a few of the company’s roughly 100 employees travel to New York to ring the opening bell of the New York Stock... Read more…

Enter Dojo: Tesla Reveals Design for Modular Supercomputer & D1 Chip

August 20, 2021

Two months ago, Tesla revealed a massive GPU cluster that it said was “roughly the number five supercomputer in the world,” and which was just a precursor to Tesla’s real supercomputing moonshot: the long-rumored, little-detailed Dojo system. Read more…

Esperanto, Silicon in Hand, Champions the Efficiency of Its 1,092-Core RISC-V Chip

August 27, 2021

Esperanto Technologies made waves last December when it announced ET-SoC-1, a new RISC-V-based chip aimed at machine learning that packed nearly 1,100 cores onto a package small enough to fit six times over on a single PCIe card. Now, Esperanto is back, silicon in-hand and taking aim... Read more…

US Closes in on Exascale: Frontier Installation Is Underway

September 29, 2021

At the Advanced Scientific Computing Advisory Committee (ASCAC) meeting, held by Zoom this week (Sept. 29-30), it was revealed that the Frontier supercomputer is currently being installed at Oak Ridge National Laboratory in Oak Ridge, Tenn. The staff at the Oak Ridge Leadership... Read more…

AMD Launches Milan-X CPU with 3D V-Cache and Multichip Instinct MI200 GPU

November 8, 2021

At a virtual event this morning, AMD CEO Lisa Su unveiled the company’s latest and much-anticipated server products: the new Milan-X CPU, which leverages AMD’s new 3D V-Cache technology; and its new Instinct MI200 GPU, which provides up to 220 compute units across two Infinity Fabric-connected dies, delivering an astounding 47.9 peak double-precision teraflops. “We're in a high-performance computing megacycle, driven by the growing need to deploy additional compute performance... Read more…

Intel Reorgs HPC Group, Creates Two ‘Super Compute’ Groups

October 15, 2021

Following on changes made in June that moved Intel’s HPC unit out of the Data Platform Group and into the newly created Accelerated Computing Systems and Graphics (AXG) business unit, led by Raja Koduri, Intel is making further updates to the HPC group and announcing... Read more…

Intel Completes LLVM Adoption; Will End Updates to Classic C/C++ Compilers in Future

August 10, 2021

Intel reported in a blog this week that its adoption of the open source LLVM architecture for Intel’s C/C++ compiler is complete. The transition is part of In Read more…

Killer Instinct: AMD’s Multi-Chip MI200 GPU Readies for a Major Global Debut

October 21, 2021

AMD’s next-generation supercomputer GPU is on its way – and by all appearances, it’s about to make a name for itself. The AMD Radeon Instinct MI200 GPU (a successor to the MI100) will, over the next year, begin to power three massive systems on three continents: the United States’ exascale Frontier system; the European Union’s pre-exascale LUMI system; and Australia’s petascale Setonix system. Read more…

Leading Solution Providers

Contributors

Hot Chips: Here Come the DPUs and IPUs from Arm, Nvidia and Intel

August 25, 2021

The emergence of data processing units (DPU) and infrastructure processing units (IPU) as potentially important pieces in cloud and datacenter architectures was Read more…

D-Wave Embraces Gate-Based Quantum Computing; Charts Path Forward

October 21, 2021

Earlier this month D-Wave Systems, the quantum computing pioneer that has long championed quantum annealing-based quantum computing (and sometimes taken heat fo Read more…

HPE Wins $2B GreenLake HPC-as-a-Service Deal with NSA

September 1, 2021

In the heated, oft-contentious, government IT space, HPE has won a massive $2 billion contract to provide HPC and AI services to the United States’ National Security Agency (NSA). Following on the heels of the now-canceled $10 billion JEDI contract (reissued as JWCC) and a $10 billion... Read more…

The Latest MLPerf Inference Results: Nvidia GPUs Hold Sway but Here Come CPUs and Intel

September 22, 2021

The latest round of MLPerf inference benchmark (v 1.1) results was released today and Nvidia again dominated, sweeping the top spots in the closed (apples-to-ap Read more…

Ahead of ‘Dojo,’ Tesla Reveals Its Massive Precursor Supercomputer

June 22, 2021

In spring 2019, Tesla made cryptic reference to a project called Dojo, a “super-powerful training computer” for video data processing. Then, in summer 2020, Tesla CEO Elon Musk tweeted: “Tesla is developing a [neural network] training computer... Read more…

Three Chinese Exascale Systems Detailed at SC21: Two Operational and One Delayed

November 24, 2021

Details about two previously rumored Chinese exascale systems came to light during last week’s SC21 proceedings. Asked about these systems during the Top500 media briefing on Monday, Nov. 15, list author and co-founder Jack Dongarra indicated he was aware of some very impressive results, but withheld comment when asked directly if he had... Read more…

2021 Gordon Bell Prize Goes to Exascale-Powered Quantum Supremacy Challenge

November 18, 2021

Today at the hybrid virtual/in-person SC21 conference, the organizers announced the winners of the 2021 ACM Gordon Bell Prize: a team of Chinese researchers leveraging the new exascale Sunway system to simulate quantum circuits. The Gordon Bell Prize, which comes with an award of $10,000 courtesy of HPC pioneer Gordon Bell, is awarded annually... Read more…

Quantum Computer Market Headed to $830M in 2024

September 13, 2021

What is one to make of the quantum computing market? Energized (lots of funding) but still chaotic and advancing in unpredictable ways (e.g. competing qubit tec Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire