Compilers and More: Exascale Programming Requirements

By Michael Wolfe

April 14, 2011

Programming at Exascale, Part 3

In an earlier column, I discussed six levels of parallelism that we’ll have in exascale systems: node, socket, core, vector, instruction, and pipeline levels, and said that to reach exascale performance, we need to take advantage of all these levels, since the final performance is the product of them all. In my most recent column, I argued that to be successful at that, we need to effectively expose, express, and exploit parallelism: expose it in the application and algorithms, express it in the language and program, and exploit it in the generated code and at runtime. Exposing parallelism is mostly a creative task, and thus must be done by humans. Expressing parallelism is where we mostly get sidetracked: what language, what kind of parallelism, how will it work with legacy software? Since parallel programming is all about performance, we need to focus on those aspects that would hinder performance, specifically locality and synchronization. Finally, successfully exploiting parallelism means mapping the parallelism exposed in the application and expressed in the program to the parallelism in the hardware. I discussed five dimensions of flexibility: scalability, dynamic parallelism, composability, load balancing, and productivity. In this column, the last of a three-part series, I’ll give my views on what programming at the exascale level is likely to require, and how we can get there from where we are today. My belief is that it will take some work, but it’s not a wholesale rewrite of 50 years of high performance expertise.

Exascale Programming: What It Won’t Be

What are the characteristics of a programming strategy for the coming exascale computers? It’s easier to say what it isn’t.

It’s not a library. Encapsulation is a well-known, often used, and important technique to building large systems. By design, encapsulation hides information about the implementation of the encapsulated object (data structure, algorithm, service) from the user of that object. Encapsulation will continue to be important for many reasons. But information hiding obscures not just the algorithm and data structures, but performance aspects, such as what kinds of parallelism are used within the encapsulated object and how that interacts with parallelism of the user of that object, or low level information such as how the data is laid out and how that affects locality in an algorithm. In particular, opaque low-level libraries (e.g., MPI for data distribution and message passing) hide too much information from the system, preventing any system-level tuning. That’s not to say a useful system won’t be built using MPI as the transport layer, but MPI or POSIX threads or other low-level libraries should not be directly used in the application.

It’s not a C++ class hierarchy or template library. Here, I’m again going out on a limb; there have been and continue to be many sets of useful C++ class libraries intended to raise the level of application programming. Take the C++ standard template library for vector; the intent of such a template is to allow a user to define a data structure and get the benefit of reusing any routines in the STL or from elsewhere built on the vector template. But you don’t really understand the performance of the vector datatype; that information hiding means you don’t know if accesses to vector V; are efficient or not. Compare that to an array access in a loop, with the corresponding vector access V[i]; the array access can often be optimized down to two instructions: load, and increment the pointer to the next address. Moreover, two-dimensional objects using the vector type (vector>) become even more opaque.

Or take Thrust, an STL-like implementation providing a high-level interface to GPU programming, built on CUDA. You can define two vectors in Thrust as

   using namespace thrust;    device_vector x(1000);    device_vector y(1000); 

Multiplying two such vectors and then accumulating the result can be done as:

   transform( x.begin(), x.end(), y.begin(), z.begin(), multiplies() );    r = reduce( z.begin(), z.end(), 0, plus() );

This is certainly easier (more productive?) to write than the equivalent CUDA C (or CUDA Fortran) code, but it’s still far easier to write the Fortran:

   r = sum( x(:) * y(:) ) 

Moreover, when the constructs are part of the language, the compiler can compose and optimize them together. As mentioned in my last column, in the Fortran case, the compiler can generate code for the multiply then accumulate the result without requiring an intermediate vector result. With the C++ library, the code for the transform method doesn’t know that its result will immediately be accumulated, so the method or (as in this case) the user has to provide a result vector. The only tool the compiler has to optimize class library calls is inlining, and it’s simply not enough to recover the performance lost by the abstraction. There have been some efforts to use run-time code generation, building the expression tree from the method calls, then generating the optimized (and composed) code from the whole expression tree; this was the technology behind Rapidmind, which is now being used in Intel’s Array Building Blocks (ArBB). Such mechanisms are promising, but what we really want is a way to define new data types and describe operations to the compiler in a way that the compiler can reason about them, compose them, reorder them, and so on; currently, the definition is basically in terms of C code, which is not expressive enough. There’s a research project just waiting to happen.

It’s not a domain-specific language. I really like the idea of DSLs, of embedding domain knowledge in the language and using that knowledge when generating and optimizing the code. However, languages, real languages, are big project; DSLs are (by definition) specialized, and hence don’t have a large enough user community to support production, maintenance and continuing development of the language and all the tools needed to support a language. We can’t expect language implementors (like PGI) to take on the development and continuing support of a plethora of languages, any more than we should expect user communities to each design, implement, and then continue to update, enhance, tune and optimize the language implementation with each new processor release from Intel. A possible alternative approach would be to implement a language to support DSLs, supported by a language vendor, including interfacing to debuggers, performance tools, editors, and so on. The various user communities would then be somewhat insulated from the details of a performance-oriented solution, and the vendor would avoid falling into the many-languages trap. There’s another potential research project.

It’s not OpenCL. OpenCL may be a necessary step towards heterogeneous programming, but it’s not the final answer. It’s very low level, “close to the metal”, as even the language designers admit. As with MPI, we may be able to build on OpenCL, but it’s not sufficient.

It’s not a whole new language. New languages have a high barrier to entry; most programmers avoid adopting a new language for fear that it will die, unless the language meets some need better than anything else, or until it has survived along enough to ameliorate the fear. But I think a new language is not called for here. We may benefit from some new features in existing languages, and maybe new ways to make programs in those languages, but most new languages really don’t add semantically much beyond managed memory.

It’s not easy. I’ve argued before that parallel programming is not easy, won’t be, and can’t be made easy. The idea of making parallel programming easy is silly.

It’s not just parallelism. Parallelism is an important aspect, perhaps the dominant aspect, but the key isn’t parallelism, it’s performance. A bad parallel algorithm doesn’t run fast just because it’s parallel. A bad implementation of a good parallel algorithm will also be slow. It’s quite easy to write slow parallel programs; this was the key failure (my opinion) of High Performance Fortran. So our programming mechanism will focus on performance, where parallelism is one aspect (locality and synchronization being two more).

Exascale Programming: What It Is

So what do we want and need when programming at exascale from whatever programming environment we get? Here is my bucket list:

  • It supports all levels of parallelism, from node parallelism down to vector and pipeline parallelism, effectively. Support is a big word here; it has to allow for a programming model that an application developer can use to think about what kinds of parallelism will map well at different levels, that a programmer can use to write a program that can be mapped well at different levels, and that the implementation (compiler and runtime) can use to exploit the parallelism. We have this today, clumsily, with different mechanisms for different levels; a bit more integration would take us a long way.
  • It can map an expression of program parallelism (a parallel loop, say) to different levels of hardware parallelism (across nodes, or to a vector unit) depending on the target. This will make it scalable up and down, from exascale to laptop. There was a great deal of work on the SISAL language to efficiently scalarize an implicitly parallel language, which turned out to be largely the dual of the parallelizing compiler problem. Such work will be part of this parallelism remapping. Remapping node-level parallelism may require changing the data distribution per node; today, this is done at the application level. We should be able to specify what parameters of the program depend on which aspects of the target machine, so the system can do the remapping.
  • It supports the programmer with lots of feedback. Vectorizing compilers have been very successful for over 35 years in delivering good vector performance from sequential loops because the compilers tell the programmer when they are successful, and more importantly, when and why they fail. This is essentially performance feedback. We are in the business of developing high performance applications, and we should be notified when we are using constructs that will restrict our performance. Static feedback and useful dynamic feedback will both be critical.
  • It supports dynamic parallelism, creating parallel tasks and threads when needed. There are many successful and useful implementations of dynamic parallelism, some limited (OpenMP) and some more aggressive (Cilk). Dynamic parallelism is somewhat at odds with locality and synchronization optimization. Using a work-stealing scheduler, an idle worker will steal a work item from the queue of another worker. However, that work item may have been placed on that worker’s queue because that’s where its data is, or because that work item depends on some other work item also assigned to that worker. However, without constructs for dynamic parallelism, we end up micromanaging thread-level parallelism in the constructs we do have.
  • It efficiently composes abstract operations, as I discussed in my previous column; whether these are native to the language, or abstract operations defined by a user or in a library, the implementation must be able to combine them naturally. Perhaps, when we define abstract operations, we need a mechanism to describe how they can compose with others. Many now-standard compiler optimizations fall into composition, such as loop vectorization and loop fusion. We need more investigation about what composing abstract operations means, beyond simply inlining.
  • It is self-balancing and self-tuning. This involves runtime introspection and behavior modification, and means the parameters or data and work distribution must be exposed to the system in order to be modified. Examples include changing the tile sizes for tiled nested loops when optimized for cache locality, or changing the data distributions when the work load is not uniform across the domain. Such behavior modification has been demonstrated in many systems, though not many integrated with the programming language and its implementation.
  • It must be resilient. The big systems are, many believe, going to be in partial failure mode much of the time. This presents challenges for the system manager and programmer. Expecting the entire system to be working, taking checkpoints and restoring from a failure point will not be efficient if failures are the norm. Some of the necessary features must be supported by the hardware (getting data off a node with a failed processor; early failure detection). Other features could be supported by some of the runtime features we develop for other reasons (redistributing data to working nodes; reserving some nodes to serve as online replacements). Such a system can survive and continue beyond many failures.

Most of these points (except for the last) have been researched and implemented in some form already, and could be reproduced with current technology (and enough motivation) in Fortran, C++, or whatever language you want. We have to extend the programming model to expose performance aspects and perhaps resilience aspects, so the user can guide how the system (compiler plus runtime) implements the program. We often get focused on either abstracting away so much that we lose sight of performance (as happened with High Performance Fortran), or we get so tied up with performance that we focus too much on details of each target machine (as happens today with OpenCL and CUDA). We need to let the programmer do the creative parts, and let the system do the mechanical work.

Final Note: This series of columns is an expanded form of the material from the PGI Exhibitor Forum presentation at SC10 in New Orleans. If you were there, you can tell me whether it’s more informative (or entertaining) in written or verbal form.

About the Author

Michael Wolfe has developed compilers for over 30 years in both academia and industry, and is now a senior compiler engineer at The Portland Group, Inc. (www.pgroup.com), a wholly-owned subsidiary of STMicroelectronics, Inc. The opinions stated here are those of the author, and do not represent opinions of The Portland Group, Inc. or STMicroelectronics, Inc.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Why HPC Storage Matters More Now Than Ever: Analyst Q&A

September 17, 2021

With soaring data volumes and insatiable computing driving nearly every facet of economic, social and scientific progress, data storage is seizing the spotlight. Hyperion research analyst and noted storage expert Mark No Read more…

GigaIO Gets $14.7M in Series B Funding to Expand Its Composable Fabric Technology to Customers

September 16, 2021

Just before the COVID-19 pandemic began in March 2020, GigaIO introduced its Universal Composable Fabric technology, which allows enterprises to bring together any HPC and AI resources and integrate them with networking, Read more…

What’s New in HPC Research: Solar Power, ExaWorks, Optane & More

September 16, 2021

In this regular feature, HPCwire highlights newly published research in the high-performance computing community and related domains. From parallel programming to exascale to quantum computing, the details are here. Read more…

Cerebras Brings Its Wafer-Scale Engine AI System to the Cloud

September 16, 2021

Five months ago, when Cerebras Systems debuted its second-generation wafer-scale silicon system (CS-2), co-founder and CEO Andrew Feldman hinted of the company’s coming cloud plans, and now those plans have come to fruition. Today, Cerebras and Cirrascale Cloud Services are launching... Read more…

AI Hardware Summit: Panel on Memory Looks Forward

September 15, 2021

What will system memory look like in five years? Good question. While Monday's panel, Designing AI Super-Chips at the Speed of Memory, at the AI Hardware Summit, tackled several topics, the panelists also took a brief glimpse into the future. Unlike compute, storage and networking, which... Read more…

AWS Solution Channel

Supporting Climate Model Simulations to Accelerate Climate Science

The Amazon Sustainability Data Initiative (ASDI), AWS is donating cloud resources, technical support, and access to scalable infrastructure and fast networking providing high performance computing (HPC) solutions to support simulations of near-term climate using the National Center for Atmospheric Research (NCAR) Community Earth System Model Version 2 (CESM2) and its Whole Atmosphere Community Climate Model (WACCM). Read more…

ECMWF Opens Bologna Datacenter in Preparation for Atos Supercomputer

September 14, 2021

In January 2020, the European Centre for Medium-Range Weather Forecasts (ECMWF) – a juggernaut in the weather forecasting scene – signed a four-year, $89-million contract with European tech firm Atos to quintuple its supercomputing capacity. With the deal approaching the two-year mark, ECMWF... Read more…

Why HPC Storage Matters More Now Than Ever: Analyst Q&A

September 17, 2021

With soaring data volumes and insatiable computing driving nearly every facet of economic, social and scientific progress, data storage is seizing the spotlight Read more…

Cerebras Brings Its Wafer-Scale Engine AI System to the Cloud

September 16, 2021

Five months ago, when Cerebras Systems debuted its second-generation wafer-scale silicon system (CS-2), co-founder and CEO Andrew Feldman hinted of the company’s coming cloud plans, and now those plans have come to fruition. Today, Cerebras and Cirrascale Cloud Services are launching... Read more…

AI Hardware Summit: Panel on Memory Looks Forward

September 15, 2021

What will system memory look like in five years? Good question. While Monday's panel, Designing AI Super-Chips at the Speed of Memory, at the AI Hardware Summit, tackled several topics, the panelists also took a brief glimpse into the future. Unlike compute, storage and networking, which... Read more…

ECMWF Opens Bologna Datacenter in Preparation for Atos Supercomputer

September 14, 2021

In January 2020, the European Centre for Medium-Range Weather Forecasts (ECMWF) – a juggernaut in the weather forecasting scene – signed a four-year, $89-million contract with European tech firm Atos to quintuple its supercomputing capacity. With the deal approaching the two-year mark, ECMWF... Read more…

Quantum Computer Market Headed to $830M in 2024

September 13, 2021

What is one to make of the quantum computing market? Energized (lots of funding) but still chaotic and advancing in unpredictable ways (e.g. competing qubit tec Read more…

Amazon, NCAR, SilverLining Team for Unprecedented Cloud Climate Simulations

September 10, 2021

Earth’s climate is, to put it mildly, not in a good place. In the wake of a damning report from the Intergovernmental Panel on Climate Change (IPCC), scientis Read more…

After Roadblocks and Renewals, EuroHPC Targets a Bigger, Quantum Future

September 9, 2021

The EuroHPC Joint Undertaking (JU) was formalized in 2018, beginning a new era of European supercomputing that began to bear fruit this year with the launch of several of the first EuroHPC systems. The undertaking, however, has not been without its speed bumps, and the Union faces an uphill... Read more…

How Argonne Is Preparing for Exascale in 2022

September 8, 2021

Additional details came to light on Argonne National Laboratory’s preparation for the 2022 Aurora exascale-class supercomputer, during the HPC User Forum, held virtually this week on account of pandemic. Exascale Computing Project director Doug Kothe reviewed some of the 'early exascale hardware' at Argonne, Oak Ridge and NERSC (Perlmutter), while Ti Leggett, Deputy Project Director & Deputy Director... Read more…

Ahead of ‘Dojo,’ Tesla Reveals Its Massive Precursor Supercomputer

June 22, 2021

In spring 2019, Tesla made cryptic reference to a project called Dojo, a “super-powerful training computer” for video data processing. Then, in summer 2020, Tesla CEO Elon Musk tweeted: “Tesla is developing a [neural network] training computer called Dojo to process truly vast amounts of video data. It’s a beast! … A truly useful exaflop at de facto FP32.” Read more…

Berkeley Lab Debuts Perlmutter, World’s Fastest AI Supercomputer

May 27, 2021

A ribbon-cutting ceremony held virtually at Berkeley Lab's National Energy Research Scientific Computing Center (NERSC) today marked the official launch of Perlmutter – aka NERSC-9 – the GPU-accelerated supercomputer built by HPE in partnership with Nvidia and AMD. Read more…

Google Launches TPU v4 AI Chips

May 20, 2021

Google CEO Sundar Pichai spoke for only one minute and 42 seconds about the company’s latest TPU v4 Tensor Processing Units during his keynote at the Google I Read more…

Esperanto, Silicon in Hand, Champions the Efficiency of Its 1,092-Core RISC-V Chip

August 27, 2021

Esperanto Technologies made waves last December when it announced ET-SoC-1, a new RISC-V-based chip aimed at machine learning that packed nearly 1,100 cores onto a package small enough to fit six times over on a single PCIe card. Now, Esperanto is back, silicon in-hand and taking aim... Read more…

Enter Dojo: Tesla Reveals Design for Modular Supercomputer & D1 Chip

August 20, 2021

Two months ago, Tesla revealed a massive GPU cluster that it said was “roughly the number five supercomputer in the world,” and which was just a precursor to Tesla’s real supercomputing moonshot: the long-rumored, little-detailed Dojo system. “We’ve been scaling our neural network training compute dramatically over the last few years,” said Milan Kovac, Tesla’s director of autopilot engineering. Read more…

CentOS Replacement Rocky Linux Is Now in GA and Under Independent Control

June 21, 2021

The Rocky Enterprise Software Foundation (RESF) is announcing the general availability of Rocky Linux, release 8.4, designed as a drop-in replacement for the soon-to-be discontinued CentOS. The GA release is launching six-and-a-half months after Red Hat deprecated its support for the widely popular, free CentOS server operating system. The Rocky Linux development effort... Read more…

Intel Completes LLVM Adoption; Will End Updates to Classic C/C++ Compilers in Future

August 10, 2021

Intel reported in a blog this week that its adoption of the open source LLVM architecture for Intel’s C/C++ compiler is complete. The transition is part of In Read more…

Iran Gains HPC Capabilities with Launch of ‘Simorgh’ Supercomputer

May 18, 2021

Iran is said to be developing domestic supercomputing technology to advance the processing of scientific, economic, political and military data, and to strengthen the nation’s position in the age of AI and big data. On Sunday, Iran unveiled the Simorgh supercomputer, which will deliver.... Read more…

Leading Solution Providers

Contributors

AMD-Xilinx Deal Gains UK, EU Approvals — China’s Decision Still Pending

July 1, 2021

AMD’s planned acquisition of FPGA maker Xilinx is now in the hands of Chinese regulators after needed antitrust approvals for the $35 billion deal were receiv Read more…

Hot Chips: Here Come the DPUs and IPUs from Arm, Nvidia and Intel

August 25, 2021

The emergence of data processing units (DPU) and infrastructure processing units (IPU) as potentially important pieces in cloud and datacenter architectures was Read more…

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from Julialang.org. I Read more…

HPE Wins $2B GreenLake HPC-as-a-Service Deal with NSA

September 1, 2021

In the heated, oft-contentious, government IT space, HPE has won a massive $2 billion contract to provide HPC and AI services to the United States’ National Security Agency (NSA). Following on the heels of the now-canceled $10 billion JEDI contract (reissued as JWCC) and a $10 billion... Read more…

Intel Launches 10nm ‘Ice Lake’ Datacenter CPU with Up to 40 Cores

April 6, 2021

The wait is over. Today Intel officially launched its 10nm datacenter CPU, the third-generation Intel Xeon Scalable processor, codenamed Ice Lake. With up to 40 Read more…

Quantum Roundup: IBM, Rigetti, Phasecraft, Oxford QC, China, and More

July 13, 2021

IBM yesterday announced a proof for a quantum ML algorithm. A week ago, it unveiled a new topology for its quantum processors. Last Friday, the Technical Univer Read more…

Frontier to Meet 20MW Exascale Power Target Set by DARPA in 2008

July 14, 2021

After more than a decade of planning, the United States’ first exascale computer, Frontier, is set to arrive at Oak Ridge National Laboratory (ORNL) later this year. Crossing this “1,000x” horizon required overcoming four major challenges: power demand, reliability, extreme parallelism and data movement. Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire