An HPC Programming Model for the Exascale Age

By Christian Simmendinger, T-Systems Solutions for Research and Daniel Grünewald, Fraunhofer ITWM, CC-HPC

June 26, 2012

As the supercomputing faithful prepare for exascale computing, there is a great deal of talk about moving beyond the two-decades-old MPI programming model . The HPC programmers of tomorrow are going to have to write codes that are able to deal with systems hundreds of times larger than the top supercomputers of today, and the general feeling is that MPI, by itself, will not make that transition gracefully. One of the alternatives being offered is a PGAS model known as GASPI, which was the subject of an extended session at last week’s International Supercomputing Conference.

GASPI, which stands for Global Address Space Programming Interface, is, as the name suggests, a partitioned global address space (PGAS) API. The GASPI standard is focused on three key objectives: scalability, flexibility and fault tolerance. It follows a single program multiple data (SPMD) approach and offers a small, yet powerful API composed of synchronization primitives, synchronous and asynchronous collectives, fine grained control over one-sided read and write communication primitives, global atomics, passive receives, communication groups and communication queues.

Essentially it uses one-sided RDMA-driven communication in a PGAS environment. As such, GASPI aims to initiate a paradigm shift from bulk-synchronous two-sided communication patterns towards an asynchronous communication and execution model.

With today’s ever increasing number of processes, a transition from bulk-synchronous communication towards an asynchronous programming model seems to be inevitable. Elapsed time for bulk-synchronous communication potentially scales with the logarithm of the number of processes, whereas the work assigned to a single process potentially scales with a factor of 1/(number of processes).

Hence, the scalability of bulk-synchronous communication patterns appears to be limited at best. Despite recent efforts to support true asynchronous communication, the message passing standard of MPI to a large extent still focuses on two-sided semantics and bulk-synchronous communication.

At the same time, fault tolerance also becomes a larger issue as machines expand in size. On systems with large number of processes, all non-local communication should be prepared for a potential failure of one of the communication partners. In GASPI this is accomplished by providing a timeout value as an argument to all non-local communication calls and the possibility to check for the state of each of the communication partners. The model also allows for the dynamic substitution of a failed process.

GASPI does not enforce a specific memory model, like, for example, the symmetric distributed memory management of OpenSHMEM. Rather GASPI offers PGAS in the form of configurable RDMA pinned memory segments. Since an application can request several segments in GASPI symmetric, asymmetric or stack based memory management models can readily coexist.

With PGAS, every thread can asynchronously read and write the entire global memory of an application. On modern machines with RDMA engines, an asynchronous PGAS programming model appears as a natural extension and abstraction of available hardware functionality. For systems with DMA engines (such as tile architectures), this also holds true for a node-local level.

While the GASPI API readily can support the various communication patterns of MPI by means of an add-on library, the reverse is not true. GASPI for example supports RDMA access to arbitrarily distributed data, which allows the programmer a direct RDMA write access from a local send halo of an unstructured mesh into the corresponding remote receive halo.

The GASPI API has been designed to coexist with MPI and hence in principle provides the possibility to complement MPI with a partitioned global address space. We note however, that while such an approach provides an opportunity for increased scalability, fault–tolerant execution will not be possible due to the corresponding limitations of MPI.

GASPI inherits much of its design from the Global address space Programming Interface (GPI), which was developed in 2005 at the Competence Center for High Performance Computing (CC-HPC) at Fraunhofer ITWM. GPI is implemented as a low-latency communication library and is designed for scalable, real-time parallel applications running on cluster systems. It provides a PGAS API and includes communication primitives, environment run-time checks and synchronization primitives such as fast barriers or global atomic counters.

GPI communication is asynchronous, one-sided and, most importantly, does not interfere with the computation on the CPU. Minimal communication overhead can be realized by overlapping communication and computation. GPI also provides a simple, run-time system to handle large data sets, as well as dynamic and irregular applications that are I/O- and compute-intensive. As of today, there are production-quality implementations for x86 and IBM Cell/B.E architectures.

GPI has been used to implement and optimize CC-HPC industry applications like the Generalized Radon Transform (GRT) method in seismic imaging or the seismic work flow and visualization suite PSPRO. Today, GPI is installed on Tier 0 supercomputer sites in Europe, including the HLRS in Stuttgart and the Jülich Supercomputing Centre.

The GPI library has yielded some promising results in a number of situations. In particular, GPI outperforms MPI in significant low-level benchmarks. For process to process communication, GPI asynchronous one-sided communication, as opposed to both MPI one-sided communication and MPI bulk-synchronous two sided-communication, delivers full hardware bandwidth. As a function of message size, GPI reaches its peak performance much earlier than MPI.

A slightly more complex type of low-level benchmark is the two dimensional fast Fourier transformation on a distributed data set. We have compared two almost identical MPI and GPI implementations which feature the same communication pattern. Contrary to MPI, GPI was able to deliver near perfect scalability in a strong scaling setup.

GPI has also shown excellent scalability in a broad spectrum of typical real world HPC applications like the Computational Fluid Dynamics (TAU code from the DLR), or BQCD, a four dimensional nearest neighbor stencil algorithm. GPI has also been used in the implementation of fastest Unbalanced Tree Search (UTS) benchmark on the market.

Since many of the GASPI key objectives are shared by GPI, these results show the inherent potential of GASPI.

In 2010 the request for a standardization of the GPI interface emerged, which ultimately lead to the inception of the GASPI project in 2011. The work was funded by the German Ministry of Education and Science and included project partners Fraunhofer ITWM and SCAI, T-Systems SfR, TU Dresden, DLR, KIT, FZJ, DWD and Scapos.

The standard is currently being implemented in two flavors: a highly portable open source implementation based on GASNet and a commercial implementation aimed at ultimate performance. This latter implementation will be based on GPI. The TU Dresden, ZIH will provide profiling support for GASPI by means of extending the VAMPIR tool suite.

The GASPI project intends to drive the dissemination and visibility of the API by means of highly visible lighthouse projects in specific application domains, including CFD, turbo-machinery, weather and climate, oil and gas, molecular dynamics, as well as in the area of sparse and dense matrices. Amongst other implementations, the GASPI project will provide an asynchronous GASPI version of the Linpack benchmark.

There are a number of other projects that pursue similar goals to GASPI, the closest in spirit being OpenSHMEM. Ultimately the GASPI project aims at establishing a de-facto standard for an API for scalable, fault-tolerant and flexible communication in a partitioned global address Space. Whether that newly emerging standard will be called GASPI, however, remains to be seen.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Supercomputer Simulations Validate NASA Crash Testing

February 17, 2020

Car crash simulation is already a challenging supercomputing task, requiring pinpoint estimation of how hundreds of components interact with turbulent forces and human bodies. Spacecraft crash simulation is far more diff Read more…

By Oliver Peckham

What’s New in HPC Research: Quantum Clouds, Interatomic Models, Genetic Algorithms & More

February 14, 2020

In this bimonthly feature, HPCwire highlights newly published research in the high-performance computing community and related domains. From parallel programming to exascale to quantum computing, the details are here. Read more…

By Oliver Peckham

The Massive GPU Cloudburst Experiment Plays a Smaller, More Productive Encore

February 13, 2020

In November, researchers at the San Diego Supercomputer Center (SDSC) and the IceCube Particle Astrophysics Center (WIPAC) set out to break the internet – or at least, pull off the cloud HPC equivalent. As part of thei Read more…

By Oliver Peckham

ORNL Team Develops AI-based Cancer Text Mining Tool on Summit

February 13, 2020

A group of Oak Ridge National Laboratory researchers working on the Summit supercomputer has developed a new neural network tool for fast extraction of information from cancer pathology reports to speed research and clin Read more…

By John Russell

Nature Serves up Another Challenge to Quantum Computing?

February 13, 2020

Just when you thought it was safe to assume quantum computing – though distant – would eventually succumb to clever technology, another potentially confounding factor pops up. It’s the Heisenberg Limit (HL), close Read more…

By John Russell

AWS Solution Channel

Challenging the barriers to High Performance Computing in the Cloud

Cloud computing helps democratize High Performance Computing by placing powerful computational capabilities in the hands of more researchers, engineers, and organizations who may lack access to sufficient on-premises infrastructure. Read more…

IBM Accelerated Insights

Intelligent HPC – Keeping Hard Work at Bay(es)

Since the dawn of time, humans have looked for ways to make their lives easier. Over the centuries human ingenuity has given us inventions such as the wheel and simple machines – which help greatly with tasks that would otherwise be extremely laborious. Read more…

Researchers Enlist Three Supercomputers to Apply Deep Learning to Extreme Weather

February 12, 2020

When it comes to extreme weather, an errant forecast can have serious effects. While advance warning can give people time to prepare for the weather as it did with the polar vortex last year, the absence of accurate adva Read more…

By Oliver Peckham

The Massive GPU Cloudburst Experiment Plays a Smaller, More Productive Encore

February 13, 2020

In November, researchers at the San Diego Supercomputer Center (SDSC) and the IceCube Particle Astrophysics Center (WIPAC) set out to break the internet – or Read more…

By Oliver Peckham

Eni to Retake Industry HPC Crown with Launch of HPC5

February 12, 2020

With the launch of its Dell-built HPC5 system, Italian energy company Eni regains its position atop the industrial supercomputing leaderboard. At 52-petaflops p Read more…

By Tiffany Trader

Trump Budget Proposal Again Slashes Science Spending

February 11, 2020

President Donald Trump’s FY2021 U.S. Budget, submitted to Congress this week, again slashes science spending. It’s a $4.8 trillion statement of priorities, Read more…

By John Russell

Policy: Republicans Eye Bigger Science Budgets; NSF Celebrates 70th, Names Idea Machine Winners

February 5, 2020

It’s a busy week for science policy. Yesterday, the National Science Foundation announced winners of its 2026 Idea Machine contest seeking directions for futu Read more…

By John Russell

Fujitsu A64FX Supercomputer to Be Deployed at Nagoya University This Summer

February 3, 2020

Japanese tech giant Fujitsu announced today that it will supply Nagoya University Information Technology Center with the first commercial supercomputer powered Read more…

By Tiffany Trader

Intel Stopping Nervana Development to Focus on Habana AI Chips

February 3, 2020

Just two months after acquiring Israeli AI chip start-up Habana Labs for $2 billion, Intel is stopping development of its existing Nervana neural network proces Read more…

By John Russell

Lise Supercomputer, Part of HLRN-IV, Begins Operations

January 29, 2020

The second phase of the build-out of HLRN-IV – the planned 16 peak-petaflops supercomputer serving the North-German Supercomputing Alliance (HLRN) – is unde Read more…

By Staff report

IBM Debuts IC922 Power Server for AI Inferencing and Data Management

January 28, 2020

IBM today launched a Power9-based inference server – the IC922 – that features up to six Nvidia T4 GPUs, PCIe Gen 4 and OpenCAPI connectivity, and can accom Read more…

By John Russell

Julia Programming’s Dramatic Rise in HPC and Elsewhere

January 14, 2020

Back in 2012 a paper by four computer scientists including Alan Edelman of MIT introduced Julia, A Fast Dynamic Language for Technical Computing. At the time, t Read more…

By John Russell

Cray, Fujitsu Both Bringing Fujitsu A64FX-based Supercomputers to Market in 2020

November 12, 2019

The number of top-tier HPC systems makers has shrunk due to a steady march of M&A activity, but there is increased diversity and choice of processing compon Read more…

By Tiffany Trader

SC19: IBM Changes Its HPC-AI Game Plan

November 25, 2019

It’s probably fair to say IBM is known for big bets. Summit supercomputer – a big win. Red Hat acquisition – looking like a big win. OpenPOWER and Power processors – jury’s out? At SC19, long-time IBMer Dave Turek sketched out a different kind of bet for Big Blue – a small ball strategy, if you’ll forgive the baseball analogy... Read more…

By John Russell

Intel Debuts New GPU – Ponte Vecchio – and Outlines Aspirations for oneAPI

November 17, 2019

Intel today revealed a few more details about its forthcoming Xe line of GPUs – the top SKU is named Ponte Vecchio and will be used in Aurora, the first plann Read more…

By John Russell

Dell Ramps Up HPC Testing of AMD Rome Processors

October 21, 2019

Dell Technologies is wading deeper into the AMD-based systems market with a growing evaluation program for the latest Epyc (Rome) microprocessors from AMD. In a Read more…

By John Russell

IBM Unveils Latest Achievements in AI Hardware

December 13, 2019

“The increased capabilities of contemporary AI models provide unprecedented recognition accuracy, but often at the expense of larger computational and energet Read more…

By Oliver Peckham

SC19: Welcome to Denver

November 17, 2019

A significant swath of the HPC community has come to Denver for SC19, which began today (Sunday) with a rich technical program. As is customary, the ribbon cutt Read more…

By Tiffany Trader

D-Wave’s Path to 5000 Qubits; Google’s Quantum Supremacy Claim

September 24, 2019

On the heels of IBM’s quantum news last week come two more quantum items. D-Wave Systems today announced the name of its forthcoming 5000-qubit system, Advantage (yes the name choice isn’t serendipity), at its user conference being held this week in Newport, RI. Read more…

By John Russell

Leading Solution Providers

SC 2019 Virtual Booth Video Tour

AMD
AMD
ASROCK RACK
ASROCK RACK
AWS
AWS
CEJN
CJEN
CRAY
CRAY
DDN
DDN
DELL EMC
DELL EMC
IBM
IBM
MELLANOX
MELLANOX
ONE STOP SYSTEMS
ONE STOP SYSTEMS
PANASAS
PANASAS
SIX NINES IT
SIX NINES IT
VERNE GLOBAL
VERNE GLOBAL
WEKAIO
WEKAIO

Jensen Huang’s SC19 – Fast Cars, a Strong Arm, and Aiming for the Cloud(s)

November 20, 2019

We’ve come to expect Nvidia CEO Jensen Huang’s annual SC keynote to contain stunning graphics and lively bravado (with plenty of examples) in support of GPU Read more…

By John Russell

Fujitsu A64FX Supercomputer to Be Deployed at Nagoya University This Summer

February 3, 2020

Japanese tech giant Fujitsu announced today that it will supply Nagoya University Information Technology Center with the first commercial supercomputer powered Read more…

By Tiffany Trader

51,000 Cloud GPUs Converge to Power Neutrino Discovery at the South Pole

November 22, 2019

At the dead center of the South Pole, thousands of sensors spanning a cubic kilometer are buried thousands of meters beneath the ice. The sensors are part of Ic Read more…

By Oliver Peckham

Top500: US Maintains Performance Lead; Arm Tops Green500

November 18, 2019

The 54th Top500, revealed today at SC19, is a familiar list: the U.S. Summit (ORNL) and Sierra (LLNL) machines, offering 148.6 and 94.6 petaflops respectively, Read more…

By Tiffany Trader

Azure Cloud First with AMD Epyc Rome Processors

November 6, 2019

At Ignite 2019 this week, Microsoft's Azure cloud team and AMD announced an expansion of their partnership that began in 2017 when Azure debuted Epyc-backed instances for storage workloads. The fourth-generation Azure D-series and E-series virtual machines previewed at the Rome launch in August are now generally available. Read more…

By Tiffany Trader

Intel’s New Hyderabad Design Center Targets Exascale Era Technologies

December 3, 2019

Intel's Raja Koduri was in India this week to help launch a new 300,000 square foot design and engineering center in Hyderabad, which will focus on advanced com Read more…

By Tiffany Trader

In Memoriam: Steve Tuecke, Globus Co-founder

November 4, 2019

HPCwire is deeply saddened to report that Steve Tuecke, longtime scientist at Argonne National Lab and University of Chicago, has passed away at age 52. Tuecke Read more…

By Tiffany Trader

Cray Debuts ClusterStor E1000 Finishing Remake of Portfolio for ‘Exascale Era’

October 30, 2019

Cray, now owned by HPE, today introduced the ClusterStor E1000 storage platform, which leverages Cray software and mixes hard disk drives (HDD) and flash memory Read more…

By John Russell

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This