MPI Updates Parallel Capabilities for Leadership-Class Supercomputers and Broader HPC Community

May 12, 2023

May 12, 2023 — The Message Passing Interface (MPI) is recognized as the ubiquitous communications framework for scalable distributed high-performance computing (HPC) programming. Created in 1994,[1] MPI is arguably the core building block of distributed HPC computing. According to an Exascale Computing Project (ECP) estimate, more than 90% of the ECP codes use MPI—either directly or indirectly.[2] In a chicken-and-egg analogy, where MPI goes so goes HPC and vice versa.

David Bernholdt

David Bernholdt, principal investigator of the ECP OMPI-X project and distinguished R&D staff at ORNL observed, “Programmers who wish to use MPI in a threaded, heterogenous environment need to understand how the updates to the MPI standard address multithreading performance bottlenecks in MPI libraries prior to the MPI 4.0 standard. Due to the efforts of many, including the OMPI-X team, programmers can now use many of these performance enhancements in the latest release of Open MPI.”

The scientific computing community’s increasing need for HPC has driven continued growth in hardware scale and heterogeneity. These innovations have placed unforeseen demands on the legacy MPI communications standard and library implementations. Many novel approaches have been integrated into the MPI 4.0 specification to adapt the venerable MPI standard and library implementations (e.g., Open MPI v5.0.x) so they can run more efficiently in modern, heavily threaded, and GPU-accelerated computational environments. Eliminating lock inefficiencies in parallel codes, supporting heterogeneity, and reducing resource utilization have been key focal points as modern HPC clusters can now run thousands to millions of concurrent threads of execution—a degree of parallelism that was simply not possible in 1994.

The rise of these architectures has in turn increased the importance of hybrid programming models in which node-level programming models such as OpenMP are coupled with MPI. This is commonly referred to as MPI+X. These changes are not just theoretical, and this is why the OMPI-X team has been advocating, driving, and developing updates to the open-source Open MPI library. This has been a community effort that encompasses significant work by many individuals, discussions in the MPI Forum, and feedback from the various organizations that comprise the MPI standards committee.

In leading the ECP OMPI-X project, Bernholdt has been both participant and advocate in correcting many of the issues that limit MPI performance in heavily threaded, heterogenous computing environments and incorporating these updates into the Open MPI library.

Technical Introduction

Bernholdt explained the driving focus behind the OMPI-X efforts, “We recognize that MPI serves a very broad community. The current leadership-class systems are only a portion of that community, but these systems are important because they act as a proving ground. For this reason, we pay attention to ensure that the forthcoming MPI specification serves the needs of the high-end systems as well as the needs of the entire community.”

He continued by noting that when the OMPI-X project started, the HPC community had only an approximate vision of what an exascale system would look like. For this reason, the team picked concepts that were important and useful. They then advocated and eventually facilitated the adoption of these concepts by the standards committee—a time-consuming and laborious process that incorporated feedback from many projects and the MPI Forum. As part of the OMPI-X effort, the team also worked to implement desirable contributions in the new standard.

These innovations appear in the MPI 4.0 standard and the Open MPI 5.0.x library software releases. The updates are extensive and are the subject of numerous publications in the literature:

  • Partitioned communications support increased flexibility and the overlap of communication and computation. Partitioned communication is applicable to highly threaded CPU-side MPI codes but has significant utility for GPU-side MPI kernel calls with low expected overheads. This includes the addition of performance-oriented partitioned point-to-point communication primitives and autotuning collective operations.
  • Sessions (and PMIx) introduces a concept of isolation into MPI by relaxing the requirements for global initialization, which currently produces a global communicator. Each MPI Session creates its own isolated MPI environment, potentially with different settings, optimization opportunities, and communication data structures.[3] Sessions enable dynamic resource allocation that leverages the Process Management Interface for Exascale (PMIx). PMIx communicates with other layers of the software stack and can be used by job schedulers such as SLURM. It also permits better interactions with file systems and dynamic process groups for managing asynchronous group construction and destruction. The “PMIx: A Tutorial” slide deck in the PMIx GitHub repository provides a quick overview of the features and benefits of PMIx.
  • Standardization of error management within MPI. The MPI 4.0 standard allows for asynchronous operations, which required updating the MPI error notification mechanism.
  • Resilience related research considers fault-tolerance constructs as possible additions to MPI, including User Level Failure Mitigation (ULFM) and Reinit. The ULFM proposal as developed by the MPI Forum’s Fault Tolerance Working Group supports the continued operation of MPI programs after node failures have impacted application execution. See the paper and research hub for more information. Reinit++ is a redesign of the Reinit[4],[5] approach and variants[6],[7] that leverages checkpoint-restart.
  • Usual performance/scalability improvements as reflected in benchmarks using the Open MPI library.

Multithreaded Implications for MPI: Performance, Portability, Scalability, and Robustness

User-level threading to exploit the performance capabilities of modern hardware has motivated the extensive analysis performed by the OMPI-X team and other investigators. Topics include the best threading models (e.g., Pthreads or Qthreads) as well as autotuning. The OMPI-X team has also dedicated efforts to improving testing and the all-important continuous integration (CI) to ensure correct operation on many HPC platforms.

Overview of the prototype implementation of MPI Sessions and PMIx in Open MPI. This example shows two nodes running a (dynamic) MPI job. The figure differentiates between four layers (see labels on the left), which are each accessible from the layer above via the interfaces indicated on the right side. The interior of the nodes delineates the required interactions of the dynamicity extensions to MPI with PRRTE using PMIx.

 

To continue reading ECP’s news on MPI, click here.


Source: Rob Farber, ECP

Shares
 MPI is arguably the core building block of distributed HPC computing. Read more…

" share_counter=""]
Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

ASC23: Application Results

June 2, 2023

The ASC23 organizers put together a slate of fiendishly difficult applications for the students this year. The apps were a mix of traditional HPC packages, like WRF-Hydro and FVCOM, plus machine learning centric programs Read more…

Q&A with Marco Pistoia, an HPCwire Person to Watch in 2023

June 2, 2023

HPCwire Person to Watch Marco Pistoia wears a lot of hats at JPMorgan Chase & Co.: managing director, distinguished engineer, head of global technology applied research and head of quantum computing. That work with J Read more…

HPC Career Notes: June 2023 Edition

June 1, 2023

In this monthly feature, we’ll keep you up-to-date on the latest career developments for individuals in the high-performance computing community. Whether it’s a promotion, new company hire, or even an accolade, we’ Read more…

Intersect360: HPC Market ‘Returning to Stable Growth’

June 1, 2023

The folks at Intersect360 Research released their latest report and market update just ahead of ISC 2023, which was held in Hamburg, Germany, last week. The headline: “We’re returning to stable growth,” per Addison Read more…

Lori Diachin to Lead the Exascale Computing Project as It Nears Final Milestones

May 31, 2023

The end goal is in sight for the multi-institutional Exascale Computing Project (ECP), which launched in 2016 with a mandate from the Department of Energy (DOE) and National Nuclear Security Administration (NNSA) to achi Read more…

AWS Solution Channel

Shutterstock 1493175377

Introducing GPU health checks in AWS ParallelCluster 3.6

GPU failures are relatively rare but when they do occur, they can have severe consequences for HPC and deep learning tasks. For example, they can disrupt long-running simulations and distributed training jobs. Read more…

 

Shutterstock 1415788655

New Thoughts on Leveraging Cloud for Advanced AI

Artificial intelligence (AI) is becoming critical to many operations within companies. As the use and sophistication of AI grow, there is a new focus on the infrastructure requirements to produce results fast and efficiently. Read more…

ASC23: LINPACK Results

May 30, 2023

With ISC23 now in the rearview mirror, let’s get back to the results from the ASC23 Student Cluster Competition. In our last articles, we looked at the competition and applications, plus introduced the teams, now it’ Read more…

ASC23: Application Results

June 2, 2023

The ASC23 organizers put together a slate of fiendishly difficult applications for the students this year. The apps were a mix of traditional HPC packages, like Read more…

Intersect360: HPC Market ‘Returning to Stable Growth’

June 1, 2023

The folks at Intersect360 Research released their latest report and market update just ahead of ISC 2023, which was held in Hamburg, Germany, last week. The hea Read more…

Lori Diachin to Lead the Exascale Computing Project as It Nears Final Milestones

May 31, 2023

The end goal is in sight for the multi-institutional Exascale Computing Project (ECP), which launched in 2016 with a mandate from the Department of Energy (DOE) Read more…

At ISC, Sustainable Computing Leaders Discuss HPC’s Energy Crossroads

May 30, 2023

In the wake of SC22 last year, HPCwire wrote that “the conference’s eyes had shifted to carbon emissions and energy intensity” rather than the historical Read more…

Nvidia Announces Four Supercomputers, with Two in Taiwan

May 29, 2023

At the Computex event in Taipei this week, Nvidia announced four new systems equipped with its Grace- and Hopper-generation hardware, including two in Taiwan. T Read more…

Nvidia to Offer a ‘1 Exaflops’ AI Supercomputer with 256 Grace Hopper Superchips

May 28, 2023

We in HPC sometimes roll our eyes at the term “AI supercomputer,” but a new system from Nvidia might live up to the moniker: the DGX GH200 AI supercomputer. Read more…

Closing ISC Keynote by Sterling and Suarez Looks Backward and Forward

May 25, 2023

ISC’s closing keynote this year was given jointly by a pair of distinguished HPC leaders, Thomas Sterling of Indiana University and Estela Suarez of Jülich S Read more…

The Grand Challenge of Simulating Nuclear Fusion: An Overview with UKAEA’s Rob Akers

May 25, 2023

As HPC and AI continue to rapidly advance, the alluring vision of nuclear fusion and its endless zero-carbon, low-radioactivity energy is the sparkle in many a Read more…

CORNELL I-WAY DEMONSTRATION PITS PARASITE AGAINST VICTIM

October 6, 1995

Ithaca, NY --Visitors to this year's Supercomputing '95 (SC'95) conference will witness a life-and-death struggle between parasite and victim, using virtual Read more…

SGI POWERS VIRTUAL OPERATING ROOM USED IN SURGEON TRAINING

October 6, 1995

Surgery simulations to date have largely been created through the development of dedicated applications requiring considerable programming and computer graphi Read more…

U.S. Will Relax Export Restrictions on Supercomputers

October 6, 1995

New York, NY -- U.S. President Bill Clinton has announced that he will definitely relax restrictions on exports of high-performance computers, giving a boost Read more…

Dutch HPC Center Will Have 20 GFlop, 76-Node SP2 Online by 1996

October 6, 1995

Amsterdam, the Netherlands -- SARA, (Stichting Academisch Rekencentrum Amsterdam), Academic Computing Services of Amsterdam recently announced that it has pur Read more…

Cray Delivers J916 Compact Supercomputer to Solvay Chemical

October 6, 1995

Eagan, Minn. -- Cray Research Inc. has delivered a Cray J916 low-cost compact supercomputer and Cray's UniChem client/server computational chemistry software Read more…

NEC Laboratory Reviews First Year of Cooperative Projects

October 6, 1995

Sankt Augustin, Germany -- NEC C&C (Computers and Communication) Research Laboratory at the GMD Technopark has wrapped up its first year of operation. Read more…

Sun and Sybase Say SQL Server 11 Benchmarks at 4544.60 tpmC

October 6, 1995

Mountain View, Calif. -- Sun Microsystems, Inc. and Sybase, Inc. recently announced the first benchmark results for SQL Server 11. The result represents a n Read more…

New Study Says Parallel Processing Market Will Reach $14B in 1999

October 6, 1995

Mountain View, Calif. -- A study by the Palo Alto Management Group (PAMG) indicates the market for parallel processing systems will increase at more than 4 Read more…

Leading Solution Providers

Contributors

CORNELL I-WAY DEMONSTRATION PITS PARASITE AGAINST VICTIM

October 6, 1995

Ithaca, NY --Visitors to this year's Supercomputing '95 (SC'95) conference will witness a life-and-death struggle between parasite and victim, using virtual Read more…

SGI POWERS VIRTUAL OPERATING ROOM USED IN SURGEON TRAINING

October 6, 1995

Surgery simulations to date have largely been created through the development of dedicated applications requiring considerable programming and computer graphi Read more…

U.S. Will Relax Export Restrictions on Supercomputers

October 6, 1995

New York, NY -- U.S. President Bill Clinton has announced that he will definitely relax restrictions on exports of high-performance computers, giving a boost Read more…

Dutch HPC Center Will Have 20 GFlop, 76-Node SP2 Online by 1996

October 6, 1995

Amsterdam, the Netherlands -- SARA, (Stichting Academisch Rekencentrum Amsterdam), Academic Computing Services of Amsterdam recently announced that it has pur Read more…

Cray Delivers J916 Compact Supercomputer to Solvay Chemical

October 6, 1995

Eagan, Minn. -- Cray Research Inc. has delivered a Cray J916 low-cost compact supercomputer and Cray's UniChem client/server computational chemistry software Read more…

NEC Laboratory Reviews First Year of Cooperative Projects

October 6, 1995

Sankt Augustin, Germany -- NEC C&C (Computers and Communication) Research Laboratory at the GMD Technopark has wrapped up its first year of operation. Read more…

Sun and Sybase Say SQL Server 11 Benchmarks at 4544.60 tpmC

October 6, 1995

Mountain View, Calif. -- Sun Microsystems, Inc. and Sybase, Inc. recently announced the first benchmark results for SQL Server 11. The result represents a n Read more…

New Study Says Parallel Processing Market Will Reach $14B in 1999

October 6, 1995

Mountain View, Calif. -- A study by the Palo Alto Management Group (PAMG) indicates the market for parallel processing systems will increase at more than 4 Read more…

ISC 2023 Booth Videos

Cornelis Networks @ ISC23
Dell Technologies @ ISC23
Intel @ ISC23
Lenovo @ ISC23
ISC23 Playlist
  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire