One of the biggest impediments to HPC application development today is the difficulty of writing software for cluster architectures. Unlike clusters, shared memory machines provide a globally accessible memory space, offering a more programmer-friendly environment for doing parallel processing with large datasets. But since clusters scale so economically, they have become the dominant high performance computing architecture today.

Unfortunately, writing applications for clusters means the programmer has to deal with the hard realities of distributed memory, where data has to be shuffled from one node to the other so that threads can talk with one another and data can be kept in a coherent state. Thus was born the Message Passing Interface (MPI), the de facto standard for parallel programming communications.

In the May issue of CTWatch Quarterly, Thom Dunning, Robert Harrison and Jeffrey Nichols write: “Without fear of contradiction, the MPI standard has been the most significant advancement in practical parallel programming in over a decade, and it is the foundation of the vast majority of modern parallel programs.”

But it's hard to find a real fan of MPI today. Most either tolerate it or hate it. Although it provides a widely portable and standardized programming interface for parallel computing, its shortfalls are numerous: hard to learn, difficult to program, no allowance for incremental parallelization, doesn't scale easily, and so on. It's widely acknowledged that MPI's limitations must be overcome to make parallel programming more accessible.

Dunning, Harrison and Nichols continue: “A completely consistent (and deliberately provocative) viewpoint is that MPI is evil. The emergence of MPI coincided with an almost complete cessation of parallel programming tool paradigm research. This was due to many factors, but in particular to the very public and very expensive failure of HPF. The downsides of MPI are that it standardized (in order to be successful itself) only the primitive and already old communicating sequential process (CSP) programming model, and MPI's success further stifled adoption of advanced parallel programming techniques since any new method was by definition not going to be as portable.”

For the NWChem quantum chemistry application that the authors are discussing in the CTWatch article, the solution to MPI's limitations was the use of the Global Arrays (GA) Toolkit. The Toolkit provides a shared memory style programming environment for use with distributed memory computers. The basic context consists of distributed array data structures — global arrays — used as if they are stored in shared memory. The needed functionality for data distribution and data access is transparent to the programmer. The GA model exposes to the programmer the non-uniform memory access (NUMA) characteristics of high performance computers and acknowledges that access to a remote portion of the shared data is slower than to the local portion.

But while the physical nature of distributed memory has been abstracted, the GA interface still requires that explicit calls be added to the code in order to manage the global data.

Which bring us to Cluster OpenMP, a distributed memory version of standard OpenMP developed by Intel. Standard OpenMP is a widely used programming interface for creating parallel applications on shared memory architectures. It's been around since 1997. Like OpenMP, Cluster OpenMP does not require that the programmer invoke explicit library calls to achieve parallelization; this is accomplished with in-line compiler directives. Like GA, it abstracts the physically distributed memory, but it avoids both MPI's and GA Toolkit's reliance on library calls to make things happen.  So you have the ability to switch off the compiler directives in the source code to restore your original serial program. Nice.

The Cluster OpenMP product was released on May 9 and is available with Intel's 9.1 Fortran and C++ compilers. Curiously, no press announcement was forthcoming from Intel about the release. But if you're wondering about Cluster OpenMP, you're in luck. In this issue, Intel's Jay Hoeflinger and Larry Meadows describe their new offering and how it can be used to turn an OpenMP program into a cluster-capable version.

Clustered JVM

This past week, Terracotta Inc., a vendor that provides scalable Java solutions for the enterprise, announced Terracotta 2.0., claimed to be the industry's first production-ready “clustered” Java Virtual Machine (JVM). In contrast with typical frameworks, Terracotta 2.0 clusters at the JVM level, instead of at the software application level, allowing application programmers to write normal Java code that will run transparently in clustered environments.

The Terracotta solution has some similarities to the Cluster OpenMP offering, inasmuch as it abstracts a cluster-wide shared memory. When shared Java objects are accessed by the application, Terracotta's cluster-aware software detects this at the intermediate byte-code level and reads/writes the data from/to the appropriate nodes to keep the objects coherent. Unlike Cluster OpenMP, the Terracotta solution requires no compiler directives; shared data is specified in the Java language itself.

In general, Java is not regarded as a conventional HPC language because of the run-time performance limitations related to its byte-code interpretive model. It's also a little weak in things such as floating-point/complex number support and control of low-level data layout. The Terracotta solution is geared towards high availability business applications that increasing need to scale out to large cluster environments. According to Terracotta engineers, their solution would also be very suitable for cluster and grid management tools, at the meta-level above the HPC applications.

In one of the great paradoxes of high performance computing, the most popular high-level languages for supercomputing applications — C and Fortran — are used not because they're so advanced, but because they're so primitive. C and Fortran source code maps easily to conventional CPU hardware, so the generated assembly code is able to achieve good performance. The result is that we end up using 30-year-old software languages to develop code for state-of-the-art supercomputers. Oh the irony!

—–

As always, comments about HPCwire are welcomed and encouraged. Write to me, Michael Feldman, at [email protected].