A growing number of problems demand parallel computing capabilities these days, and the processing power of high performance computers have kept pace with this demand. Yet a hurdle to unlocking the true potential of parallel computing remains: affordable, easy-to-use parallel programming tools. The “software gap” — the gap between hardware capabilities and actual benefits we can extract through programming — is growing wider. Many applications are available for parallel computers, yet the custom development required by these tools is exceedingly complex, takes months or years to develop, and runs in batch mode over hours or days.

As engineers and scientists explore parallel programming approaches, whether they are first timers or experienced users, an increasing array of choices is available to them. What is better for your next project — a parallel extension to high-level desktop tools? Programming toolkits and APIs for C and FORTRAN developers? An interactive parallel computing platform?

No one-size-fits-all solution exists. The optimal choice is driven by several factors, including ease of use, processing power needs and vendors' market focus. The weight of each factor depends on your unique project, situation and environment. To help you sort through these factors to determine the best parallel tool for your unique needs, the following are key questions to ask your potential vendor, and why the questions matter.

Ease of Use

1. “Can I use my familiar desktop tools to develop the parallel application?”

Familiar desktop tools such as MATLAB and Mathematica have largely supplanted C and FORTRAN in developing custom models, algorithms, and simulations, due to their ease of use, high-level constructs, and interactive and graphical environment. If you have such a tool of choice, then the ability to use it to develop new parallel applications would reduce the learning curve substantially, and may dramatically accelerate the development of custom parallel applications. Furthermore, the more the parallelization of your code is transparent and automatic, the closer would be “time to partial satisfaction” — the ability to run your model in parallel, testing and scaling it with real data.

2. “Does your solution require users to rewrite/restructure/overhaul their existing serial/parallel code?”

Related to the previous question, this question probes how different is the serial code from the parallel code. The best parallel code is a serial code that executes in parallel. There are two ways to reuse code: by executing in a global array syntax, or through task parallelism. A system that allows for the reuse of code with both paradigms is far more powerful than a solution with one or the other, separately.

3. “Do I have to worry about how many processors I have access to explicitly?”

A measure of parallel abstraction is that a program should execute independent of the number of processors it has. Performance might vary, but the correctness of the program should not. Does the user or the system explicitly worry about the number of available processors?

4. “Do I have to worry about data distribution explicitly?”

Another measure of parallel abstraction is whether the user must know how data is distributed in order to execute. Again, performance might vary, but the correctness of the program should not. Suppose, for example, you are inverting 1000 medium sized matrices, but each one is distributed. Does the user or the system worry about the data? Or suppose you divide a problem into pieces, but it doesn't divide up evenly. Do you worry about those annoying remainders, and those blocks that cross processor boundaries, or does the system take care of this automatically?

5. “Does your solution hide the complexities of message passing programming?”

Message passing, especially the MPI standard, is a low level method for programming parallel computers. It has traditionally given expert users access to the performance they seek, though often this performance does not come easily. Although experimental hybrids are popping up now where higher level languages are being equipped with message passing, such solutions still require the users to be versed in message-passing programming.

6. “What is the SLOC (software lines of code) expansion factor when going from serial to parallel?”

Every user dreams that his or her serial program will just work in parallel. How close is that to reality? One measure is to count the lines of code in the serial prototype as compared to the working parallel implementation. The less the original code needs to be modified, the better.

Power

7. “Does it scale to large memory and processor sizes?”

Today's mainstream 8- and 16-processor clusters easily offer an order of magnitude in performance over a desktop PC. For some applications, this is more than ample. But many of today's toughest computational problems are getting larger and more complex every year. A 10 MB data file generated by airborne radar today may swell to a terabyte-sized data set generated by an array of satellites.

8. “Does your solution support both embarrassingly parallel and data parallel algorithms?”

Coarse-grained parallelism (sometimes called “embarrassingly parallel” or “task parallelism”) is a powerful method to carry out many independent calculations in parallel, such as Monte Carlo simulations, or “un-rolling” serial FOR loops. Fine-grained parallelism (sometimes called “data parallel” or “global array syntax”) is used for high-level matrix and vector operations on huge data sets. It turns out that most modern production-level parallel applications require both. Some of the computations may be on data sets that fill the machine's huge memories and require global operations such as sum of squares, or an average, or Fourier transforms or a linear system solution. However, the global operation may also interoperate with a FOR loop over smaller pieces of data, and the support for and interoperability between the coarse- and fine-grained parallelism may be critical. Taking that example, parallelizing the Monte Carlo may be a large piece of a computation, but a global analysis of the statistics that have emerged may also be equally important.

9. “Can you provide commercially robust improvements of parallel libraries I am currently using?”

Many parallel numerical libraries, perhaps SCALAPACK being the most famous, are open source and familiar to some users. Nonetheless, these libraries are newer, more complex, and less well tested than their older counterparts. These libraries often go under the heading of research projects. Ask your provider if they just plugged in the library or if they have made proprietary tests and improvements to the basic software. Many vendors just take the libraries hoping that they are correct, leaving the user in a “buyer beware” situation.

10. “Can I integrate my legacy (or newly written or found on the Internet) C/FORTRAN serial and parallel code?”

The ability to put a “front end” on traditional parallel and serial codes can be extremely valuable for debugging and productivity. You should not be prevented from readily plugging in existing codes — most popularly serial C or FORTRAN, surrounded with a parallel FOR loop, or C/MPI and FORTRAN/MPI. Existing codes in other high-level and low-level languages should also integrate nicely with no user requirement about data distribution. For example, suppose the user wants to solve 100 linear systems of equations with a custom solver written in C, and the data for each of the 100 systems is not local to a processor, the programmer should not have to be concerned with getting the data into the right place.

Business

11. “How focused is the vendor on parallel computing?”

Depending on the scope and importance of the new parallel programming project, a vendor's commitment may be relevant. For example, if a small team is purchasing an 8-processor cluster for offloading occasional computations, the vendor's focus on parallel computing may not be a critical issue. But for many enterprise environments, where parallel codes are in service for years, and sometimes decades, a vendor's focus is critical. One relevant metric is the fraction of revenue that comes from serving the high-performance parallel computing segment, because this can serve as a leading indicator of how much time and money companies invest to understand and solve a market's problems.

12. “How stable and robust is the technology?”

You need to understand the product's maturity. A reasonable measure is the number of major releases the product has gone through (remember Windows 2.0, 3.0, 3.1, and 95? Widespread acceptance of Windows did not really start until 3.1). This scope would also extend to the research and community developments for the various open-source or academic projects that were ultimately taken commercial.

13. “Is this the right platform for my application?”

The right HPC software will not just get the job done in a current project, but also serve as the foundation for future work. How rich and broad such a platform needs to be is situation-specific, and would depend on questions like: What suite of desktop tools does the team want to use in the development of custom parallel applications? How important is the product's extensibility, through an API or SDK, to plug in future off-the-shelf or custom codes? How important is it that the platform supports both interactivity during the application development, refinement, and discovery process; as well as large batch runs?

14. “Is the vendor an innovator?”

Naturally, this may not be important in every situation. Dell has built a tremendous business in personal computers arguably not by delivering leading edge capabilities, but through a very efficient logistics chain. But when solving the largest, most complex — and often most important problems — vendor innovation matters. In those cases, the problem with choosing a follower is not just in the timing of a new product or feature availability, but that the company processes, culture, and approach may hinder it from delivering necessary breakthroughs. Put another way, “if they are not the lead dog, their view never changes” — and this has potentially important implications for their customers.

—–

About the authors:

Alan Edelman is a professor at MIT and the chief science officer at Interactive Supercomputing. He has won the prestigious Gordon Bell Prize in parallel computing, has been awarded many other prizes for mathematics and numerical computation, is the inventor of the idea behind the Basic Linear Algebra Communications Subroutines, holds several patents in high performance computing, and has taught almost a generation of high-powered graduate students at MIT how to program and think about high performance computing. When he was a graduate student himself, he microcoded the Connection Machine. He and his students began building a parallel platform in 1997 with the intent of having a system that would contain sufficiently many hooks to support parallel versions of very high level languages.