On January 9, 2005, HPCwire published an article that stated Massively Parallel Technologies (MPT), a provider of on-demand high-performance computing, had developed the first mathematical derivation of Amdahl's Law. This story [109101] can be found here: http://www.tgc.com/hpcwire/hpcwireWWW/05/0121/109101.html.

After the story ran, HPCwire received several letters from prominent industry insiders who felt the claims were exaggerated and shaky. HPCwire editor Tim Curns spoke with the CTO of Massively Parallel Technologies, Kevin Howard, for some clarification regarding the announcement. The interview [109182] can be found here: http://www.tgc.com/hpcwire/hpcwireWWW/05/0204/109182.html

While opinions vary, several members of the HPC community wrote to HPCwire to comment on MPT's explanations.

I urge readers to send any future questions or comments to me at [email protected].

Mr. Curns:

I've just read your interview of Kevin Howard, MPT, and think you did a great job. His responses to your straightforward questions will allow the community to make their own judgments. It's interesting that their whitepaper is proprietary. Additionally, I refer to an article entitled “Superlinear Speed- up and the Halting Problem” published in SOFTWARE–PRACITCE AND EXPERIENCE, Vol 16, No. 8, pp 781-782, August 1986, that I wrote. The summary (below) refutes MPT's claim of “superlinear speed-up.”

Briefly, take the new algorithm (in this case, the one propounded by MPT) and run it on a single processor (of the same type that would be used on the parallel processor that would run the algorithm) that is simulating a parallel processor (that is the “round-robin” approach mentioned below–the single processor simulates one instruction from each of the n processors of the parallel processor, then goes on and simulates the next instruction . . .). Doing so will take a factor of n times longer than the parallel processor would take. In other words, the parallel processor running the new algorithm gains only a factor of n! So much for the new development.

My compliments to you on HPCwire.

Best regards,

Paul Schneck

Summary

This paper describes a technique for refuting the claim that “new” algorithms on parallel processing systems with n processors will yield a speed-up of more than n. The technique uses a round-robin service approach which is borrowed from that used in automata theory. It is shown that a sequential processor with this service technique can yield a speed-up of 1/n times that claimed for a parallel processor.

Dear HPCWire,

I was one of the fellows who worked with David Bailey on expressing our concern over MPT's claims. While I applaud your effort to attempt to pursue the issue, MPT's claims are still wholly unsubstantiated and Mr. Howard's response contains nothing more than marketing jargon meant to obfuscate rather than demonstrate their complete lack of scientific data. Furthermore, the use of Gene Amdahl, who is now 86 years old, as a poster boy to help legitimize this work as real intellectual property, is nothing short of shameful.

First and foremost, there is nothing to be discovered in Amdahl's Law. Like any mathematical formulation of a phenomena, Amdahl's Law contains terms that are abstractions of other higher order terms. These terms can be formulated in any number of ways, for example the serial component can be broken down into terms that parameterize the application's performance on the basis of its load on the memory subsystem, FP pipeline or branch prediction hardware. The parallel component can similarly be broken down, for example, into the amount of data exchanged at each time step, the mode of that exchange, the parameters of the communication link (Bandwidth/Latency or LOG-GP) and of course the above computational component as well. What they managed to “derive mathematically” is a common homework assignment for students in their first parallel processing course.

Second, it's quite clear to anyone with a rudimentary science education that one cannot invent hardware on the fly. Overlapping computation and communication is possible only with the hardware to perform it. MPT seems to make this sound like it's possible on any system. The 100Mb Ethernet clusters that run TCP/IP, and there are no less than dozens of papers on the CPU overhead incurred by TCP/IP packet processing. The examples cited by MPT in their release and on their website, (like the success of SETI@Home and their work on image processing, BLAST and the HPCS Benchmarks) are almost entirely “Embarrassingly Parallel,” meaning that there is very little data exchanged during the computation, if any at all. In this case, I should say the term is especially appropriate. If MPT would like to try some real applications, I might recommend them try the NAS Parallel Benchmarks, SPEC HPC or ASCI Benchmarks.

At best, MPT has managed to create a parallel programming model that makes embarrassingly parallel jobs easy to create for the average scientist. At worst, they have perpetuated an elaborate hoax aimed at making money off the uninformed in order to demonstrate profits to a venture capital firm with an rather poor due diligence team.

I would point out to those casual readers of HPCWire who might be susceptible to MPT's pitch, that there is no magic in parallel processing. Unlike the experiments in “cold fusion”; the phenomena at work are well established and well understood. There is nothing to be had for free. We, the researchers in parallel computing, will continue to work towards improving the efficiency of architectures and simplifying parallel programming models. It is difficult work and misinformation as such is being spread by MPT makes that task even more challenging.

While I could continue to refute other facts in their response, it largely serves no point. I shudder to think that arbitrary marketing hype and pseudo- science is the future of the HPC Commercial releases…but maybe so, at the very least, we can choose to ignore that section of your periodical.

Anyways, I appreciate your effort in pursuing this with MPT. It's a shame that they didn't capitalize on the opportunity.

Philip J. Mucci

Innovative Computing Laboratory, University of Tennesse, Knoxville, TN. Center for Parallel Computers, Royal Institute of Technology, Stockholm, Sweden.

Dear Tim,

There is a simple explanation why you have received several letters from industry HPC experts who felt doubts about your HPCwire article “Massively Parallel Tech Mathematically Derives Amdahl's Law.”

Amdahl's Law was mathematically derived many years ago by Gene himself. There is no need to derive it again. What MPT has done is something different. MPT has extended (generalized)this Law by including several factors that impact the notion of efficiency and scalability but were ignored by Amdahl. Kevin Howard has mentioned some extentions in your interview discussion published. I have not seen the new MPT formulations, but I believe that they are useful and provide more practical tools for assessing quality of parallel algorithms, software and the parallel code-machine fit.

Dr. Janusz Kowalik