THE BEOWULF FACTOR IN HIGH PERFORMANCE COMPUTING

December 8, 2000

by Christopher Lazou

San Diego, CALIF. — In the late 1980’s it became fashionable to give presentations with sound bites, such as, “supercomputers and the killer micro.” The message was that cheap micros will replace the large scale supercomputers then in use, for solving the pressing problems of our time.

Today the “killer Beowulf” has arrived; a cheap solution to supercomputing. A lot of time and money albeit in small parcels have been spent by “Geegs” to install clusters of Pentium, Alphas’, Power 3 or other makes of commodity Chip, on a DIY (Do It Yourself) basis. These systems are mostly experimental relatively small, 32-64 nodes, cheap in capital outlay but fraught with danger and poor reliability, especially in novice hands. “Buy (DIY) in haste, repent in leisure”, as the Romans used to say.

Yet there are others with a much more serious mission, for example the Beowulf machine at Sandia, with 2600 Alpha EV67 chips, is using LINUX and Myrinet2000 as interconnect. Bill Camp at Sandia wants to enhance this system to 20,000 Alpha EV7 processors by year 2004-5. The largest civilian Beowulf system planned, paid from the $45 million NSF money, is the one at the Pittsburgh Supercomputing Centre, consisting of 2,728 Alpha EV67 processors in 682 nodes using QSNet and rated at 6Tflop/s peak performance.

Thus, in the last three years the Beowulf paradigm has been used to build departmental and desktop systems from off-the-shelf components. These typically consist of 32 – 64, or several hundred or in a few cases 1000s, of Intel Pentium III, IBM power 3 or Compaq Alpha chips, cobbled together using Myrinet or QSNet as interconnect. This Do It Yourself (DIY) system building, is cheaper on paper, as costs often do not include the engineering and commissioning manpower or the vendor profit margins. It has become an attractive option in academic institutions strapped for money, especially in those with experimental non mission critical research. Apart from the money constrains, those who take this path tend to be young academics with a technical scientific background who find the idea of bolting their own systems together challenging. This activity gives them job satisfaction even though it diverts energy from their scientific task.

Experience from the US (ASCI) programme showed that the scalable systems from commodity chips is viable and provides capability computing, while not as efficient in throughput for conventional large scale applications.

Experience also shows, that prototype Beowulf systems with 32-CPUs compare favourably with the Cray T3E, but not when using larger systems. These prototype systems with Pentium, IBM Power 3 and Alpha Chips, are available and with benchmark results, at Daresbury Laboratory. For example, the applications platform developed under the QUASI project is a 32-CPU Alpha cluster connected with QSNet and running LINUX. In the UK some clusters are mainly for single application, e.g. university of Liverpool, while Daresbury Laboratory, UK, NOAA, Sandia, Cornell and Pittsburgh university, in the USA, are trying to prove the Beowulf paradigm for HPC.

At present a 32-node Beowulf system using Myrinet interconnect can be assembled for about $100 thousand. It is scalable to 128 nodes and even if you add 1TBytes of archive disks, this can cost about $200 thousand. For the moment, HPC Beowulf systems are mainly experimental, with lots of question marks on whether they are suited to these applications. The cost per peak Mflop/s is however very seductive. The cost of a traditional HPC system, such as the Cray T3E, Fujitsu VPP5000 or NEC SX-5 is typically 5 times more expensive than a commodity SMP system.

What is often neglected is the performance one gets from these systems. For instance taking the Compaq Alpha as having performance 100, the SGI Origin 3200 is 61, the Compaq Alpha ES40/667 is 113, the IBM RS/6000 Power 3 is 94, the SunBlade 1000/750 is 85 and the NEC SX-5 is 3975. Thus the NEC SX-5, achieves 39.75 times more performance than the baseline Alpha Chip using a Daresbury benchmark. This is why one reads (HPCWire June 2nd 2000), that on sparse matrix problems using FMSLIB, a 16 CPU NEC SX-5 delivers 126.5 Gflop/s sustained performance out of the 128Gflop/s peak. To get the same sustain performance on scalar SMPs one would need to buy a system with Tflop/s peak.

Although a lot of effort is going into proving the Beowulf paradigm, there are still a lot of issues to be resolved. It crucially includes bandwidth of interconnect, share memory size, effect of variable message passing on L2 cache, reliability of clusters, and the immaturity of LINUX with many HPC needed functions missing. These issues cumulatively add up to deliver poor performance at present. This state of affairs is, however, temporary and once the many HPC components from high end computer vendors are imported into LINUX and standardised as open source, the picture should change fundamentally. Whether performance would improve remains to be seen.

The Beowulf paradigm is also gaining momentum with a number of small companies providing build and maintenance services, for made to order systems, using commodity Chips and Interconnect of the customer’s choice. Currently Myrinet2000 and QSNet are the preferred ones.

In summary, LINUX is a disruptive technology but whether Beowulf should be considered as one is my view not proven. Although Beowulf could be a threat to scalable clusters offered presently by computer vendors, it is unlikely to have a significant impact on the Parallel Vector Processor (PVP) supercomputer line. The reason is simple. The PVP market was never strong in strapped-for-money university departments. I guess, the initial impact is likely to be seen in commodity servers, where LINUX is in the process of being customised and adopted by all the main vendors, IBM, Compaq, HP, SUN, Fujitsu and so on.

It is bemusing to observe that some of the problems of the 1970’s are revisiting computing. The programming difficulties of optimising vector codes are now re-appearing as part of multi-level cache; the challenge is how to achieve optimisation across different levels of cache without loss of coherence. The problems derived from ruefully inadequate Operating System functionality and non-standard source, are to be imported in computing production, in the guise of open source LINUX. In the past, modifying Operating Systems to enhance functionality and paying the price of unreliability, was a necessity, now we are told, it is to be done as of preferred choice.

Maybe “Supercomputing history repeats itself” or, maybe “History proceeds by changing the subject”.

Copyright: Christopher Lazou, HiPerCom Consultants, Ltd., UK. Email: [email protected]

============================================================

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industry updates delivered to you every week!

MLPerf Inference 4.0 Results Showcase GenAI; Nvidia Still Dominates

March 28, 2024

There were no startling surprises in the latest MLPerf Inference benchmark (4.0) results released yesterday. Two new workloads — Llama 2 and Stable Diffusion XL — were added to the benchmark suite as MLPerf continues Read more…

Q&A with Nvidia’s Chief of DGX Systems on the DGX-GB200 Rack-scale System

March 27, 2024

Pictures of Nvidia's new flagship mega-server, the DGX GB200, on the GTC show floor got favorable reactions on social media for the sheer amount of computing power it brings to artificial intelligence.  Nvidia's DGX Read more…

Call for Participation in Workshop on Potential NSF CISE Quantum Initiative

March 26, 2024

Editor’s Note: Next month there will be a workshop to discuss what a quantum initiative led by NSF’s Computer, Information Science and Engineering (CISE) directorate could entail. The details are posted below in a Ca Read more…

Waseda U. Researchers Reports New Quantum Algorithm for Speeding Optimization

March 25, 2024

Optimization problems cover a wide range of applications and are often cited as good candidates for quantum computing. However, the execution time for constrained combinatorial optimization applications on quantum device Read more…

NVLink: Faster Interconnects and Switches to Help Relieve Data Bottlenecks

March 25, 2024

Nvidia’s new Blackwell architecture may have stolen the show this week at the GPU Technology Conference in San Jose, California. But an emerging bottleneck at the network layer threatens to make bigger and brawnier pro Read more…

Who is David Blackwell?

March 22, 2024

During GTC24, co-founder and president of NVIDIA Jensen Huang unveiled the Blackwell GPU. This GPU itself is heavily optimized for AI work, boasting 192GB of HBM3E memory as well as the the ability to train 1 trillion pa Read more…

MLPerf Inference 4.0 Results Showcase GenAI; Nvidia Still Dominates

March 28, 2024

There were no startling surprises in the latest MLPerf Inference benchmark (4.0) results released yesterday. Two new workloads — Llama 2 and Stable Diffusion Read more…

Q&A with Nvidia’s Chief of DGX Systems on the DGX-GB200 Rack-scale System

March 27, 2024

Pictures of Nvidia's new flagship mega-server, the DGX GB200, on the GTC show floor got favorable reactions on social media for the sheer amount of computing po Read more…

NVLink: Faster Interconnects and Switches to Help Relieve Data Bottlenecks

March 25, 2024

Nvidia’s new Blackwell architecture may have stolen the show this week at the GPU Technology Conference in San Jose, California. But an emerging bottleneck at Read more…

Who is David Blackwell?

March 22, 2024

During GTC24, co-founder and president of NVIDIA Jensen Huang unveiled the Blackwell GPU. This GPU itself is heavily optimized for AI work, boasting 192GB of HB Read more…

Nvidia Looks to Accelerate GenAI Adoption with NIM

March 19, 2024

Today at the GPU Technology Conference, Nvidia launched a new offering aimed at helping customers quickly deploy their generative AI applications in a secure, s Read more…

The Generative AI Future Is Now, Nvidia’s Huang Says

March 19, 2024

We are in the early days of a transformative shift in how business gets done thanks to the advent of generative AI, according to Nvidia CEO and cofounder Jensen Read more…

Nvidia’s New Blackwell GPU Can Train AI Models with Trillions of Parameters

March 18, 2024

Nvidia's latest and fastest GPU, codenamed Blackwell, is here and will underpin the company's AI plans this year. The chip offers performance improvements from Read more…

Nvidia Showcases Quantum Cloud, Expanding Quantum Portfolio at GTC24

March 18, 2024

Nvidia’s barrage of quantum news at GTC24 this week includes new products, signature collaborations, and a new Nvidia Quantum Cloud for quantum developers. Wh Read more…

Alibaba Shuts Down its Quantum Computing Effort

November 30, 2023

In case you missed it, China’s e-commerce giant Alibaba has shut down its quantum computing research effort. It’s not entirely clear what drove the change. Read more…

Nvidia H100: Are 550,000 GPUs Enough for This Year?

August 17, 2023

The GPU Squeeze continues to place a premium on Nvidia H100 GPUs. In a recent Financial Times article, Nvidia reports that it expects to ship 550,000 of its lat Read more…

Shutterstock 1285747942

AMD’s Horsepower-packed MI300X GPU Beats Nvidia’s Upcoming H200

December 7, 2023

AMD and Nvidia are locked in an AI performance battle – much like the gaming GPU performance clash the companies have waged for decades. AMD has claimed it Read more…

DoD Takes a Long View of Quantum Computing

December 19, 2023

Given the large sums tied to expensive weapon systems – think $100-million-plus per F-35 fighter – it’s easy to forget the U.S. Department of Defense is a Read more…

Synopsys Eats Ansys: Does HPC Get Indigestion?

February 8, 2024

Recently, it was announced that Synopsys is buying HPC tool developer Ansys. Started in Pittsburgh, Pa., in 1970 as Swanson Analysis Systems, Inc. (SASI) by John Swanson (and eventually renamed), Ansys serves the CAE (Computer Aided Engineering)/multiphysics engineering simulation market. Read more…

Choosing the Right GPU for LLM Inference and Training

December 11, 2023

Accelerating the training and inference processes of deep learning models is crucial for unleashing their true potential and NVIDIA GPUs have emerged as a game- Read more…

Intel’s Server and PC Chip Development Will Blur After 2025

January 15, 2024

Intel's dealing with much more than chip rivals breathing down its neck; it is simultaneously integrating a bevy of new technologies such as chiplets, artificia Read more…

Baidu Exits Quantum, Closely Following Alibaba’s Earlier Move

January 5, 2024

Reuters reported this week that Baidu, China’s giant e-commerce and services provider, is exiting the quantum computing development arena. Reuters reported � Read more…

Leading Solution Providers

Contributors

Comparing NVIDIA A100 and NVIDIA L40S: Which GPU is Ideal for AI and Graphics-Intensive Workloads?

October 30, 2023

With long lead times for the NVIDIA H100 and A100 GPUs, many organizations are looking at the new NVIDIA L40S GPU, which it’s a new GPU optimized for AI and g Read more…

Shutterstock 1179408610

Google Addresses the Mysteries of Its Hypercomputer 

December 28, 2023

When Google launched its Hypercomputer earlier this month (December 2023), the first reaction was, "Say what?" It turns out that the Hypercomputer is Google's t Read more…

AMD MI3000A

How AMD May Get Across the CUDA Moat

October 5, 2023

When discussing GenAI, the term "GPU" almost always enters the conversation and the topic often moves toward performance and access. Interestingly, the word "GPU" is assumed to mean "Nvidia" products. (As an aside, the popular Nvidia hardware used in GenAI are not technically... Read more…

Shutterstock 1606064203

Meta’s Zuckerberg Puts Its AI Future in the Hands of 600,000 GPUs

January 25, 2024

In under two minutes, Meta's CEO, Mark Zuckerberg, laid out the company's AI plans, which included a plan to build an artificial intelligence system with the eq Read more…

Google Introduces ‘Hypercomputer’ to Its AI Infrastructure

December 11, 2023

Google ran out of monikers to describe its new AI system released on December 7. Supercomputer perhaps wasn't an apt description, so it settled on Hypercomputer Read more…

China Is All In on a RISC-V Future

January 8, 2024

The state of RISC-V in China was discussed in a recent report released by the Jamestown Foundation, a Washington, D.C.-based think tank. The report, entitled "E Read more…

Intel Won’t Have a Xeon Max Chip with New Emerald Rapids CPU

December 14, 2023

As expected, Intel officially announced its 5th generation Xeon server chips codenamed Emerald Rapids at an event in New York City, where the focus was really o Read more…

IBM Quantum Summit: Two New QPUs, Upgraded Qiskit, 10-year Roadmap and More

December 4, 2023

IBM kicks off its annual Quantum Summit today and will announce a broad range of advances including its much-anticipated 1121-qubit Condor QPU, a smaller 133-qu Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire