With several new sales recently in Italy and France, Cray Inc. is re- establishing a notable presence in the European HPC computing market. The latest sale is a Cray XT3 computer to the Swiss National Supercomputing Center (CSCS). This article uses Cray's announcement (http://news.tgc.com/msgget.jsp?mid=363011) as a starting point and intersperses comments from customers, including opinion and analysis to review this relatively new system.

For those not familiar with Swiss computing facilities and institutions, CSCS is the Swiss National Supercomputing Center, providing, developing and promoting technical and scientific services for the Swiss research community in the fields of high-performance and high throughput computing. It was created in 1991 to provide leadership-class computing for high-end science nationwide and to support breakthrough science, academic partnerships and world-class connectivity.

The Paul Scherrer Institut (PSI), located in Villigen, Switzerland, is a multi-disciplinary research center for natural sciences and technology. PSI collaborates with national and international universities, other research institutions, and industry in the areas of solid-state research and material sciences, particle physics and astrophysics, life sciences, energy research and environmental research.

ETH Zurich (Swiss Federal Institute of Technology Zurich) is professional home to 18,000 people from 80 nations. Nearly 360 professors teach mainly in engineering sciences and architecture, system-oriented sciences, mathematics and natural sciences and carry out research, highly valued worldwide. Distinguished by the successes of 21 Nobel laureates, ETH Zurich is committed to providing its students with unparalleled education and outstanding leadership skills.

To put this procurement announcement in context, last summer I listened to a presentation given by Marie-Christine Sawley, on their Roadmap (2004 to 2011) for delivering a Swiss national computing service enabling strong leadership in scientific development. Their mission is to provide the infrastructure and management for serving the scientific community, enabling them to deliver better science. The driving computing forces identified from their user survey were:

• computing capability – higher scientific quality requires intensive resources;
• service sustainability – a clear line on provision of service functions and benefits from economy of scale; and
• flexibility, the capacity to evolve computing facilities to keep pace with technology advances.

Some of the scientific “Grand Challenge” problems, which Swiss scientists are investigating, include plasma physics, material sciences, meteorology/climate and astronomy/cosmology (many body problem). Another area is computational biology. For example, simulating a heart muscle cell model using one million cells and 30 million equations requires two to three years of computing power, even when running on a computer delivering 1 Tflop/s sustained performance.

Analysis of molecular bio-informatics is, in general, data intensive while the simulation of atoms and molecules is, in general, compute-intensive. By integrating these two activities together, one has a recipe for a massive system. In the CSCS case, a need of a 10x increase in sustained performance and 5x increase in throughput relative to their current systems was identified. Their criteria for system procurement included satisfying user requirements, purchase price, assessing the ability of a vendor to deliver what is promised in the contract, site preparation costs, operating costs and total cost of ownership (TCO) over 5 years. The system on offer had to be a production, not a development one.  “With Horizon, CSCS will implement an important component of its Roadmap strategy unveiled in November 2003. This is achieved by making early systems of powerful new supercomputing architectures available to the national research community and international collaborators as advanced scientific instruments,” said Sawley, after the order for the Cray XT3 was finalized.

“This new state-of-the-art scientific instrument will allow us to simulate complex problems that were only recently considered intractable. Problems that once would have taken months to complete will now take hours or minutes,” said Andreas Adelmann, the initiator of the Horizon project.

From the above statements the message is clear. Horizon is a highly scalable capability system for very demanding, high-end computational scientific and engineering research applications. The system is designed to support a broad range of applications and positions CSCS, as a leadership- class computing resource supplier for the research community of Switzerland. It is also positioned to attract highly visible, value added international collaborations.

CSCS plans to team up in collaborations with leading US supercomputing sites Pittsburgh Supercomputing Center, Oak Ridge National Laboratory, and Sandia National Laboratories to make the Cray XT3 technology mature for a broad spectrum of scientific production work.

These centers have already initiated collaboration and formed the SOS consortium. On March 21-23, there was an SOS workshop, attendance by invitation only, in Davos, Switzerland. The workshop theme was “Science and Supercomputers.” The perceived wisdom is that, “Today science is enabled by supercomputing, but tomorrow, science breakthroughs will be driven by supercomputers.” The workshop explored what is needed to prepare for an age when manipulating huge data sets and simulating complex physical phenomena is used routinely to predict and explain new scientific phenomena. The workshop also examined the computational characteristics needed to facilitate this transition and how can the existing and emerging supercomputer architectures be directed to help science. For further details go to the CSCS website: www.cscs.ch/form/SOS.php.

CSCS is also establishing a joint development program with Cray in order to develop and optimize applications and make contributions to the software technology specific to the Cray XT3. Therefore, apart from the base Horizon system, CSCS will receive an additional Cray XT3 rack with two compute cabinets this April. This will be used for a development partnership between CSCS and Cray.

What makes the Cray XT3 system different? Bill Camp, Computing Director, Sandia Laboratories, who in the best innovative tradition, took a risk and pursued his vision to co-develop (by Sandia and Cray) the original Red Storm technology, made the following observation last June at the International Supercomputer Conference in Europe.

“For most large scientific and engineering applications the performance is more determined by parallel scalability and less by the speed of individual CPUs,” said Camp. “There must be balance between processor, interconnect, and I/O performance to achieve overall performance. To-date, only a few tightly coupled, parallel computer systems have been able to demonstrate a high level of scalability on a broad set of scientific and engineering applications.”

“Once parallelism requires more than 64 to 128 processors the MPP architecture is more cost efficient,” Camp continued. “For the five codes using more than 80% of Sandia's computer cycles, the average efficiency ratio was 1.4 times compared to clusters.”

Camp's philosophy is to use commodity nearly everywhere, as customization tends to drive costs up.

“The Earth Simulator and the Cray X-1 are fully custom vector parallel systems with good balance,” stated Camp. “This drives their initial high cost (and their high performance). Clusters are nearly entirely high volume with no truly custom parts, which drives their low cost (and their low scalability). Red Storm uses custom parts only where they are critical to performance and reliability. This translates to high scalability at minimal cost/performance.”

“When one compares the top configurations, a 100Tflop/s version of the Cray XT3 (Red Storm) with the 360Tflop/s version of the IBM Blue Gene/L, one can see that although the node speed is the same, many of the other system components differ giving Red Storm an edge in architectural balance,” Camp said. “For example, the memory per node is 4 to 16 times larger, the network latency is about 3 times smaller, the network bandwidth per link is about 25 times faster, the bandwidth (bytes/flop) is also about 25 times better and the bi-section (bytes/flop) is over 20 times better on the Cray XT3 (Red Storm) system compared to the Blue Gene/L.”

Although the IBM Blue Gene/L performs very highly on Linpack, is compact and has low power consumption, one awaits with bated breath to see how 'well' it performs on the more balanced HPCC benchmark tests.

According to a recent report I read, the Cray XT3 bandwidth is also outstanding compared with other commodity based systems. For example, in relation to the IBM P4 + Federation and the IBM P5-595, the Cray XT3 bandwidth for a stream triad (bytes/flop) is about 5 times and 2.7 times faster, respectively.

According to Cray Inc., the Cray XT3 system is designed to offer maximum reliability to its users, with a large mean time between interruptions (MTBI) for any user process. Based on the Red Storm technology co-developed by Sandia National Laboratories and Cray (with strong reference to the earlier Cray T3E and ASCI Red architectures), the Cray XT3 comprises technical features for new levels of scalability and sustained application performance. The SeaStar, 3D- torus interconnect gives Horizon a peak bisection bandwidth of 14.1TeraBytes/s. A “special” micro-kernel on the compute node minimizes operating system overhead and operating system jitter, which impedes computational applications on large-scale clusters from scaling to thousands of CPUs.

As reported in my January article (http://news.taborcommunications.com/msgget.jsp?mid=328047&xsl=story.xsl), the performance results from the HPC Challenge benchmark clearly show the strength of highly integrated systems with high bandwidth, low latency memory and network subsystems. The Cray XT3, with its specially engineered SeaStar, 3D- torus interconnect, is but one exemplar.

(Brands and names and trademarks are the property of their respective owners). Copyright: Christopher Lazou, HiPerCom Consultants, Ltd., UK. April 2005