The Leading Source for Global News and Information Covering the Ecosystem of High Productivity Computing
October 19, 2007
SiCortex Machine Gets Warm Reception at Argonne
On Monday, the Department of Energy's (DOE's) Argonne National Laboratory became the proud owners of the first SiCortex SC5832 system deployed in the field. Introduced in November 2006, the SC5832, along with its smaller sibling, the SC648, represent a new approach to high performance computing. The SiCortex machines have garnered a good deal of critical acclaim over the past year, but this will be the first time the community will be able to see one operate with real applications. The 5.8 teraflop machine will be operated by Argonne's Mathematics and Computer Science (MCS) Division to further its mission in researching and developing software for high performance computing architectures.
The SiCortex architecture represents a departure from the traditional commodity cluster computing model. The company has custom engineered everything from the chips to the chassis to produce something more akin to a mini-supercomputer than an HPC cluster. The SC5832 is comprised of 5832 MIPS64 cores, each dissipating just 600 milliwatts of power. Each processor node consists of 6 MIPS64 cores, two DDR-2 memory controllers, a DMA engine, a PCI Express I/O controller and an internode fabric switch. There are no cables to speak of; the nodes communicate via the backplane.
To the application user, it appears to be a conventional HPC system, with the Linux operating system and the traditional MPI interface. Its MIPS64 foundation is hidden beneath the SiCortex PathScale compiler. The architecture's main claim to fame is its low power consumption. The SC5832 needs just 18 kilowatts. Even clusters based on the newest energy-sipping quad-core Opterons will need somewhere north of 28 kilowatts for the same 5.8 teraflops.
According to the team at Argonne, the SC5832 was a slam-dunk to install. The machine came in the door on Monday morning and was assembled and up running within two and a half hours. "Of all the new machine installations I have been through here at Argonne, this was by far the smoothest," said Narayan Desai, Argonne National Laboratory systems engineer. The whole system takes up just over 130 cubic feet of space -- about the size of four large refrigerators.
The unique attributes of the SiCortex machine were a big draw for Argonne, a lab with a reputation for trying out innovative hardware. Concurrent with the SiCortex deployment, Argonne's new Blue Gene/P system is also in the process of being installed. Both systems represent different scales of similar approaches, namely creating high levels of performance with large numbers of low-power RISC processors, hooked together with a high bandwidth communication fabric. As such, these systems achieve exceptional performance-per-watt, something Argonne and other DOE labs are becoming increasingly focused on.
"Here at Argonne, we've been exploring and pushing the envelope with aggressive machines, but our end goal is really to be fielding systems that are energy efficient," explains Pete Beckman, computer scientist with Argonne's MCS Division. "As we scale up to exaflops, computing power can't scale up in the same way as it has in the past. We certainly won't be able to have 40 megawatt machine rooms. The SiCortex and Blue Gene/P machines both represent this new model of lower power, more cores, improved fault tolerance and RAS to enable petascale computing and beyond."
Both machines will be used to support the MCS Division's research in parallel computing. But while the Blue Gene/P machine is primarily an evolution of the Blue Gene/L architecture, the SiCortex system is being evaluated as a new architecture to replace conventional HPC clusters. Part of that evaluation will be to determine the feasibility of retargeting traditional cluster applications to the SC5832.
This could offer a few challenges. Compared to conventional x86-based systems of similar performance, the SiCortex machine contains many more processor cores. Since SiCortex has made the tradeoff of replacing relatively power-hungry x86 cores (8 GLOPS/core for a 2 GHz Opteron) with low-power MIPS64 cores (1 GLOPS/core for a 0.5 GHz processor), they need about eight times as many cores to achieve the equivalent raw performance. That means the software has to be able to scale to greater levels to take advantage of the larger number of processing units. A traditional fat-node cluster architecture is also likely to have more memory per processor than the more Blue Gene-like SiCortex architecture. Cluster applications that have made liberal use of memory may need to distribute their data more intelligently to run in this new environment.
If one's codes are already running on a 2000-plus node cluster or an IBM Blue Gene type machine, then the software is likely to be scaled to these levels already. Argonne, being a leadership computing facility with a lot of Blue Gene experience, has plenty of highly parallelized code. By Wednesday, Argonne users were already running applications on the newly installed SiCortex machine, in some cases using more than 90 percent of the SC5832's cores. These applications included nek5000 (fluid dynamics) on 24 cores, FLASH (astrophysics) on 5832 cores, PETSc (PDE library) on 5802 cores, and Pneo (neuroscience) on 3600 cores. Although the groups that own these codes have their own production machines at the lab, they were grateful to grab some cycles on the new SiCortex system. Other "standard" applications were also exercised, including HPL on 5776 cores, NAMD on 4800 cores, and POP on 4324 cores.
One of the more compelling characteristics of the SiCortex systems is that it's almost completely open source. All the system software components, including the MPI library (Argonne's MPICH, actually), the operating system (Linux), the drivers for the machine's communication fabric, and the resource management system, are accessible to the user. By contrast, many cluster vendors offer more tightly integrated solutions, which include proprietary implementations of their software stack -- the vendor's so-called secret sauce. SiCortex seems convinced that by opening up the system, the HPC user community will be encouraged to build up the software ecosystem around its new architecture.
The Argonne folks love the open source approach. It means they can replace components and patch the software as easily as with their own codes. According to Ewing (Rusty) Lusk, director of Argonne's MCS Division, for research purposes, the open source model is ideal for them. He says they've already replaced a few Linux system components for their own customization of the software stack. As a result, the system will be able to do quick network booting via the Parallel Virtual File System (PVFS).
"These guys have really drunk the Kool-Aid on open source," says Lusk. "Every aspect of the source code, except perhaps the innards of the PathScale compiler itself, is open source. In terms of having a machine that you can look at, understand and replace parts of, there couldn't be anything better."
Article Tools
Share Options
(Digg, Technorati, more)
Subscribe
Sponsored Links
RSS Feeds
Feeds by Topic
- Applications
- Developer Tools
- Interconnects
- Middleware
- Networks
- Processors
- Storage
- Systems
- Visualization
Feeds by Industry
Feeds by Content Type
Top Headlines
3D Seismic Data: Taking a Smarter Approach to Interpretation
Jul 09 | Engineer Live | The demand for computational tools to underpin the 3D seismic interpretation process has never been more apparent. Read more...
Engineering Unemployment Soared in 2Q to 8.6%
Jul 08 | EE Times | Unemployment for U.S. engineers has reached record levels, according to government figures. Read more...
Gartner Adjusts 2009 IT Spend Downward Again
Jul 08 | Network World | Global spending for 2009 projected to drop 6 percent, for a total of $3.2 trillion. Read more...
Concurrent and Parallel Are Not The Same
Jul 08 | Linux Magazine | Portability or efficiency? Neither is guaranteed when writing explicit parallel code. Read more...
800 TFLOP Real-Time Ray Tracing GPU Unveiled, Not for Gamers
Jul 07 | Ars Technica | Japanese company builds custom ASIC to accelerate real-time ray traced rendering for the auto industry. Read more...
Featured Whitepapers
Building High Performance Computing in a Green and Modular Solution Building Block
Apr 14 | | Many HPC IT departments are feeling the rising pressure to deliver more capacity computing and performance while trying to reduce the total cost of ownership. This white paper discusses how an environmentally-friendly and open-standards HPC building block based computing system using flexible interconnect options helps address capacity computing needs.
Multimedia
Webcast: Dell Expands HPC Access and Adoption with Intel Cluster Ready Program
Source: Addison Snell, GM/VP, Tabor Research; sponsored by Dell
Many organizations that could benefit from the use of HPC clusters find that it is complicated to get the systems up and running because of limited IT resources or the complexities of the clusters themselves. Learn how the Intel Cluster Ready program, for which Dell was an original partner, seeks to address this challenge for entry level and mid-range HPC users.
Video White Paper: Architecting a Better Network Storage Solution
BlueArc's Titan architecture represents an evolutionary step in file servers by creating a hardware-based file system that can scale bandwidth, IOPS, and overall data capacity well beyond conventional software-based devices. With its ability to virtualize a massive storage pool of up to four usable petabytes of tiered storage, Titan can scale with growing data requirements, offering a competitive advantage for businesses, researchers, or other enterprises seeking to better manage data growth while still ensuring optimal performance.
Webcast: HPC Development Solutions: Sun Studio & Sun HPC ClusterTools
Sun Studio Compilers and Tools and Sun HPC ClusterTools allow you to create high performance parallel applications for OpenSolaris, Solaris and Linux. Sun Studio Express 11/08 includes MPI performance analysis capabilities and full OpenMP 3.0 compiler support. Learn about all this and the latest in Sun HPC ClusterTools 8.1.
Newsletters
Stay informed! Subscribe to HPCwire email Newsletters.
HPC Job Bank
-
HPC Systems Administrator
McGill University -
High Performance Computing Data Center Manager
National Renewable Energy Laboratory -
High Performance Computing Systems Administrator
Idaho National Laboratory -
Solution Engineer- HPC
Dell Inc -
Parallel Programmer
Cardiff University -
Principal Software Research Engineer
Indiana University -
Post Doctoral Researcher - Exascale Stream Computing
IBM - Visit the HPCwire Job Bank
Featured Events
-
July 13-16, 2009
WORLDCOMP'09
Las Vegas , NV
USA
-
August 24-28, 2009
Data Mining: Levels I, II, & III
Denver , CO
USA