On Monday, the Department of Energy's (DOE's) Argonne National Laboratory became the proud owners of the first SiCortex SC5832 system deployed in the field. Introduced in November 2006, the SC5832, along with its smaller sibling, the SC648, represent a new approach to high performance computing. The SiCortex machines have garnered a good deal of critical acclaim over the past year, but this will be the first time the community will be able to see one operate with real applications. The 5.8 teraflop machine will be operated by Argonne's Mathematics and Computer Science (MCS) Division to further its mission in researching and developing software for high performance computing architectures.

The SiCortex architecture represents a departure from the traditional commodity cluster computing model. The company has custom engineered everything from the chips to the chassis to produce something more akin to a mini-supercomputer than an HPC cluster. The SC5832 is comprised of 5832 MIPS64 cores, each dissipating just 600 milliwatts of power. Each processor node consists of 6 MIPS64 cores, two DDR-2 memory controllers, a DMA engine, a PCI Express I/O controller and an internode fabric switch. There are no cables to speak of; the nodes communicate via the backplane.

To the application user, it appears to be a conventional HPC system, with the Linux operating system and the traditional MPI interface. Its MIPS64 foundation is hidden beneath the SiCortex PathScale compiler. The architecture's main claim to fame is its low power consumption. The SC5832 needs just 18 kilowatts. Even clusters based on the newest energy-sipping quad-core Opterons will need somewhere north of 28 kilowatts for the same 5.8 teraflops.

According to the team at Argonne, the SC5832 was a slam-dunk to install. The machine came in the door on Monday morning and was assembled and up running within two and a half hours. "Of all the new machine installations I have been through here at Argonne, this was by far the smoothest," said Narayan Desai, Argonne National Laboratory systems engineer. The whole system takes up just over 130 cubic feet of space -- about the size of four large refrigerators.

The unique attributes of the SiCortex machine were a big draw for Argonne, a lab with a reputation for trying out innovative hardware. Concurrent with the SiCortex deployment, Argonne's new Blue Gene/P system is also in the process of being installed. Both systems represent different scales of similar approaches, namely creating high levels of performance with large numbers of low-power RISC processors, hooked together with a high bandwidth communication fabric. As such, these systems achieve exceptional performance-per-watt, something Argonne and other DOE labs are becoming increasingly focused on.

"Here at Argonne, we've been exploring and pushing the envelope with aggressive machines, but our end goal is really to be fielding systems that are energy efficient," explains Pete Beckman, computer scientist with Argonne's MCS Division. "As we scale up to exaflops, computing power can't scale up in the same way as it has in the past. We certainly won't be able to have 40 megawatt machine rooms. The SiCortex and Blue Gene/P machines both represent this new model of lower power, more cores, improved fault tolerance and RAS to enable petascale computing and beyond."

Both machines will be used to support the MCS Division's research in parallel computing. But while the Blue Gene/P machine is primarily an evolution of the Blue Gene/L architecture, the SiCortex system is being evaluated as a new architecture to replace conventional HPC clusters. Part of that evaluation will be to determine the feasibility of retargeting traditional cluster applications to the SC5832.

This could offer a few challenges. Compared to conventional x86-based systems of similar performance, the SiCortex machine contains many more processor cores. Since SiCortex has made the tradeoff of replacing relatively power-hungry x86 cores (8 GLOPS/core for a 2 GHz Opteron) with low-power MIPS64 cores (1 GLOPS/core for a 0.5 GHz processor), they need about eight times as many cores to achieve the equivalent raw performance. That means the software has to be able to scale to greater levels to take advantage of the larger number of processing units. A traditional fat-node cluster architecture is also likely to have more memory per processor than the more Blue Gene-like SiCortex architecture. Cluster applications that have made liberal use of memory may need to distribute their data more intelligently to run in this new environment.

If one's codes are already running on a 2000-plus node cluster or an IBM Blue Gene type machine, then the software is likely to be scaled to these levels already. Argonne, being a leadership computing facility with a lot of Blue Gene experience, has plenty of highly parallelized code. By Wednesday, Argonne users were already running applications on the newly installed SiCortex machine, in some cases using more than 90 percent of the SC5832's cores. These applications included nek5000 (fluid dynamics) on 24 cores, FLASH (astrophysics) on 5832 cores, PETSc (PDE library) on 5802 cores, and Pneo (neuroscience) on 3600 cores. Although the groups that own these codes have their own production machines at the lab, they were grateful to grab some cycles on the new SiCortex system. Other "standard" applications were also exercised, including HPL on 5776 cores, NAMD on 4800 cores, and POP on 4324 cores.

One of the more compelling characteristics of the SiCortex systems is that it's almost completely open source. All the system software components, including the MPI library (Argonne's MPICH, actually), the operating system (Linux), the drivers for the machine's communication fabric, and the resource management system, are accessible to the user. By contrast, many cluster vendors offer more tightly integrated solutions, which include proprietary implementations of their software stack -- the vendor's so-called secret sauce. SiCortex seems convinced that by opening up the system, the HPC user community will be encouraged to build up the software ecosystem around its new architecture.

The Argonne folks love the open source approach. It means they can replace components and patch the software as easily as with their own codes. According to Ewing (Rusty) Lusk, director of Argonne's MCS Division, for research purposes, the open source model is ideal for them. He says they've already replaced a few Linux system components for their own customization of the software stack. As a result, the system will be able to do quick network booting via the Parallel Virtual File System (PVFS).

"These guys have really drunk the Kool-Aid on open source," says Lusk. "Every aspect of the source code, except perhaps the innards of the PathScale compiler itself, is open source. In terms of having a machine that you can look at, understand and replace parts of, there couldn't be anything better."