Visit additional Tabor Communication Publications
October 19, 2007
On Monday, the Department of Energy's (DOE's) Argonne National Laboratory became the proud owners of the first SiCortex SC5832 system deployed in the field. Introduced in November 2006, the SC5832, along with its smaller sibling, the SC648, represent a new approach to high performance computing. The SiCortex machines have garnered a good deal of critical acclaim over the past year, but this will be the first time the community will be able to see one operate with real applications. The 5.8 teraflop machine will be operated by Argonne's Mathematics and Computer Science (MCS) Division to further its mission in researching and developing software for high performance computing architectures.
The SiCortex architecture represents a departure from the traditional commodity cluster computing model. The company has custom engineered everything from the chips to the chassis to produce something more akin to a mini-supercomputer than an HPC cluster. The SC5832 is comprised of 5832 MIPS64 cores, each dissipating just 600 milliwatts of power. Each processor node consists of 6 MIPS64 cores, two DDR-2 memory controllers, a DMA engine, a PCI Express I/O controller and an internode fabric switch. There are no cables to speak of; the nodes communicate via the backplane.
To the application user, it appears to be a conventional HPC system, with the Linux operating system and the traditional MPI interface. Its MIPS64 foundation is hidden beneath the SiCortex PathScale compiler. The architecture's main claim to fame is its low power consumption. The SC5832 needs just 18 kilowatts. Even clusters based on the newest energy-sipping quad-core Opterons will need somewhere north of 28 kilowatts for the same 5.8 teraflops.
According to the team at Argonne, the SC5832 was a slam-dunk to install. The machine came in the door on Monday morning and was assembled and up running within two and a half hours. "Of all the new machine installations I have been through here at Argonne, this was by far the smoothest," said Narayan Desai, Argonne National Laboratory systems engineer. The whole system takes up just over 130 cubic feet of space -- about the size of four large refrigerators.
The unique attributes of the SiCortex machine were a big draw for Argonne, a lab with a reputation for trying out innovative hardware. Concurrent with the SiCortex deployment, Argonne's new Blue Gene/P system is also in the process of being installed. Both systems represent different scales of similar approaches, namely creating high levels of performance with large numbers of low-power RISC processors, hooked together with a high bandwidth communication fabric. As such, these systems achieve exceptional performance-per-watt, something Argonne and other DOE labs are becoming increasingly focused on.
"Here at Argonne, we've been exploring and pushing the envelope with aggressive machines, but our end goal is really to be fielding systems that are energy efficient," explains Pete Beckman, computer scientist with Argonne's MCS Division. "As we scale up to exaflops, computing power can't scale up in the same way as it has in the past. We certainly won't be able to have 40 megawatt machine rooms. The SiCortex and Blue Gene/P machines both represent this new model of lower power, more cores, improved fault tolerance and RAS to enable petascale computing and beyond."
Both machines will be used to support the MCS Division's research in parallel computing. But while the Blue Gene/P machine is primarily an evolution of the Blue Gene/L architecture, the SiCortex system is being evaluated as a new architecture to replace conventional HPC clusters. Part of that evaluation will be to determine the feasibility of retargeting traditional cluster applications to the SC5832.
This could offer a few challenges. Compared to conventional x86-based systems of similar performance, the SiCortex machine contains many more processor cores. Since SiCortex has made the tradeoff of replacing relatively power-hungry x86 cores (8 GLOPS/core for a 2 GHz Opteron) with low-power MIPS64 cores (1 GLOPS/core for a 0.5 GHz processor), they need about eight times as many cores to achieve the equivalent raw performance. That means the software has to be able to scale to greater levels to take advantage of the larger number of processing units. A traditional fat-node cluster architecture is also likely to have more memory per processor than the more Blue Gene-like SiCortex architecture. Cluster applications that have made liberal use of memory may need to distribute their data more intelligently to run in this new environment.
If one's codes are already running on a 2000-plus node cluster or an IBM Blue Gene type machine, then the software is likely to be scaled to these levels already. Argonne, being a leadership computing facility with a lot of Blue Gene experience, has plenty of highly parallelized code. By Wednesday, Argonne users were already running applications on the newly installed SiCortex machine, in some cases using more than 90 percent of the SC5832's cores. These applications included nek5000 (fluid dynamics) on 24 cores, FLASH (astrophysics) on 5832 cores, PETSc (PDE library) on 5802 cores, and Pneo (neuroscience) on 3600 cores. Although the groups that own these codes have their own production machines at the lab, they were grateful to grab some cycles on the new SiCortex system. Other "standard" applications were also exercised, including HPL on 5776 cores, NAMD on 4800 cores, and POP on 4324 cores.
One of the more compelling characteristics of the SiCortex systems is that it's almost completely open source. All the system software components, including the MPI library (Argonne's MPICH, actually), the operating system (Linux), the drivers for the machine's communication fabric, and the resource management system, are accessible to the user. By contrast, many cluster vendors offer more tightly integrated solutions, which include proprietary implementations of their software stack -- the vendor's so-called secret sauce. SiCortex seems convinced that by opening up the system, the HPC user community will be encouraged to build up the software ecosystem around its new architecture.
The Argonne folks love the open source approach. It means they can replace components and patch the software as easily as with their own codes. According to Ewing (Rusty) Lusk, director of Argonne's MCS Division, for research purposes, the open source model is ideal for them. He says they've already replaced a few Linux system components for their own customization of the software stack. As a result, the system will be able to do quick network booting via the Parallel Virtual File System (PVFS).
"These guys have really drunk the Kool-Aid on open source," says Lusk. "Every aspect of the source code, except perhaps the innards of the PathScale compiler itself, is open source. In terms of having a machine that you can look at, understand and replace parts of, there couldn't be anything better."
Jun 19, 2013 |
Supercomputer architectures have evolved considerably over the last 20 years, particularly in the number of processors that are linked together. One aspect of HPC architecture that hasn't changed is the MPI programming model.
Jun 18, 2013 |
The world's largest supercomputers, like Tianhe-2, are great at traditional, compute-intensive HPC workloads, such as simulating atomic decay or modeling tornados. But data-intensive applications--such as mining big data sets for connections--is a different sort of workload, and runs best on a different sort of computer.
Jun 18, 2013 |
Researchers are finding innovative uses for Gordon, the 285 teraflop supercomputer housed at the San Diego Supercomputer Center (SDSC) that has a unique Flash-based storage system. Since going online, researchers have put the incredibly fast I/O to use on a wide variety of workloads, ranging from chemistry to political science.
Jun 17, 2013 |
The advent of low-power mobile processors and cloud delivery models is changing the economics of computing. But just as an economy car is good at different things than a full size truck, an HPC workload still has certain computing demands that neither the fastest smartphone nor the most elastic cloud cluster can fulfill.
Jun 14, 2013 |
For all the progress we've made in IT over the last 50 years, there's one area of life that has steadfastly eluded the grasp of computers: understanding human language. Now, researchers at the Texas Advanced Computing Center (TACC) are utilizing a Hadoop cluster on its Longhorn supercomputer to move the state of the art of language processing a little bit further.
05/10/2013 | Cleversafe, Cray, DDN, NetApp, & Panasas | From Wall Street to Hollywood, drug discovery to homeland security, companies and organizations of all sizes and stripes are coming face to face with the challenges – and opportunities – afforded by Big Data. Before anyone can utilize these extraordinary data repositories, however, they must first harness and manage their data stores, and do so utilizing technologies that underscore affordability, security, and scalability.
04/15/2013 | Bull | “50% of HPC users say their largest jobs scale to 120 cores or less.” How about yours? Are your codes ready to take advantage of today’s and tomorrow’s ultra-parallel HPC systems? Download this White Paper by Analysts Intersect360 Research to see what Bull and Intel’s Center for Excellence in Parallel Programming can do for your codes.
Join HPCwire Editor Nicole Hemsoth and Dr. David Bader from Georgia Tech as they take center stage on opening night at Atlanta's first Big Data Kick Off Week, filmed in front of a live audience. Nicole and David look at the evolution of HPC, today's big data challenges, discuss real world solutions, and reveal their predictions. Exactly what does the future holds for HPC?
Join our webinar to learn how IT managers can migrate to a more resilient, flexible and scalable solution that grows with the data center. Mellanox VMS is future-proof, efficient and brings significant CAPEX and OPEX savings. The VMS is available today.