The DOE's National Energy Research Scientific Computing Center (NERSC) recently announced that Cray won the bid to deliver a 100-teraflop, $52 million supercomputer that can be significantly expanded over time. Bill Kramer, general manager of the NERSC center, talked with HPCwire about the evaluation criteria and how Cray came out ahead in performance and price/performance.

HPCwire: Talk about the NERSC user base and workload.

Kramer: NERSC's mission is to support not just big science, but the entire range of open science. We serve the largest, most diverse group of users within the DOE. Each year, we support about 2,500 users working on 300-400 projects. Our users produce 1,200-1,400 peer-reviewed scientific papers every year.

HPCwire: Does that include the INCITE program?

Kramer: Absolutely. We've been involved in INCITE since the program began four years ago. The DOE approached NERSC to be the sole site for the prototype program, which was called "Big Splash," and that worked out well enough to justify the larger INCITE program. In 2005, 15 percent of our cycles were dedicated to three INCITE projects.

HPCwire: Other than your big, diverse user base, what sets NERSC apart from other DOE centers and labs?

Kramer: We're leaders in extreme scaling with high utilization. We've run codes efficiently on our full 6,000-CPU "Seaborg" Power system, and in 2003 we established our Scaling Project, with the aim of making it possible to run codes on tens of thousands of processors, and eventually hundreds of thousands of processors, with high utilization. The project identified codes with good potential for running at very large scales, and also identified the bottlenecks that would need to be addressed for the codes to scale efficiently. This work is also important for the INCITE program.

HPCwire: You have some of the world's best performance evaluation experts, people like Lenny Oliker and Kathy Yelick.

Kramer: We're fortunate to have access to them. On the performance modeling side, we have Erich Strohmaier at NERSC and do work with Alan Snavely in San Diego.

HPCwire: How often does NERSC procure big systems?

Kramer: We go through a major, full-competitive system procurement every three years or so. This is the biggest investment we make as a center.

HPCwire: Can you describe the benchmarks you used in your latest procurement?

Kramer: They're our own. We call them the Sustained System Performance Metric, or SSP for short. The suite includes seven key applications drawn from NERSC's workload. We ask vendors to run the applications at three different sizes, from 256 to 2048 CPUs, then we combine the system's performance by taking the geometric mean of the performance rates, without any weighting.

HPCwire: Does the mix of applications ever change?

Kramer: Each procurement is different as the workload changes, but there is also consistency. Usage by some disciplines gets more and some less. Astrophysics is getting more emphasis right now, for example, and climate temporarily lessened a bit after an important phase of IPCC computational work was completed. At any given time, the SSP metric represents 85 percent of our disciplines.

HPCwire: How do you account for price to come up with price/performance scores?

Kramer: We multiply the sustained SSP performance per CPU by the number of CPUs in the proposed system, and we look at this over a three-year period, since a system may evolve over the first several years it is installed. This gives us a figure for the total number of sustained teraflops we can expect from the system during the three years until our next major procurement. We then divide this SSP number by the total cost of the system, which includes software and electricity but not my staff costs for the three years. The result is expressed as teraflop-years per dollar.

HPCwire: And Cray won.

Kramer: Cray came out ahead on both total performance and price/performance. We're not disclosing price/performance results for any of the vendors, but on the performance side we were looking for between 7.5 and 10 sustained teraflops across our SSP benchmarks. The Cray system proposed was above that range, which we thought was very impressive. Now that we have finished our discussions with Cray, we now expect the system to have an SSP of 16.1 teraflops.

HPCwire: Were you satisfied with your SSP benchmarks themselves?

Kramer: There's always room for improvement, but overall the SSP benchmarks did a nice job of meeting our requirements. We think benchmarks have four purposes. First, they need to be able to evaluate candidate systems. Second, they need to be able to validate that you're getting the system performance you expected to get. Third, they need to be able to do this throughout the machine's lifetime, because we have seen cases where performance degrades over time for systems, so unless you have a way to monitor it formally, it gets complicated figuring out what happened. Finally, benchmarks should be able to provide guidance, especially to vendors, for designing future systems. Most well-known benchmarks look mostly at the first use, but the SSP does the first three very well, and can be used for the fourth.

HPCwire: Do you collaborate with anyone else in constructing your SSP benchmarks?

Kramer: We're now coordinating across agencies for the first time. That includes the DOE, NSF and the DOD MOD program. You might remember that the NRC and HECRTF reports recommended that agencies coordinate more of their activities. The goal here is to identify areas of overlap, where we can use common benchmarks. This can save time in procurements for us and for the vendors. We've looked at this with the other agencies. Right now, the overlap isn't large because our users' needs are fairly different, but we did adopt some things in common. For example, for chemistry we're using the DOD's GAMESS application in our benchmark suite. More of this will happen over time.

HPCwire: Did you look at using any synthetic benchmarks, such as the ones Alan Snavely and his team have been working on?

Kramer: We're certainly open to that. For now, we agree with the DOD MOD program that synthetic benchmarks have a lot of promise but are too new to rely heavily on for making procurement decisions. The accuracy of modeling needs to be within five percent to be useful in making a real decision. Once there is stronger correlation with actual application results, synthetic benchmarks will have a more important role to play.

HPCwire: Do the vendors also benefit from going through a procurement process like yours?

Kramer: I think there have been some important benefits for Cray. The process helped to crystallize Cray's software roadmap. For example, Cray is planning to use Berkeley Lab's checkpoint/restart as the basis for their implementation.

NERSC and Cray will also benefit in the area of the new petascale I/O interface. It will take about 12 to 18 months of work to fully realize, but then Cray will be able to integrate their system with the NERSC Global Filesystem, a high performance, facility-wide file system, based on the GPFS system, that we're using with all of our architectures. This will make Cray more portable and better able to integrate into a variety of existing environments. Our contract calls for the establishment of a Cray Center of Excellence at NERSC. The first two areas that will be addressed through this collaborative center are system management and storage management.

HPCwire: In closing, how would you characterize this procurement?

Kramer: We think it worked really well. We're getting an excellent system from Cray that will have a tremendous positive impact on our users. As measured on our SSP benchmark, the Cray system will boost our computational power by 9x over today. That's a major improvement. It will also be fun to work with Cray to help make the new "Hood" system really solid for NERSC and the broader community.