January 31, 2014

‘Edison’ Lights Up Research at NERSC

Tiffany Trader

The National Energy Research Scientific Computing (NERSC) Center, located at Lawrence Berkeley National Laboratory, has taken acceptance of “Edison,” a Cray XC30 supercomputer named in honor of famed American inventor Thomas Alva Edison.


The important milestone occurs just as NERSC is commemorating 40 years of scientific advances, prompting NERSC Director Sudip Dosanjh to comment: “As we celebrate NERSC’s 40th anniversary, it’s quite fitting we start the year by dedicating Edison, a system that embodies our guiding principle over the last four decades: computing in the service of science.”

Edison has a peak computational output of 2.4 petaflops, but NERSC officials like to emphasize the machine’s sustained performance on real applications. NERSC’s 5,000 researchers publish an average of 1,700 peer-reviewed research articles every year, surpassing the productivity of all other DOE Office of Science computing centers.

In line with supporting productive science for NERSC’s diverse community of researchers, Edison was built and configured to accommodate both traditional modeling and simulation workloads as well as the growing body of data analytics work.

In an earlier interview with HPCwire, Jeff Broughton, NERSC deputy for operations and leader of the Edison procurement, reported that NERSC’s user community is seeing “a big increase in data-intensive computing, with a focus on high throughput and single-node performance.” Thus the center needed a system that could “meet the needs of data-intensive applications while continuing to support conventional HPC.”

Tracing the changing paradigm, Director Dosanjh explains that historically NERSC exported most of its data, but that situation has flipped. NERSC still transmits data from large-scale simulations to other sites, but there’s even more experimental data coming in the door, such that NERSC is now a net importer. The center takes in a petabyte of data each month, much of it related to the biosciences, climate and high-energy physics.

Accomplishing this mixed-mode capability requires a balanced approach to system design. Simulating and modeling complex phenomena has always pushed the limits of computation. Such workloads benefit from a large number of tightly-coupled processors, but don’t necessarily need a lot of memory per CPU. Data analysis problems, like genome sequencing and searching for new materials, on the other hand, perform better with more memory per node. At one time, analysis workloads were compatible with small clusters, but with increasing data loads, they are outgrowing the smaller machines.

“If you have a computing resource like Edison, one with the flexibility to run different classes of problems, then you can apply the full capacity of your system to the problem at hand, whether that be high-throughput genome sequencing or highly parallel climate simulations,” Broughton explains.

To ensure optimal results across application types, data movement cannot be constrained. To that point, the Cray XC30 machine has been outfitted with a high speed interconnect, very high memory bandwidth as well as high memory per node. It also has very high input/output speeds (140 gigabytes per second) to the file system and disk system.

“Data movement is the limiting factor for a large fraction of the 600 codes that run at NERSC,” Dosanjh told HPCwire in April. “Floating point units are often idle waiting for data to arrive. Many codes spend a few percent of their total runtime performing floating point operations, the rest of the time is spent accessing memory or calculating memory addresses. Edison will be a very effective platform for running the very broad range of science codes at NERSC.”

An official dedication ceremony for Edison will be held at the Department of Energy’s Lawrence Berkeley National Laboratory (Berkeley Lab) on Feb. 5.

Here is a snapshot of the system’s specs:

  • 332 terabytes memory
  • 2.39 petaflops of peak performance
  • 124,608 processing cores
  • 462 terabytes/second global memory bandwidth
  • 11 terabytes/second network bisection bandwidth
  • 7.56 petabytes disk storage
  • 163 gigabytes/second I/O bandwidth