On November 12, 2015, the Department of Energy’s (DOE) Lawrence Berkeley National Laboratory officially opened its state-of-the-art facility for computational science built on a hillside overlooking the UC Berkeley campus and San Francisco Bay. The modern building, named Wang Hall in honor of noted semiconductor researcher and long-time UC Berkeley professor Shyh Wang, will house the National Energy Research Scientific Computing Center (NERSC) and will become the center of operations for DOE’s Energy Sciences Network (ESnet).
“It’s a miracle that we sit here today for the opening,” commented Berkeley Lab deputy director Horst Simon at the dedication ceremony. Horst was referring to the more than a decade of planning, and detours that involved alternative sites and lawsuits. “This building will really change computational science,” he said.
The dedication ceremony celebrated the opening with a symbolic connecting of the 400 Gbps optical network. Left to right: DOE’s Barbara Helland, UC President Janet Napolitano, Berkeley Lab Director Paul Alivisatos, Dila Wang, UC Berkeley Chancellor Nicholas Dirks, Ipitek CEO Michael Salour, and Berkeley Lab Deputy Director Horst Simon.
One of the world’s leading supercomputing centers for open science, NERSC serves nearly 6,000 researchers in the US and internationally. Occupying 149,000 square feet, Wang Hall, also known as the Computational Research and Theory facility, was designed from top-to-bottom for energy-efficiency, taking advantage of the region’s mild climate to cool the supercomputers and eliminating the need for mechanical cooling.
The facility will soon house two supercomputers inside NERSC, which is located on the second level of Wang Hall: Edison, a Cray XC30 system, is NERSC’s curerent flagship computer, and the first phase of Cori, a 10-cabinet Cray XC40 machine based on the Haswell multicore processor, is nearing completion soon.
Speaking with HPCwire in an earlier article, Jay Srinivasan, NERSC’s Computational Systems Group lead, characterized the Haswell-based Cori as a familiar system that will bridge the gap between Hopper’s retirement and the KNL-based Cori in 2016. The upgraded Xeons mean that users can continue to run their applications without interruption in support of their efforts to develop new energy sources, improve energy efficiency, and understand climate change.
According to NERSC, Phase 1 Cori (also known as the “Cori Data Partition”) will have about the same sustained performance as Hopper (a Cray XE6, with a peak performance of 1.28 petaflops, which will be retired later in 2015). The Lustre file system and dragonfly topology based on the Aries interconnect are identical to the center’s current Edison supercomputer. Each of the more than 1,400 Haswell compute nodes touts 128 gigabytes of memory, twice the per-node memory of Edison. Early user access is expected in November 2015.
System Architecture for Cori Phase 1:
- 1,630 compute nodes.
- 128 GB of memory per node.
- Two 2.3 GHz 16-core Haswell processors per node.
- Each core has its own L1 and L2 caches, with 64 KB (32 KB instruction cache, 32 KB data) and 256 KB, respectively; there is also a 40-MB shared L3 cache per socket.
- Cray Aries high speed “dragonfly” topology interconnect, cabinets, and cooling (same as in Edison).
- Lustre file system with 30 PB of disk and > 700 GB/second I/O bandwidth.
- Flash ‘Burst Buffer’ to accelerate I/O performance, a layer of NVRAM that sits between memory and disk. The Phase 1 system will provide approximately 750 GB/second of I/O performance and about 750TB of storage.
- > 12 login/interactive nodes.
- SLURM workload manager.
The second phase of the Cori system will be installed and merged with Phase 1 in mid-2016, according to NERSC. Phase 2 will be based on Intel Xeon Phi Knights Landing (KNL) parts, which are self-hosted. The system will offer a sustained performance that is at least ten times that of Hopper for an estimated peak system performance in in the neighborhood of 30 petaflops.
For even more information on this unique building, check out the writeup here, which goes into detail about how Wang Hall was architected to include a custom-built seismically isolated floor to protect the expensive computer equipment and personnel.