Per a newly-inked contract with Penguin Computing, the Department of Energy’s National Nuclear Security Administration (NNSA) is set to receive its third joint procurement of a commodity supercomputing environment to replace its 2011-2012 era systems. The new installation, called the Tri-Laboratory Commodity Technology System 1, or CTS-1, replaces the the Tri-Lab Linux Capacity Cluster 2 (TLCC2). Where TLCC2 provided an aggregate total computing capability of three petaflops, the Penguin clusters will aggregate 7-9 petaflops and will employ the forthcoming Broadwell-EP processors.

The three-year contract, valued at $39 million, provides NNSA’s Advanced Simulation and Computing (ASC) program with Penguin’s Open Compute-derived Tundra Extreme Scale (ES) series architecture to support national security workloads at Los Alamos, Sandia and Lawrence Livermore national laboratories. The selection process primarily focused on “performance features, supplier attributes, and price” according to the procurement guidelines established by the ASC.

As pointed out by the NNSA, advances in computational technology have brought down the cost of HPC systems from approximately $100,000,000 per teraflops in 1995 to less than $17,000 per teraflops in 2011-2012 (a factor of nearly 6,000) to to less than $5,000 per teraflops today, a factor of 20,000.

This is one of the biggest wins for Penguin Computing and also serves as validation for the open hardware model. “CTS-1 shows how the Open Compute and Open Rack design elements can be applied to high-performance computing and deliver similar benefits as its original development for Internet companies,” said Philip Pokorny, chief technology officer at Penguin Computing.

The Tundra ES system leverages the form factors that Facebook introduced with Open Compute and adapts and extends them for HPC with the aim of increased density, efficiency and serviceability benefits. For example, where Facebook’s standard is to fit three servers in two rack units, Tundra squeezes three dual-processor servers per horizontal rack unit, essentially doubling density. This arrangement accomodates up to 96 nodes per rack with the associated shared power infrastructure and fabric/networking switches. With a high-power shelf from Emerson Network Power, Penguin can offer up to 52.8kW power per rack (although current equipment maxes out below this max limit) compared to the current OCP limit of ~12kW.

Liquid cooling technology will be supplied by Asetek (more on that here). A liquid heat exchange is an important element in dense rack designs, according to Penguin, especially at high-altitude settings where cooling is challenged. A Penguin Computing spokesperson said the company is seeing more interest in liquid cooling as the technology reaches an economic tipping point in satisfying the need for denser computing and energy-efficiency.

On the storage side, there is a small amount of storage in management nodes on the Tundra ES racks, and Tundra can support integrated storage for say more data-optimized workloads, but as it stands now, the DOE will be handling its monolithic storage needs in a separate procurement (which we will be following up on).

In line with its mission to provide simulation capabilities and computational resources for the Stockpile Stewardship Program (SSP), the ASC runs acquisition programs for two kinds of computing platform classes: Commodity Technology systems (CTS) and Advanced Technology systems (ATS). The contract with Penguin represents the first CTS procurement, while Cray was selected to supply the first ATS system, Trinity, currently under construction at the Strategic Computing Center of Los Alamos National Laboratory.

The CTS model is designed to maximize the purchasing power of the Tri-Laboratories by employing standardized hardware and software environments across the three labs. “By deploying a common hardware environment multiple times at all three laboratory sites over multiple government fiscal years, the time and cost associated with any one cluster is greatly reduced,” explains the RFP documents.

Advanced Technology Systems, by contrast (according to NNSA/ASC Program Office documents) are “the vanguards of high performance computing platform market and incorporate features that, if successful, will become future commodity technologies. These large, first-of-a-kind systems will require application software modifications in order to take full advantage of exceptional capabilities offered by new technology.”

Dan Dowling, vice president of engineering services at Penguin Computing, further explained that the contract is based on the notion, developed by NNSA, of a “scalable unit,” or SU, a smallish-cluster that ranges from about 150 to 200 nodes and can be connected, like legos, to form much larger systems. The original proposal called for an architecture with the “flexibility to aggregate between 1 to 24 SUs into a single cluster, potentially accessible to a single job.” The modular approach is intended to streamline the building, shipping, installation and acceptance processes, but it also makes it so that each lab can configure the SUs into clusters according to mission needs.

The SU method produces a common Linux cluster hardware environment for the Tri-Laboratories, which eases administration, minimizes software changes and maximizes availability to end-users. At the same time as the ASC stands to benefit from a familiar environment, they also want to be able to leverage ongoing technology improvements. To this point, Penguin said that there is discussion about supplying the Tri-Lab partners with additional racks that make use of heterogenous computing elements like GPGPUs, Phi coprocessors and ARM chips, but the initial contract specifies only standard x86 parts. This is the aforementioned Broadwell-EP line, which Penguin will soon plug into its Relion 1930e servers “for cost-effective and reliable system performance.” Tundra in fact was designed on Intel Scalable System framework principles.

The HPC clusters will be delivered to the laboratories between April 2016 and September 2018. Each scalable unit represents approximately 200 teraflops of computing power, which at 7 petaflops total, translates into roughly 35 SUs (about 70 racks). That means there’s at least a half teraflops performance per processor. Likely the real number is higher, but Intel hasn’t disclosed specs on the processor yet. According to Intel’s last public comment on the subject, the chip launch was pushed from Q4 2015 to Q1 2016.