With support from the National Science Foundation and the University of Tennessee, Knoxville, the National Institute for Computational Science (NICS) is expanding access to Beacon, its newest HPC cluster, providing researchers with a powerful research tool. Efforts are underway to optimize a number of science and engineering applications for this system utilizing both Intel Xeon processors and Intel Xeon Phi coprocessors.

By working with researchers to optimizing scientific codes to run on the advanced Intel architecture, the NICS team has determined that Beacon can register impressive performance gains for its users. In particular, scientists and engineers investigating such complex fields as nano-electronics, astrophysics, chemistry, biochemistry, subatomic physics and applied mathematics will be able to tackle larger, more complicated problems while controlling costs.

NICS’s drive to create the high performance computer cluster was the result of a number of factors. Optimizing application performance was at the top of the list – researchers needed to be able to modernize their code, taking full advantage of any inherent parallelism in order to manage increasingly demanding big science applications.

Another challenge facing the NICS team was the need to accommodate a wide variety of users with an equally diverse set of requirements. For example, a researcher running a complex simulation might need extensive raw computer power, while a bioinformatics scientist might require large amounts of memory.

For NICS, the primary driver was to learn how to build more efficient clusters to support researchers investigating increasingly complex, computer problems without significantly increasing hardware costs, power and cooling requirements, or software development costs.

Building Beacon

The solution, the Beacon system, is a Cray CS300-AC cluster supercomputer equipped with Intel Xeon processers and Intel Xeon Phi coprocessors. The system includes 48 compute nodes and six I/O nodes, with a total of 768 conventional cores and 11,520 accelerator cores. Compute nodes include Intel Xeon processors E5-2670 and Intel Xeon Phi coprocessors 5110P. Integrated into the storage environment are Intel Solid-State Drives. To optimize code, software developers use the Intel Cluster Studio XE suite.

Building a hybrid system consisting of Intel processors and coprocessors opened up new possibilities for software development and infrastructure testing by the NICS team. According to Glenn Brook, CTO at the Joint Institute for Computational Sciences at the University of Tennessee, “…the (hybrid) environment allows us to explore a variety of programming and processing scenarios. At the same time, the environment is designed to help us examine energy efficiency, data movement, and other variables. We hope to find new ways to maximize performance, minimize energy consumption, and reduce costs.”

Intel provided Intel Cluster Studio XE software development tools to help researchers optimize codes for the new architecture. The fact that team members were already familiar with the tools streamlined the optimization work. Noted Brook, “With the Intel Software Development Tools, optimizing for the Intel Xeon Phi coprocessor is not substantially different than optimizing for the Intel Xeon processor E5 family.”

Speeding Up Performance, Reducing Costs

Working with the optimized code, the NICS researchers are realizing a number of benefits. For example, Brook reports that an optimized computational fluid dynamics (CFD) code achieves about 2.25 times the performance on an Intel Xeon Phi coprocessor as compared to running the identical code on two Intel Xeon E5-2670 processors. “Those results indicate that researchers can build clusters that use Intel Xeon Phi coprocessors to boost performance while reducing costs,” he says.

And, Brook adds, ultimately, Beacon’s enhanced price/performance allows researchers to solve larger, more complex problems while controlling costs. By using Intel Xeon Phi coprocessors, organizations can build smaller clusters with fewer nodes and achieve the same performance as much larger clusters – a savings in hardware acquisition, energy costs, and floor space.

The Beacon project also demonstrates the feasibility of performing big science on sustainable systems. The cluster’s processing power earned it a spot on the November 2012 and June 2013 Top500 list, while its reduced energy consumption allowed Beacon to take top ranking on the November 2012 Green500 list. The cluster was rated at nearly 2.5 billion floating-point operations per second (gigaFLOPS) per watt.

“We hope to expand the Beacon project, creating more of a production environment that is available for science and engineering research,” Brook says. “At the same time, we will continue to evaluate the ways the Intel MIC Architecture can help reduce energy consumption and control the demand for human resources in software development.”