Last week, the Texas Advanced Computing Center (TACC) revealed that it would be continuing the legacy of the Lonestar supercomputer series and reviving its relationship with Cray, the supercomputer maker behind the very first Lonestar system (circa 1997-2002).

The original Lonestar was a 50-gigaflops Cray T3E with 88 processsors. With Lonestar 5, TACC is installing a Cray XC40 machine with more than 30,000 Intel Xeon processing cores delivering 1.25 petaflops of computing power. Specs include 1,252 nodes of dual-socket 12-core Intel Xeon E5-2600 v3 processors, two large shared memory nodes with 1 TB each, eight large shared memory nodes with 500 GB each and a 1.2 PB DDN storage system, running on Cray Aries interconnect. The system will replace the Dell PowerEdge-based Lonestar 4 in serving Texas researchers with a wide variety of application needs.

The new Cray is also notable for being the second petaflops system deployed at TACC. Although seven years have passed since the IBM Roadrunner crossed that important 1,000X threshold, petascale supercomputers are still far from ubiquitous; the most recent TOP500 lists only 68 machines in this class.

Bill Barth, director of High Performance Computing at TACC, said that Cray offered the best deal when it came to technology, interconnects, storage, the whole package. Like other academic and government lab sites, TACC is faced with the challenge of meeting the needs of thousands of unique applications over thousands of users.

“If you look at our existing Lonestar 4 or Stampede, we have a huge mix of application workloads, from single-core jobs up through ultra-scale jobs that demand large fractions of the whole system. Working with a vendor to design a system that has the best balance for that extreme scale of workloads is a challenge,” he said.

Barth also touched on the decision to go with a straight x86 design, noting that while it’s an interesting time to be looking at accelerators in HPC, TACC already has a large deployment of accelerators and other architecture types with Stampede and its other systems.

The existing Lonestar 4 is a little unique in the Lonestar line in that it was a quarter paid for by the NSF, Barth tells HPCwire, so some portion of its hours contributed to the XSEDE program. The new Lonestar is 100 percent Texas-funded and Texas-focused. The Cray supercomputer will primarily support UT Austin but its user base will extend to the UT system and its research institutions as well as partners at Texas A&M and Texas Tech. “We’ll cover a good chunk state of Texas research computing,” observed Barth.

The HPC director added that while TACC supports all kinds of users, there is an effort underway to serve the bio- and bio-med community. With the ability to support protected data formats related to the Health Insurance Portability and Accountability Act (HIPAA) and the Federal Information Security Management Act (FISMA), the system can be used for both electronic health records and HPC-driven clinical work. In line with this, Lonestar 5 will be an important resource for a new medical school coming to the UT Austin campus in 2016.

While Lonestar 5 is not an NSF system, TACC is involved in supporting a number of NSF efforts, including Stampede its flagship system; Wrangler its data-focused system; and also Maverick, which was privately funded, but does contribute cycles to NSF researchers. TACC is also involved in two cloud-related efforts, Chameleon and Jetstream, via partnerships with UChicago and Indiana University respectively.

Lonestar 5 will be installed in late August or early September, according to TACC officials, and will reach full production by October. Lonestar 4 came on online February 2011 and is coming up on the five-year mark that spells end-of-life for most big supercomputers. As soon as users are migrated over, TACC will retire Lonestar 4, with the possible exception of a few components that are in heavy use, notably 14 large 1 terabyte shared memory nodes that have a lot of RAM and are popular for certain applications.

Lonestar5 will be installed in the spot where Ranger (Stampede’s predecessor) was formerly located before being taken offline. Barth says that right now TACC staff are making the necessary power and plumbing retrofits to prepare for the Cray system, which, for example, has 3-phase instead of single phase voltage requirements.

Barth said he expects the application porting process to be very straightforward. Even though they are switching suppliers, the staff and users are familiar with many of the components, the Intel compilers, Haswell chips, and the scheduler, Slurm, which is already used on Stampede. “I think it’s going to go great,” he shared. “We expect to have most of our existing Lonestar users and people coming over from other systems up and running very quickly.”

So how does the State of Texas get the funding needed for a system of this magnitude without federal assistance? Barth cited support from University of Texas, University of Texas system and other institutions around the state as factors, and added: “We have a good track record of deploying high-quality productive science systems on behalf of our institution and our sister institutions around the state and I think that track record allows us to have the trust from our community to get people to commit to the follow-on system.”

It’s been about eight years since the University of Texas at Austin has had a Cray system on site, according to Barth, so they are excited to be getting another. “We like the diversity of architecture and support models of working with different vendors,” he shared.

When TACC was first established in 2001, it inherited a 50-gigaflop Cray T3E. Around that timeframe, TACC also had a 16-processor Cray SV1 vector-processing shared-memory system that it ran on behalf of the Center for Space Research at the University of Texas at Austin.

Circa 2004, Cray served as integrator for the second Lonestar machine, considered the first terascale x86 cluster available to the open science community. Lonestar 2 was based on Dell PowerEdge 1750 servers, and delivered a peak performance of 3.7 teraflops (6.3 teraflops after upgrades). Dell continued to be the main supplier up through Lonestar 4, which delivered a peak speed of 302 teraflops.