Since 1986 - Covering the Fastest Computers in the World and the People Who Run Them

Language Flags
June 2, 2014

Titan Moves Full Steam Ahead

Tiffany Trader
OLCF Titan graphic

News being what it is, there’s a tendency to focus on what went wrong rather than what went right, but according to an update from the Oak Ridge Leadership Computing Facility, the early glitches associated with their Cray XK7 “Titan” system have been ironed out. Representatives with OLCF say that Titan “has overcome a challenging launch and is now showing impressive stability.”

“The machine is running very well,” states Don Maxwell, task lead in the OLCF’s HPC Operations Group.

“Node failures are on par with what we’d expect,” Maxwell continues. “Things are going very well. We’ve only experienced one unscheduled outage in over five months and no unscheduled outages in 2014.”

Chris Fuson of the User Assistance and Outreach Group reported similar sentiments and keyed in on the machine’s stability. “That just shows the maturity of the machine.”

“There is more uptime,” he said, “so more jobs can get through the queue and run to completion.”

“We don’t necessarily advertise that we are up more than normal,” he added. “The direct result of increased stability is that the users have an improved experience.”

Shortly after Titan was delivered, issues were uncovered that necessitated two rounds of “rolling repairs.” On April 8, 2013, the lab announced that the first set of repairs was completed. Now it’s revealing that work on a second set of repairs was completed on December 17, 2013. The repairs were made in sections, which rendered about 20 percent of the machine unaccessible. Since the repairs were finalized, the machine has been “very stable and heavily utilized,” according to lab officials.

OLCF Project Director Buddy Bland points out that having to do some troubleshooting early on is par for the course with a cutting-edge machine. He explains: “As we’ve seen many times with very large, first-of-a-kind systems, you’re likely to find abnormalities and manufacturing defects that might never be found anywhere else, just because there are so many different parts from so many different places.”

Addressing the repairs and bringing Titan to its full operating capacity was the result of a coordinated effort. Maxwell heads up a crew of four Cray employees and three ORNL staffers, who are tasked with maintaining Titan’s health. Duties include planning scheduled downtimes for software upgrades, and troubleshooting to stay abreast of near-term and far-term issues. Providing 24/7 support means that team member take turns being on call.

Maxwell also added that since the new year, there haven’t been any unscheduled outages or downtimes. When jobs can complete with fewer restarts, the overall user experience is improved. The numbers reflect this higher than normal availability and utilization. Since January 1, 2014, OLCF users have completed 110,587 jobs on Titan and have used 1,611,330,832 core hours. The combined usage of the 2014 INCITE projects is higher than ever before at this point in the allocation cycle. Another interesting stat is the high degree of capability usage – 62 percent of the time utilized on Titan so far this year across all allocations has come from jobs using more than 20 percent of the resource.

SC14 Virtual Booth Tours

AMD SC14 video AMD Virtual Booth Tour @ SC14
Click to Play Video
Cray SC14 video Cray Virtual Booth Tour @ SC14
Click to Play Video
Datasite SC14 video DataSite and RedLine @ SC14
Click to Play Video
HP SC14 video HP Virtual Booth Tour @ SC14
Click to Play Video
IBM DCS3860 and Elastic Storage @ SC14 video IBM DCS3860 and Elastic Storage @ SC14
Click to Play Video
IBM Flash Storage
@ SC14 video IBM Flash Storage @ SC14  
Click to Play Video
IBM Platform @ SC14 video IBM Platform @ SC14
Click to Play Video
IBM Power Big Data SC14 video IBM Power Big Data @ SC14
Click to Play Video
Intel SC14 video Intel Virtual Booth Tour @ SC14
Click to Play Video
Lenovo SC14 video Lenovo Virtual Booth Tour @ SC14
Click to Play Video
Mellanox SC14 video Mellanox Virtual Booth Tour @ SC14
Click to Play Video
Panasas SC14 video Panasas Virtual Booth Tour @ SC14
Click to Play Video
Quanta SC14 video Quanta Virtual Booth Tour @ SC14
Click to Play Video
Seagate SC14 video Seagate Virtual Booth Tour @ SC14
Click to Play Video
Supermicro SC14 video Supermicro Virtual Booth Tour @ SC14
Click to Play Video