News being what it is, there’s a tendency to focus on what went wrong rather than what went right, but according to an update from the Oak Ridge Leadership Computing Facility, the early glitches associated with their Cray XK7 “Titan” system have been ironed out. Representatives with OLCF say that Titan “has overcome a challenging launch and is now showing impressive stability.”

“The machine is running very well,” states Don Maxwell, task lead in the OLCF’s HPC Operations Group.

“Node failures are on par with what we’d expect,” Maxwell continues. “Things are going very well. We’ve only experienced one unscheduled outage in over five months and no unscheduled outages in 2014.”

Chris Fuson of the User Assistance and Outreach Group reported similar sentiments and keyed in on the machine’s stability. “That just shows the maturity of the machine.”

“There is more uptime,” he said, “so more jobs can get through the queue and run to completion.”

“We don’t necessarily advertise that we are up more than normal,” he added. “The direct result of increased stability is that the users have an improved experience.”

Shortly after Titan was delivered, issues were uncovered that necessitated two rounds of “rolling repairs.” On April 8, 2013, the lab announced that the first set of repairs was completed. Now it’s revealing that work on a second set of repairs was completed on December 17, 2013. The repairs were made in sections, which rendered about 20 percent of the machine unaccessible. Since the repairs were finalized, the machine has been “very stable and heavily utilized,” according to lab officials.

OLCF Project Director Buddy Bland points out that having to do some troubleshooting early on is par for the course with a cutting-edge machine. He explains: “As we’ve seen many times with very large, first-of-a-kind systems, you’re likely to find abnormalities and manufacturing defects that might never be found anywhere else, just because there are so many different parts from so many different places.”

Addressing the repairs and bringing Titan to its full operating capacity was the result of a coordinated effort. Maxwell heads up a crew of four Cray employees and three ORNL staffers, who are tasked with maintaining Titan’s health. Duties include planning scheduled downtimes for software upgrades, and troubleshooting to stay abreast of near-term and far-term issues. Providing 24/7 support means that team member take turns being on call.

Maxwell also added that since the new year, there haven’t been any unscheduled outages or downtimes. When jobs can complete with fewer restarts, the overall user experience is improved. The numbers reflect this higher than normal availability and utilization. Since January 1, 2014, OLCF users have completed 110,587 jobs on Titan and have used 1,611,330,832 core hours. The combined usage of the 2014 INCITE projects is higher than ever before at this point in the allocation cycle. Another interesting stat is the high degree of capability usage – 62 percent of the time utilized on Titan so far this year across all allocations has come from jobs using more than 20 percent of the resource.