Visit additional Tabor Communication Publications
October 11, 2011
Jaguar's days as a CPU-only supercomputer are numbered. Over the next year, the 2.3 petaflop machine at the Department of Energy's Oak Ridge National Lab (ORNL) will be upgraded by Cray with the new NVIDIA "Kepler" GPUs, producing a system with about 10 times Jaguar's peak performance. The transformed supercomputer will be renamed Titan and should deliver in the neigborhood of 20 peak petaflops sometime in late 2012.
The current Jaguar system, which has already been upgraded numerous times since it was first deployed in 2009, currently sits at number three on the TOP500 list with a Linpack reading of 1.76 petaflops. Titan will certainly keep the machine in the top 5, even as machines with tens of petaflops start making their way into the big labs over the next couple years.
Titan will also represent the US entry in the echelons of top tier GPU-accelerated supercomputing. As it stands today, three of the top five systems are GPU accelerated: Tianhe-1A and Nebulae in China, and TSUBAME 2.0 in Japan. The current top GPU machine in the US is Edge, a 240-teraflop Appro cluster at Lawrence Livermore National Laboratory. Even Russia, Germany, Italy have larger systems.
According to Steve Scott, the newly minted chief technology officer for NVIDIA's Tesla Business Unit, the fact that ORNL is making such a significant commitment to GPU computing is a big endorsement for the architecture. It's no secret that HPC is now constrained by energy use. Moore's Law has managed to shrink the transistor geometries, but the power wall has become the defining limitation for performance increases. "It's all about power efficiency" Scott told HPCwire, "which is why we think the GPU story is so compelling."
While GPUs are not truly general-purpose processors, their ability to perform data-parallel computation in a much more energy-efficient manner than CPUs has vaulted them to prominence in the HPC realm. "It's hard to overstate the importance of the sea change that has happened in high performance computing," notes Scott. "This wonderful ride we've been on for the past 30 years -- every time we halve the size of transistor, the voltage drops, power stays the same, and performance improves exponentially -- has been fantastic, but it's done."
Although the US, in general, has been a bit late in embracing GPU technology for HPC, the Titan supercomputer has been on the drawing board at Oak Ridge for at least a couple of years. But the technology necessary to implement that machine is just now catching up with those requirements.
Beginning this fall, most of 18,688 of Jaguar's current XT5 nodes will be retrofitted with Cray's new XK6 blades, which the company unveiled in May. The immediate result is that the current dual-socket 6-core AMD Opteron nodes will be swapped out for a single 16-core "Interlagos" CPU node and the interconnect will upgraded from SeaStar 2 to Gemini. Each XK6 blade encompasses four compute nodes, with an Opteron on each one, and the ability to connect each of those CPUs to a Tesla GPU on a PCIe daughter card.
Initially, 960 of those XK6 nodes will be outfitted with the Fermi-class Tesla M2090 GPUs, with the other odd 17 thousand remaining as CPU-only blades for the time being. This first phase of Titan is expected to be completed before the end of the year. Then in the second half of 2012, all 18,688 nodes, including the original Fermi-equipped blades, will be populated with NVIDIA's next-generation Kepler Teslas.
NVIDIA has not provided detailed specs on the Kepler GPUs, but according to Scott their performance per watt will be more be than double that of the Fermi parts, while fitting into the same power envelope. Given the current Fermi Tesla cards (GPUs plus memory) deliver 665 gigaflops, the new Kepler GPU should yield at least 1330 gigaflops.
For the time being, Oak Ridge is promising only 10 to 20 petaflops for the final system, although the peak performance could go considerably higher. According to Buddy Bland, project director at ORNL's Leadership Computing Facility, they currently don't have the money in hand to upgrade all 18K nodes. The actual scope of the Titan build-out will "depend on the budget available."
Theoretically though, if all existing nodes are populated with the new Kepler parts, the system should deliver at least 24.8 petaflops of GPU power. An equal number of Interlagos CPUs should contribute more than two additional petaflops on top of that. By the time all the dust has settled, Titan could be within spitting distance of 30 petaflops.
The amount of power the new system will draw is also unknown, but it will certainly have a better performance per watt ratio than Jaguar, which sucks up nearly 7 MW for its 2.33 peak petaflops. By contrast, Japan's Fermi-accelerated TSUBAME system uses just 1.4 MW for its 2.29 petaflops. Since ORNL's new machine will use the more efficient Kepler GPUs, its efficiency should be significantly better. "We view Titan as the leading indicator of where people are going as they look to solve the energy challenges for the next five to ten years," says Scott.
How all those peak flops turn into actual application performance remains to be seen. Extracting high levels of sustained computation from these multi-petaflop machines is notoriously difficult, with only a handful of codes able to attain more than a petaflop of performance. Adding GPUs to the mix has made that harder, at least in the short term.
In this regard, Oak Ridge, with one of the premier computational lab's on the planet, has a good chance of pushing the envelope. Using smaller GPU clusters, computations scientists at ORNL and elsewhere have been busy porting six flagship science codes to CUDA, include Wang-Landau/LSMS for material science; S3D for engine combustion; PFLOTRAN for underground C02 sequestration and for underground contaminant containment; Denovo for radiation transport code in nuclear engineering; CAM-SE for climate change modeling; and LAMMPS, a molecular dynamics simulation code. Scott says ORNL, Cray and NVIDIA have been working together to adapt these science codes for heterogenous computing so that they are ready to go when Titan boots up.
This first phase of Titan is expected to generate more than $60 million in revenue for Cray, which could end up in the company's hands before the end of the year. Over the lifetime of the contract, Cray is looking to collect more than $97 million, although if upgrade options are exercised, that number could go considerably higher.
May 22, 2013 |
At some point in the not-too-distant future, building powerful, miniature computing systems will be considered a hobby for high schoolers, just as robotics or even Lego-building are today. That could be made possible through recent advancements made with the Raspberry Pi computers.
May 16, 2013 |
When it comes to cloud, long distances mean unacceptably high latencies. Researchers from the University of Bonn in Germany examined those latency issues of doing CFD modeling in the cloud by utilizing a common CFD and its utilization in HPC instance types including both CPU and GPU cores of Amazon EC2.
May 15, 2013 |
Supercomputers at the Department of Energy’s National Energy Research Scientific Computing Center (NERSC) have worked on important computational problems such as collapse of the atomic state, the optimization of chemical catalysts, and now modeling popping bubbles.
May 10, 2013 |
Program provides cash awards up to $10,000 for the best open-source end-user applications deployed on 100G network.
05/10/2013 | Cleversafe, Cray, DDN, NetApp, & Panasas | From Wall Street to Hollywood, drug discovery to homeland security, companies and organizations of all sizes and stripes are coming face to face with the challenges – and opportunities – afforded by Big Data. Before anyone can utilize these extraordinary data repositories, however, they must first harness and manage their data stores, and do so utilizing technologies that underscore affordability, security, and scalability.
04/15/2013 | Bull | “50% of HPC users say their largest jobs scale to 120 cores or less.” How about yours? Are your codes ready to take advantage of today’s and tomorrow’s ultra-parallel HPC systems? Download this White Paper by Analysts Intersect360 Research to see what Bull and Intel’s Center for Excellence in Parallel Programming can do for your codes.
In this demonstration of SGI DMF ZeroWatt disk solution, Dr. Eng Lim Goh, SGI CTO, discusses a function of SGI DMF software to reduce costs and power consumption in an exascale (Big Data) storage datacenter.
The Cray CS300-AC cluster supercomputer offers energy efficient, air-cooled design based on modular, industry-standard platforms featuring the latest processor and network technologies and a wide range of datacenter cooling requirements.