SGI Colors New Shared Memory Machines Ultraviolet

By Michael Feldman

November 16, 2009

After what may be the longest development cycle ever for a supercomputer, SGI has unveiled the first commercial implementation of its Ultraviolet architecture. The company first announced “Project Ultraviolet” at SC03. Now six years later, it has launched Altix UV, the company’s first scale-up HPC system based on x86 technology. The Altix UV’s connection to the 2003 design is tenuous at best, but the new architecture does fulfill Ultraviolet’s original promise of delivering a shared memory architecture able to scale from a few sockets all the way up to a petascale supercomputer.

SGI Altix UV

Besides being simpler to program than distributed memory clusters, shared memory systems are especially well suited to I/O bound and memory-bound applications; codes that depend upon a lot of inter-processor communication; and any type of application that uses large — as in terabyte-sized — in-memory databases. These shared memory systems can also be used in conjunction with large clusters to provide an “analysis supernode.”

The two initial products, the Altix UV 1000 and Altix UV 100, are both based on Intel Nehalem-equipped blades, which are hooked together with SGI’s 5th generation NUMAlink fabric. The software stack includes everything from the OS on up, including the SGI Foundation Software, data management packages (XFS, CXFS, DMF), SGI’s ProPack and System Management tools, job schedulers (Altair PBSP and Moab) and developer tools and libraries. The machines come with either SUSE Linux Enterprise Server or Red Hat Enterprise Linux.

The blades themselves contain two eight-core Nehalem EX chips, each with a bank of four DDR3 memory channels. If a larger memory to core ratio is desired, there are 6- and 4-core options, as well as a single-socket configuration. An optional I/O riser allows for a choice of expansion slots or external I/O ports. Up to two PCIe slots are available on each blade and these can be used to plug in external storage (SGI or otherwise) or GPGPUs.

SGI’s secret sauce is the UV hub, which sits on each blade and acts as the node controller. The hub, along with the NUMAlink 5 interconnect, is the technology that makes the supersized shared memory possible. The new interconnect delivers sub-microsecond latencies and 15.0 GB/sec of aggregate bandwidth per blade. The hub itself manages data traffic between the local CPU resources and the rest of the system, arbitrating between the local QuickPath Interconnect (QPI) links and the NUMAlink fabric.

According to Jill Matzke, Altix product manager, the SGI engineers decided to limit themselves to two sockets per blade in order to avoid overtaxing the QPI bandwidth, which needs to feed the NUMAlink fabric and I/O. Since Nehalem EX is designed to support up to 8 sockets per board, one might wonder why SGI didn’t opt for the dual-socket-capable Nehalem EP chips. Apparently, EX was chosen because it offered more QPI and memory bandwidth, both of which were essential to the UV design. In any case, the Nehalem EP design does not lend itself to external node controllers, such as the UV hub.

The Altix UV 100 is aimed at the mid-range market, scaling from a single 3U rackmount unit containing two dual-socket blades, up to a 7 teraflop, 96-socket machine that fits into a couple of racks. The upper limit on memory capacity on this product is 6 TB. The UV 100 is aimed at users who need a moderate to large SMP environment for their x86 applications. At the maximum 96-socket configuration, 768 cores are available, which doubles to 1,536 threads thanks to Nehalem-style multithreading support.

The Altix UV 1000 is a cabinet solution that scales all the way to the top, that is, 256 sockets (yielding 2,048 cores or 4,096 threads) and 16 TB of memory. At the max configuration, this model delivers 18.6 peak teraflops in a 42U space. The 16 TB limit on the UV 1000 corresponds to the maximum memory reach of the Intel Nehalem processor. However, the UV 1000 design can actually scale beyond this limit by connecting multiple 256-socket systems in a 2-D torus topology. In this case, the system would be partitioned with multiple OS images but support a much larger shared global address space — up into petabytes. The upper limit supported by the UV hub is 32,768 sockets, which would equate to about 2 petaflops. SGI is certainly willing to help interested parties develop such systems, but the vast majority of customers will be able to fit their applications within the 256-socket, single system image machine.

Note the current Itanium-based Altix 4700 reaches to 128 GB because that CPU’s memory address is wider, although core count on those systems tops out at 1024. That said, just getting a handful of terabytes of global memory on an x86 platform is likely to be a big attraction for HPC users. “We are seeing people ordering many more terabytes of memory on UV than they ever did on Altix with Itanium, simply because of the overall capability and the price-performance,” says Matzke.

Although UV supports highly-scaled applications in a global memory model, today the majority of global memory applications scale to just 32 or maybe 64 threads. However, UV, like most shared memory machines, can also deliver great performance for MPI applications by properly exploiting the unified memory and the speed of the interconnect fabric. Moreover, an MPI offload engine has been incorporated into the UV hub to further accelerate this class of applications. SGI has demonstrated a 3X improvement in the HPCC GUPS benchmark with the offload engine enabled. According to Matzke, “70 percent of the people that buy these systems are running MPI, but have other application demands that make it really shine on this kind of an architecture.”

According to Geoffrey Noer, SGI’s senior director of product marketing, the company is currently taking orders for the new UV machines, with the first shipments expected by second quarter of 2010 (following Intel’s release of the Nehalem EX CPUs). Initial customers include the University of Tennessee (1024 cores, 4 TB memory), the North German Supercomputing Alliance, known as HLRN (two systems, 4,352 cores, 18 TB memory), CALMIP in France (128 cores, 1 TB memory), and the University of Hokkaido (180 cores, 360 GB memory). A number of UV systems have also been purchased by the federal government for certain “defense applications” (which shall remain nameless). SGI is not making UV pricing public, but potential buyers can always obtain a quote under NDA.

Although many customers using Itanium Altix systems will undoubtedly transition to the x86 UV platform, Noer says SGI will continue to offer the Altix 450 and 4700 systems. And even though they are not publicly divulging specific plans for future Itanium-based shared memory machines, Noer did have this to offer: “It’s important not to look at Altix UV as a direct replacement for the 4700…. We are working with Intel on next-generation processor technologies as well.  For those customers that are getting the benefits out of the larger address space and benefits with the 4700, they absolutely don’t need to switch to Altix UV.”

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Nvidia Debuts Turing Architecture, Focusing on Real-Time Ray Tracing

August 16, 2018

From the SIGGRAPH professional graphics conference in Vancouver this week, Nvidia CEO Jensen Huang unveiled Turing, the company's next-gen GPU platform that introduces new RT Cores to accelerate ray tracing and new Tenso Read more…

By Tiffany Trader

HPC Coding: The Power of L(o)osing Control

August 16, 2018

Exascale roadmaps, exascale projects and exascale lobbyists ask, on-again-off-again, for a fundamental rewrite of major code building blocks. Otherwise, so they claim, codes will not scale up. Naturally, some exascale pr Read more…

By Tobias Weinzierl

STAQ(ing) the Quantum Computing Deck

August 16, 2018

Quantum computers – at least for now – remain noisy. That’s another way of saying unreliable and in diverse ways that often depend on the specific quantum technology used. One idea is to mitigate noisiness and perh Read more…

By John Russell

HPE Extreme Performance Solutions

Introducing the First Integrated System Management Software for HPC Clusters from HPE

How do you manage your complex, growing cluster environments? Answer that big challenge with the new HPC cluster management solution: HPE Performance Cluster Manager. Read more…

IBM Accelerated Insights

Super Problem Solving

You might think that tackling the world’s toughest problems is a job only for superheroes, but at special places such as the Oak Ridge National Laboratory, supercomputers are the real heroes. Read more…

NREL ‘Eagle’ Supercomputer to Advance Energy Tech R&D

August 14, 2018

The U.S. Department of Energy (DOE) National Renewable Energy Laboratory (NREL) has contracted with Hewlett Packard Enterprise (HPE) for a new 8-petaflops (peak) supercomputer that will be used to advance early-stage R&a Read more…

By Tiffany Trader

STAQ(ing) the Quantum Computing Deck

August 16, 2018

Quantum computers – at least for now – remain noisy. That’s another way of saying unreliable and in diverse ways that often depend on the specific quantum Read more…

By John Russell

NREL ‘Eagle’ Supercomputer to Advance Energy Tech R&D

August 14, 2018

The U.S. Department of Energy (DOE) National Renewable Energy Laboratory (NREL) has contracted with Hewlett Packard Enterprise (HPE) for a new 8-petaflops (peak Read more…

By Tiffany Trader

CERN Project Sees Orders-of-Magnitude Speedup with AI Approach

August 14, 2018

An award-winning effort at CERN has demonstrated potential to significantly change how the physics based modeling and simulation communities view machine learni Read more…

By Rob Farber

Intel Announces Cooper Lake, Advances AI Strategy

August 9, 2018

Intel's chief datacenter exec Navin Shenoy kicked off the company's Data-Centric Innovation Summit Wednesday, the day-long program devoted to Intel's datacenter Read more…

By Tiffany Trader

SLATE Update: Making Math Libraries Exascale-ready

August 9, 2018

Practically-speaking, achieving exascale computing requires enabling HPC software to effectively use accelerators – mostly GPUs at present – and that remain Read more…

By John Russell

Summertime in Washington: Some Unexpected Advanced Computing News

August 8, 2018

Summertime in Washington DC is known for its heat and humidity. That is why most people get away to either the mountains or the seashore and things slow down. H Read more…

By Alex R. Larzelere

NSF Invests $15 Million in Quantum STAQ

August 7, 2018

Quantum computing development is in full ascent as global backers aim to transcend the limitations of classical computing by leveraging the magical-seeming prop Read more…

By Tiffany Trader

By the Numbers: Cray Would Like Exascale to Be the Icing on the Cake

August 1, 2018

On its earnings call held for investors yesterday, Cray gave an accounting for its latest quarterly financials, offered future guidance and provided an update o Read more…

By Tiffany Trader

Leading Solution Providers

SC17 Booth Video Tours Playlist

Altair @ SC17

Altair

AMD @ SC17

AMD

ASRock Rack @ SC17

ASRock Rack

CEJN @ SC17

CEJN

DDN Storage @ SC17

DDN Storage

Huawei @ SC17

Huawei

IBM @ SC17

IBM

IBM Power Systems @ SC17

IBM Power Systems

Intel @ SC17

Intel

Lenovo @ SC17

Lenovo

Mellanox Technologies @ SC17

Mellanox Technologies

Microsoft @ SC17

Microsoft

Penguin Computing @ SC17

Penguin Computing

Pure Storage @ SC17

Pure Storage

Supericro @ SC17

Supericro

Tyan @ SC17

Tyan

Univa @ SC17

Univa

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This