HPC Center Traces Storage Selection Experience

By Nicole Hemsoth

July 8, 2011

We often hear about national labs and universities settling on a particular vendor for server and storage solutions, but details are usually in short supply when it comes to how vendors stacked up against one another in a head-to-head bidding war.

HP announced last week that the University of Utah’s Center for High Performance Computing (CHPC) moved into its Converged Infrastructure arena by selecting the HP X9320 IBRIX Network Storage System coupled with ProLiant SL160z G6 servers. This announcement, like many others of this ilk was full of the expected hyperbole about scalability and cost, so we followed up with the Brian Haymore, who heads the HPC storage team at CHPC to find out how they evaluated the competing vendors to enhance the center’s Updraft cluster and what ultimately led to their storage decision.

The I/O issue isn’t new for Haymore’s team. He says that this pain point was one they recognized early on but that came into more focus when they would have one or two users running large cases on the clusters, then having everyone else wanting to go to the scratch file system to look at the results they’d run weeks or months ago. He said that at this point the file system would be dead in the water–quite a problem when their people expected interactive responsiveness. He claims they knew the applications were saturating everything the current file system could offer and that it wasn’t a network saturation issue. He remained convinced that NFS just wouldn’t offer the scalability for some applications and that proprietary solutions might offer the only remedy.

The chemical and fuels engineering group at CHPC was running an application that was authored by the Center for the Simulation of Accidental Fires and Explosions. This application is a composite of code contributed from scientists across the country, which fine-tunes its results but is difficult to modify from an I/O perspective. This meant that for Haymore’s team, the storage selection process required more than just looking at price points—they needed a file system that was going to fit with the application without manipulating application itself.

With that in mind, the I/O difficulties were at the heart of performance hitches. During the baseline test, which was against their standard NFS server they were running at about 90 seconds per iteration with about 45 percent of that time being gobbled by I/O. In other words, half of the time that baseline system over the standard NFS server was spent in I/O activity.

Four vendors were vying for a chance to improve the I/O capabilities at CHPC, including Panasas, HP with its IBRIX solution, partners Dell and Terascala with their Lustre offering, and the partnership to provide GPFS from IBM and DDN. Haymore told us that while these were the four main vendors considered, others, including Isilon were evaluated early on. Isilon’s solution would only have been suitable if the application could be changed, which was not a possibility.

Haymore says that Panasas provided no performance increase with their application. His team wanted to dig deeper with the Panasas engineering team to look for the choke point but they were unable to gain any traction with that process. Eventually, he says, this option timed out and they considered other alternatives.

While they were able to realize a tripling in performance with the Dell and Terascala Lustre offering, the excitement over the performance increase was hampered by a troubling series of mysterious I/O errors that affected 50 percent of the runs, even those that used the exact same dataset. As Haymore described, there seemed to be no rhyme or reason—the “file system just puked.”

He says that they found good support from the Dell Terascala team but ultimately they were never able to resolve the error after determining it was not a tuning error and instead was likely a bug that had been filed with the Lustre package that could not be fixed in a reasonable timeframe. Besides, as Haymore noted, aside from these more practical concerns about stability, the very status of the Lustre file system was in question as it was being handed off to Oracle.

In the end, the choice boiled down to the DDN/IBM GPFS and HP’s IBRIX solutions as they both performed almost exactly the same. He says that in this case, the tipping point wasn’t based on pricing alone—rather, he said, the support model was a major factor. As Haymore pointed out, getting your hardware from DDN and software from IBM required two hops for support whereas with HP, it was a single, unified support model—an important factor in his team’s final decision.

Make no mistake, however, price did play a role. While he admits that even at the beginning he expected the HP solution to be quite expensive, he says that they were able to accommodate their budget—the icing on the cake, as far as Haymore was concerned.

On that note, we asked if he went into the closed bidding process thinking that one solution would win out. He says that he would have counted on Lustre as being the champion if he had to make an early pre-benchmarking guess. This is because, as he put it, “Part of us doing our jobs is to keep our finger on the pulse of what the big boys are doing and for us, those big boys are the national labs. Lustre is heavily deployed there but it’s hard to tell if it’s because that’s what won the bid on a price point or if it was really the king of performance….We don’t know why it is always selected. We just figured we’d mimic national labs since it’s been their trend for the last several years.” While he notes that they do use other file systems, he says he’s still surprised at the errors they faced with Lustre.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

DARPA Continues Investment in Post-Moore’s Technologies

July 24, 2017

The U.S. military long ago ceded dominance in electronics innovation to Silicon Valley, the DoD-backed powerhouse that has driven microelectronic generation for decades. With Moore's Law clearly running out of steam, the Read more…

By George Leopold

Graphcore Readies Launch of 16nm Colossus-IPU Chip

July 20, 2017

A second $30 million funding round for U.K. AI chip developer Graphcore sets up the company to go to market with its “intelligent processing unit” (IPU) in 2017 with scale-up production for enterprise datacenters and Read more…

By Tiffany Trader

Trinity Supercomputer’s Haswell and KNL Partitions Are Merged

July 19, 2017

Trinity supercomputer’s two partitions – one based on Intel Xeon Haswell processors and the other on Xeon Phi Knights Landing – have been fully integrated are now available for use on classified work in the Nationa Read more…

By HPCwire Staff

Fujitsu Continues HPC, AI Push

July 19, 2017

Summer is well under way, but the so-called summertime slowdown, linked with hot temperatures and longer vacations, does not seem to have impacted Fujitsu's output. The Japanese multinational has made a raft of HPC and A Read more…

By Tiffany Trader

HPE Extreme Performance Solutions

HPE Servers Deliver High Performance Remote Visualization

Whether generating seismic simulations, locating new productive oil reservoirs, or constructing complex models of the earth’s subsurface, energy, oil, and gas (EO&G) is a highly data-driven industry. Read more…

Researchers Use DNA to Store and Retrieve Digital Movie

July 18, 2017

From abacus to pencil and paper to semiconductor chips, the technology of computing has always been an ever-changing target. The human brain is probably the computer we use most (hopefully) and understand least. This mon Read more…

By John Russell

The Exascale FY18 Budget – The Next Step

July 17, 2017

On July 12, 2017, the U.S. federal budget for its Exascale Computing Initiative (ECI) took its next step forward. On that day, the full Appropriations Committee of the House of Representatives voted to accept the recomme Read more…

By Alex R. Larzelere

Summer Reading: IEEE Spectrum’s Chip Hall of Fame

July 17, 2017

Take a trip down memory lane – the Mostek MK4096 4-kilobit DRAM, for instance. Perhaps processors are more to your liking. Remember the Sh-Boom processor (1988), created by Russell Fish and Chuck Moore, and named after Read more…

By John Russell

Women in HPC Luncheon Shines Light on Female-Friendly Hiring Practices

July 13, 2017

The second annual Women in HPC luncheon was held on June 20, 2017, during the International Supercomputing Conference in Frankfurt, Germany. The luncheon provides participants the opportunity to network with industry lea Read more…

By Tiffany Trader

Graphcore Readies Launch of 16nm Colossus-IPU Chip

July 20, 2017

A second $30 million funding round for U.K. AI chip developer Graphcore sets up the company to go to market with its “intelligent processing unit” (IPU) in Read more…

By Tiffany Trader

Fujitsu Continues HPC, AI Push

July 19, 2017

Summer is well under way, but the so-called summertime slowdown, linked with hot temperatures and longer vacations, does not seem to have impacted Fujitsu's out Read more…

By Tiffany Trader

Researchers Use DNA to Store and Retrieve Digital Movie

July 18, 2017

From abacus to pencil and paper to semiconductor chips, the technology of computing has always been an ever-changing target. The human brain is probably the com Read more…

By John Russell

The Exascale FY18 Budget – The Next Step

July 17, 2017

On July 12, 2017, the U.S. federal budget for its Exascale Computing Initiative (ECI) took its next step forward. On that day, the full Appropriations Committee Read more…

By Alex R. Larzelere

Women in HPC Luncheon Shines Light on Female-Friendly Hiring Practices

July 13, 2017

The second annual Women in HPC luncheon was held on June 20, 2017, during the International Supercomputing Conference in Frankfurt, Germany. The luncheon provid Read more…

By Tiffany Trader

Satellite Advances, NSF Computation Power Rapid Mapping of Earth’s Surface

July 13, 2017

New satellite technologies have completely changed the game in mapping and geographical data gathering, reducing costs and placing a new emphasis on time series Read more…

By Ken Chiacchia and Tiffany Jolley

Intel Skylake: Xeon Goes from Chip to Platform

July 13, 2017

With yesterday’s New York unveiling of the new “Skylake” Xeon Scalable processors, Intel made multiple runs at multiple competitive threats and strategic Read more…

By Doug Black

Perverse Incentives? How Economics (Mis-)shaped Academic Science

July 12, 2017

The unintended consequences of how we fund academic research—in the U.S. and elsewhere—are strangling innovation, putting universities into debt and creatin Read more…

By Ken Chiacchia, Senior Science Writer, Pittsburgh Supercomputing Center

Google Pulls Back the Covers on Its First Machine Learning Chip

April 6, 2017

This week Google released a report detailing the design and performance characteristics of the Tensor Processing Unit (TPU), its custom ASIC for the inference Read more…

By Tiffany Trader

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Quantum Bits: D-Wave and VW; Google Quantum Lab; IBM Expands Access

March 21, 2017

For a technology that’s usually characterized as far off and in a distant galaxy, quantum computing has been steadily picking up steam. Just how close real-wo Read more…

By John Russell

HPC Compiler Company PathScale Seeks Life Raft

March 23, 2017

HPCwire has learned that HPC compiler company PathScale has fallen on difficult times and is asking the community for help or actively seeking a buyer for its a Read more…

By Tiffany Trader

Trump Budget Targets NIH, DOE, and EPA; No Mention of NSF

March 16, 2017

President Trump’s proposed U.S. fiscal 2018 budget issued today sharply cuts science spending while bolstering military spending as he promised during the cam Read more…

By John Russell

CPU-based Visualization Positions for Exascale Supercomputing

March 16, 2017

In this contributed perspective piece, Intel’s Jim Jeffers makes the case that CPU-based visualization is now widely adopted and as such is no longer a contrarian view, but is rather an exascale requirement. Read more…

By Jim Jeffers, Principal Engineer and Engineering Leader, Intel

Nvidia’s Mammoth Volta GPU Aims High for AI, HPC

May 10, 2017

At Nvidia's GPU Technology Conference (GTC17) in San Jose, Calif., this morning, CEO Jensen Huang announced the company's much-anticipated Volta architecture a Read more…

By Tiffany Trader

Facebook Open Sources Caffe2; Nvidia, Intel Rush to Optimize

April 18, 2017

From its F8 developer conference in San Jose, Calif., today, Facebook announced Caffe2, a new open-source, cross-platform framework for deep learning. Caffe2 is the successor to Caffe, the deep learning framework developed by Berkeley AI Research and community contributors. Read more…

By Tiffany Trader

Leading Solution Providers

How ‘Knights Mill’ Gets Its Deep Learning Flops

June 22, 2017

Intel, the subject of much speculation regarding the delayed, rewritten or potentially canceled “Aurora” contract (the Argonne Lab part of the CORAL “ Read more…

By Tiffany Trader

Reinders: “AVX-512 May Be a Hidden Gem” in Intel Xeon Scalable Processors

June 29, 2017

Imagine if we could use vector processing on something other than just floating point problems.  Today, GPUs and CPUs work tirelessly to accelerate algorithms Read more…

By James Reinders

Russian Researchers Claim First Quantum-Safe Blockchain

May 25, 2017

The Russian Quantum Center today announced it has overcome the threat of quantum cryptography by creating the first quantum-safe blockchain, securing cryptocurrencies like Bitcoin, along with classified government communications and other sensitive digital transfers. Read more…

By Doug Black

MIT Mathematician Spins Up 220,000-Core Google Compute Cluster

April 21, 2017

On Thursday, Google announced that MIT math professor and computational number theorist Andrew V. Sutherland had set a record for the largest Google Compute Engine (GCE) job. Sutherland ran the massive mathematics workload on 220,000 GCE cores using preemptible virtual machine instances. Read more…

By Tiffany Trader

Google Debuts TPU v2 and will Add to Google Cloud

May 25, 2017

Not long after stirring attention in the deep learning/AI community by revealing the details of its Tensor Processing Unit (TPU), Google last week announced the Read more…

By John Russell

Groq This: New AI Chips to Give GPUs a Run for Deep Learning Money

April 24, 2017

CPUs and GPUs, move over. Thanks to recent revelations surrounding Google’s new Tensor Processing Unit (TPU), the computing world appears to be on the cusp of Read more…

By Alex Woodie

Six Exascale PathForward Vendors Selected; DoE Providing $258M

June 15, 2017

The much-anticipated PathForward awards for hardware R&D in support of the Exascale Computing Project were announced today with six vendors selected – AMD Read more…

By John Russell

Top500 Results: Latest List Trends and What’s in Store

June 19, 2017

Greetings from Frankfurt and the 2017 International Supercomputing Conference where the latest Top500 list has just been revealed. Although there were no major Read more…

By Tiffany Trader

  • arrow
  • Click Here for More Headlines
  • arrow
Share This