Since 1986 - Covering the Fastest Computers in the World and the People Who Run Them

Language Flags
June 18, 2012

A Cloud SPEC on the Horizon

Robert Gelber

A benchmark to measure cloud computing performance is in the works. The Standard Performance Evaluation Corporation (SPEC) has created a new working group tasked with developing a set of metrics for cloud services. Computerworld reported on the story this past Friday.

cloud barsThis week, the International Supercomputing Conference (ISC’12) will unveil the 500 fastest computers on the planet. The systems are benchmarked by the number of floating point operations they can retire within a given second. Unlike HPC systems, cloud services lack an industry-wide method to gauge definitive performance. To alleviate the issue, SPEC’s Open Systems Group (OSG) has formed a new subcommittee, OSGCloud, comprised of participants from AMD, Dell, IBM, Ideas International, Intel, Karlshuhe Institute of Technology, Oracle, Red Hat and VMware, and accomplished SPEC benchmark developer Yun Chao. OSGCloud will collaborate with other SPEC groups “to define cloud benchmark methodologies, determine and recommend application workloads, identify cloud metrics for existing SPEC benchmarks, and develop new cloud benchmarks.”

The group has their work cut out for them, as resources are highly mutable in cloud environments. “…you don’t just have a single-sized computer most of the time. You are allowed to vary to your needs,” said Rema Hariharan, chair of OSGCloud to Computerworld. Some performance measurements may find their roots in existing tests like throughput, but others will have to be created from scratch. For example, elasticity might play a role in the new test. In this case, providers would be vetted based on how quickly they can adapt services to meet customer needs.

While final decisions have yet to be made, SPEC plans to release the new benchmark with a focus on three main groups. These include hardware and software vendors that enable cloud services, service providers (IaaS, PaaS, SaaS), and enterprise users looking to compare providers.

One concept under discussion splits up services according to the details they give regarding hardware. Black box services may be judged without delivering information about platform software or server components. On the other hand, white box providers would disclose equipment specs like processors, GPUs, storage, interconnects and so on.

OSGCloud has considered a number of benchmarks and measurement tools in a 50-page report on the subject.

“We have gotten off to a great start in creating guidelines for benchmarks with clearly defined, standardized metrics, but we would like to see wider participation, especially from cloud providers and users,” Hariharan disclosed in the group’s launch statement.

Ultimately, a definitive set of measurements could prove helpful for both cloud users and providers. Users would have the benefit of side-by-side comparisons based on the needs of their workloads, while service providers could use the benchmarks to differentiate their offering from competitors.

SC14 Virtual Booth Tours

AMD SC14 video AMD Virtual Booth Tour @ SC14
Click to Play Video
Cray SC14 video Cray Virtual Booth Tour @ SC14
Click to Play Video
Datasite SC14 video DataSite and RedLine @ SC14
Click to Play Video
HP SC14 video HP Virtual Booth Tour @ SC14
Click to Play Video
IBM DCS3860 and Elastic Storage @ SC14 video IBM DCS3860 and Elastic Storage @ SC14
Click to Play Video
IBM Flash Storage
@ SC14 video IBM Flash Storage @ SC14  
Click to Play Video
IBM Platform @ SC14 video IBM Platform @ SC14
Click to Play Video
IBM Power Big Data SC14 video IBM Power Big Data @ SC14
Click to Play Video
Intel SC14 video Intel Virtual Booth Tour @ SC14
Click to Play Video
Lenovo SC14 video Lenovo Virtual Booth Tour @ SC14
Click to Play Video
Mellanox SC14 video Mellanox Virtual Booth Tour @ SC14
Click to Play Video
Panasas SC14 video Panasas Virtual Booth Tour @ SC14
Click to Play Video
Quanta SC14 video Quanta Virtual Booth Tour @ SC14
Click to Play Video
Seagate SC14 video Seagate Virtual Booth Tour @ SC14
Click to Play Video
Supermicro SC14 video Supermicro Virtual Booth Tour @ SC14
Click to Play Video