MLPerf – Will New Machine Learning Benchmark Help Propel AI Forward?

By John Russell

May 2, 2018

Let the AI benchmarking wars begin. Today, a diverse group from academia and industry – Google, Baidu, Intel, AMD, Harvard, and Stanford among them – released MLPerf, a nascent benchmarking tool “for measuring the speed of machine learning software and hardware.” Arrival of MLPerf follows what has been a smattering of ad hoc AI performance comparisons trickling to market. Last week, RiseML blog compared Google’s TPUv2 against Nvidia V100. Today Intel posted a blog with data showing for select machine translation using RNNs “the Intel Xeon Scalable processor outperforms NVidia V100 by 4x on the AWS Sockeye Neural Machine Translation model.”

For quite some time there has been vigorous discussion around the need for meaningful AI benchmarks with proponents suggesting that the lack of meaningful benchmark tools has restrained AI adoption. Quoted in the MLPerf announcement is AI pioneer Andrew Ng, “AI is transforming multiple industries, but for it to reach its full potential, we still need faster hardware and software.” The hope is better, standardized benchmarks will help AI technology developers create such products and allow adopters to make informed AI-enabling technology selections.

MLPerf says its primary goals are to:

  • Accelerate progress in ML via fair and useful measurement
  • Enable fair comparison of competing systems yet encourage innovation to improve the state-of-the-art of ML
  • Keep benchmarking effort affordable so all can participate
  • Serve both the commercial and research communities
  • Enforce replicability to ensure reliable results

Comparisons of AI performance (h/w and s/w) have so far largely been issued by parties with vested interest, such as Intel’s blog today entitled, “Amazing Inference Performance with Intel Xeon Scalable Processors.” This isn’t a knock on Intel. Such comparisons often contain useful insight, but they are also often structured to demonstrate one vendor’s superiority over a competitor. A standardized benchmark mitigates tweaking of tests to get the result one wants.

The MLPerf effort is emulating, for example, past efforts such as SPEC (The Standard Performance Evaluation Corporation). “[T]he SPEC benchmark helped accelerate improvements in general purpose computing. SPEC was introduced in 1988 by a consortium of computing companies. CPU Performance improved 1.6X/year for the next 15 years. MLPerf combines best practices from previous benchmarks including: SPEC’s use of a suite of programs, SORT’s use one division to enable comparisons and another division to foster innovative ideas, DeepBench’s coverage of software deployed in production, and DAWNBench’s time-to-accuracy metric,” says MLPerf.

Addison Snell, CEO of Intersect360 Research, noted, “AI is on the minds of so many enterprises today, that any effort to provide neutral benchmarking guidance is of heightened importance, especially with the range of competing technologies at play. However, AI is such a diverse field, I doubt any single benchmark will become dominant over time. Consider all the zeal around big data and analytics five years ago; despite everyone’s attempts to define it, the industry didn’t provide a unified, common benchmark. I expect the same will happen with AI.”

MLPerf is a “good and useful” step said Steve Conway, senior research vice president, Hyperion Research, “because there has been a real lack of benchmarks for buyers and sellers for years to show the differences between AI products and solutions. This benchmark appears to be written for bounded problems that predominate today in early AI. Later on we are going to need additional benchmarks as AI starts getting into unbounded problems that will be the most economically important problems. Bounded problems are relatively simple like voice and image recognition or game playing. An unbounded problem is diagnosing a cancer versus a bounded problem of reading an MRI; it’s being able to recommend decision on really complicated questions.”

MLPerf is available now on GitHub but still in a very early stage, as emphasized by MLPerf, “This release is very much an ‘alpha’ release — it could be improved in many ways. The benchmark suite is still being developed and refined, see the Suggestions section below to learn how to contribute. We anticipate a significant round of updates at the end of May based on input from users.”

Currently there are reference implementations for each of the seven benchmarks in the MLPerf suite (excerpted from GitHub):

  • Image classification– Resnet-50 v1 applied to Imagenet.
  • Object detection– Mask R-CNN applied to COCO.
  • Speech recognition– DeepSpeech2 applied to Librispeech.
  • Translation– Transformer applied to WMT English-German.
  • Recommendation– Neural Collaborative Filtering applied to MovieLens 20 Million (ml-20m).
  • Sentiment analysis– Seq-CNN applied to IMDB dataset.
  • Reinforcement– Mini-go applied to predicting pro game moves.

Each reference implementation provides the following: code that implements the model in at least one framework; a Dockerfile which can be used to run the benchmark in a container; a script which downloads the appropriate dataset; A script which runs and times training the model; and documentaiton on the dataset, model, and machine setup.

According to the GitHub site, the benchmarks have been tested on the following machine configuration:

  • 16 CPUs, one Nvidia P100.
  • Ubuntu 16.04, including docker with nvidia support.
  • 600GB of disk (though many benchmarks do require less disk).

It will be interesting to watch whether the industry coalesces around a few AI benchmarks or if benchmarks proliferate. In such a young market, many are likely to offer benchmarking tools and services. For example, Stanford – which is MLPerf member – recently ran its first DAWNBench v1 Deep Learning results.

Stanford reported: “April 20, 2018 marked first deep learning benchmark and competition that measures end-to-end performance: the time/cost required to achieve a state-of-the-art accuracy level for common deep learning tasks, as well as the latency/cost of inference at this state-of-the-art accuracy level. Focusing on end-to-end performance provided an objective means of normalizing across differences in computation frameworks, hardware, optimization algorithms, hyperparameter settings, and other factors that affect real-world performance.”

One DAWN competitor, fast.ai– a young company offering AI training and developing AI software tools – reached out to HPCwire touting its performance (see company blog for results). These benchmarks matter, and it seems very likely that any Stanford-run exercise is serious and should be taken seriously. That said, others may be less so. An effort such as MLPerf could help clear the currently muddy waters going forward when comparing AI claims.

Link to MLPerf user guide: https://mlperf.org/assets/static/media/MLPerf-User-Guide.pdf

* Additional reporting by Tiffany Trader

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

ASC18: Final Results Revealed & Wrapped Up

May 17, 2018

It was an exciting week at ASC18 in Nanyang, China. The student teams braved extreme heat, extremely difficult applications, and extreme competition in order to cross the cluster competition finish line. The gala awards ceremony took place on Wednesday. The auditorium was packed with student teams, various dignitaries, the media, and other interested parties. So what happened? Read more…

By Dan Olds

ASC18: Tough Applications & Tough Luck

May 17, 2018

The applications at the ASC18 Student Cluster Competition were tough. Tougher than the $3.99 steak special at your local greasy spoon restaurant. The apps are so tough that even Chuck Norris backs away from them slowly. Read more…

By Dan Olds

Spring Meetings Underscore Quantum Computing’s Rise

May 17, 2018

The month of April 2018 saw four very important and interesting meetings to discuss the state of quantum computing technologies, their potential impacts, and the technology challenges ahead. These discussions happened in Read more…

By Alex R. Larzelere

HPE Extreme Performance Solutions

HPC and AI Convergence is Accelerating New Levels of Intelligence

Data analytics is the most valuable tool in the digital marketplace – so much so that organizations are employing high performance computing (HPC) capabilities to rapidly collect, share, and analyze endless streams of data. Read more…

IBM Accelerated Insights

Mastering the Big Data Challenge in Cognitive Healthcare

Patrick Chain, genomics researcher at Los Alamos National Laboratory, posed a question in a recent blog: What if a nurse could swipe a patient’s saliva and run a quick genetic test to determine if the patient’s sore throat was caused by a cold virus or a bacterial infection? Read more…

Quantum Network Hub Opens in Japan

May 17, 2018

Following on the launch of its Q Commercial quantum network last December with 12 industrial and academic partners, the official Japanese hub at Keio University is now open to facilitate the exploration of quantum applications important to science and business. The news comes a week after IBM announced that North Carolina State University was the first U.S. university to join its Q Network. Read more…

By Tiffany Trader

ASC18: Final Results Revealed & Wrapped Up

May 17, 2018

It was an exciting week at ASC18 in Nanyang, China. The student teams braved extreme heat, extremely difficult applications, and extreme competition in order to cross the cluster competition finish line. The gala awards ceremony took place on Wednesday. The auditorium was packed with student teams, various dignitaries, the media, and other interested parties. So what happened? Read more…

By Dan Olds

Spring Meetings Underscore Quantum Computing’s Rise

May 17, 2018

The month of April 2018 saw four very important and interesting meetings to discuss the state of quantum computing technologies, their potential impacts, and th Read more…

By Alex R. Larzelere

Quantum Network Hub Opens in Japan

May 17, 2018

Following on the launch of its Q Commercial quantum network last December with 12 industrial and academic partners, the official Japanese hub at Keio University is now open to facilitate the exploration of quantum applications important to science and business. The news comes a week after IBM announced that North Carolina State University was the first U.S. university to join its Q Network. Read more…

By Tiffany Trader

Democratizing HPC: OSC Releases Version 1.3 of OnDemand

May 16, 2018

Making HPC resources readily available and easier to use for scientists who may have less HPC expertise is an ongoing challenge. Open OnDemand is a project by t Read more…

By John Russell

PRACE 2017 Annual Report: Exascale Aspirations; Industry Collaboration; HPC Training

May 15, 2018

The Partnership for Advanced Computing in Europe (PRACE) today released its annual report showcasing 2017 activities and providing a glimpse into thinking about Read more…

By John Russell

US Forms AI Brain Trust

May 11, 2018

Amid calls for a U.S. strategy for promoting AI development, the Trump administration is forming a senior-level panel to help coordinate government and industry research efforts. The Select Committee on Artificial Intelligence was announced Thursday (May 10) during a White House summit organized by the Office of Science and Technology Policy (OSTP). Read more…

By George Leopold

Emerging Advanced Scale Tech Trends Focus of Annual Tabor Conference

May 9, 2018

At Tabor Communications' annual Advanced Scale Forum (ASF) held this week in Austin, the focus was on enterprise adoption of HPC-class technologies and high performance data analytics (HPDA). It’s a confab that brings together end users (CIOs, IT planners, department heads) and vendors and encourages... Read more…

By the Editorial Team

Google I/O 2018: AI Everywhere; TPU 3.0 Delivers 100+ Petaflops but Requires Liquid Cooling

May 9, 2018

All things AI dominated discussion at yesterday’s opening of Google’s I/O 2018 developers meeting covering much of Google's near-term product roadmap. The e Read more…

By John Russell

MLPerf – Will New Machine Learning Benchmark Help Propel AI Forward?

May 2, 2018

Let the AI benchmarking wars begin. Today, a diverse group from academia and industry – Google, Baidu, Intel, AMD, Harvard, and Stanford among them – releas Read more…

By John Russell

How the Cloud Is Falling Short for HPC

March 15, 2018

The last couple of years have seen cloud computing gradually build some legitimacy within the HPC world, but still the HPC industry lies far behind enterprise I Read more…

By Chris Downing

Russian Nuclear Engineers Caught Cryptomining on Lab Supercomputer

February 12, 2018

Nuclear scientists working at the All-Russian Research Institute of Experimental Physics (RFNC-VNIIEF) have been arrested for using lab supercomputing resources to mine crypto-currency, according to a report in Russia’s Interfax News Agency. Read more…

By Tiffany Trader

Inventor Claims to Have Solved Floating Point Error Problem

January 17, 2018

"The decades-old floating point error problem has been solved," proclaims a press release from inventor Alan Jorgensen. The computer scientist has filed for and Read more…

By Tiffany Trader

Researchers Measure Impact of ‘Meltdown’ and ‘Spectre’ Patches on HPC Workloads

January 17, 2018

Computer scientists from the Center for Computational Research, State University of New York (SUNY), University at Buffalo have examined the effect of Meltdown Read more…

By Tiffany Trader

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Deep Learning at 15 PFlops Enables Training for Extreme Weather Identification at Scale

March 19, 2018

Petaflop per second deep learning training performance on the NERSC (National Energy Research Scientific Computing Center) Cori supercomputer has given climate Read more…

By Rob Farber

AI Cloud Competition Heats Up: Google’s TPUs, Amazon Building AI Chip

February 12, 2018

Competition in the white hot AI (and public cloud) market pits Google against Amazon this week, with Google offering AI hardware on its cloud platform intended Read more…

By Doug Black

Leading Solution Providers

US Plans $1.8 Billion Spend on DOE Exascale Supercomputing

April 11, 2018

On Monday, the United States Department of Energy announced its intention to procure up to three exascale supercomputers at a cost of up to $1.8 billion with th Read more…

By Tiffany Trader

Lenovo Unveils Warm Water Cooled ThinkSystem SD650 in Rampup to LRZ Install

February 22, 2018

This week Lenovo took the wraps off the ThinkSystem SD650 high-density server with third-generation direct water cooling technology developed in tandem with par Read more…

By Tiffany Trader

HPC and AI – Two Communities Same Future

January 25, 2018

According to Al Gara (Intel Fellow, Data Center Group), high performance computing and artificial intelligence will increasingly intertwine as we transition to Read more…

By Rob Farber

Google Chases Quantum Supremacy with 72-Qubit Processor

March 7, 2018

Google pulled ahead of the pack this week in the race toward "quantum supremacy," with the introduction of a new 72-qubit quantum processor called Bristlecone. Read more…

By Tiffany Trader

HPE Wins $57 Million DoD Supercomputing Contract

February 20, 2018

Hewlett Packard Enterprise (HPE) today revealed details of its massive $57 million HPC contract with the U.S. Department of Defense (DoD). The deal calls for HP Read more…

By Tiffany Trader

CFO Steps down in Executive Shuffle at Supermicro

January 31, 2018

Supermicro yesterday announced senior management shuffling including prominent departures, the completion of an audit linked to its delayed Nasdaq filings, and Read more…

By John Russell

Deep Learning Portends ‘Sea Change’ for Oil and Gas Sector

February 1, 2018

The billowing compute and data demands that spurred the oil and gas industry to be the largest commercial users of high-performance computing are now propelling Read more…

By Tiffany Trader

Nvidia Ups Hardware Game with 16-GPU DGX-2 Server and 18-Port NVSwitch

March 27, 2018

Nvidia unveiled a raft of new products from its annual technology conference in San Jose today, and despite not offering up a new chip architecture, there were still a few surprises in store for HPC hardware aficionados. Read more…

By Tiffany Trader

  • arrow
  • Click Here for More Headlines
  • arrow
Share This