SC17 Cluster Competition: Who Won and Why? Results Analyzed and Over-Analyzed

By Dan Olds

November 28, 2017

Everyone by now knows that Nanyang Technological University of Singapore (NTU) took home the highest LINPACK Award and the Overall Championship from the recently concluded SC17 Student Cluster Competition.

We also already know how the teams did in the Highest LINPACK and Highest HPCG competitions, with Nanyang grabbing bragging rights for both benchmarks.

Now it’s time to dive into the results, see how the other teams did, and figure out what’s what. Let’s walk through the application and task results and see what happened.

The Interview: All of the teams did pretty well on the interview portion of the competition. This is a pressure packed part of the competition. HPC subject matter experts grill the students on everything from how they configured their cluster to detailed questions about each of the benchmarks and applications. There’s no hiding from their probing questions.

Nanyang had the highest interview score, notching an almost perfect 97%, but they were closely followed by Team Texas and Tsinghua, who tied for second with 96%.

Team Chicago Fusion (IIT/MHS/SHS) deserves an honorable mention for only being 3% off the winning mark on the interview portion of the scoring.

All of the teams did well in this area, as you can tell by the average/median score of 93%.

The ‘mystery application’ is an app that students only learn about when they’re at the competition. There’s no preparing for it, it’s like suddenly being told in a basketball game that for one quarter, the hoop height will be increased to 15 feet or decreased to five.

The mystery app for 2017 is MPAS A, an application developed Los Alamos National Lab and the National Center for Atmospheric Research to build weather simulations. Students were given the task of modeling what would happen to the rest of the atmosphere if excess carbon was sequestered in Antarctica.

This is Team Chicago Fusion’s best application – they nailed it and left it for dead with a score of 100%. Nanyang almost scored the bullseye with a score of 99% and Tsinghua was an eyelash behind, posting a score of 98%. NTHU finished just out of the money with a 97% score.

As you can see by the high median score, most of the teams were bunched up on the good side of the average – meaning that most teams scored well on this application with a few outliers on the low low side.

The next task up is the Reproducibility exercise. This is where the teams take a paper that was submitted for SC16 and try to reproduce the results – either proving the paper is valid, or…well, not so valid.

The paper this year has an intriguing title, “The Vectorization of the Tersoff Multi-Body Potential: An Exercise in Performance Portability”, and shows how to use a vectorization scheme to achieve high cross-platform (CPU and accelerator) performance.

Student teams have to use the artifact section of the paper to reproduce the results and either prove or disprove the paper, then submit a report detailing and justifying how they arrived at their conclusion.

Nanyang posted another win, building on their lead over the rest of the pack. Team Texas took home second place, only six points behind Nanyang. NEU finds the winner board for the first time in the competition with their third place showing.

Team Chicago Fusion gets an Honorable Mention for their score of 82%, just a couple of points away from second and third place, while Team Illinois Urbana-Champaign and Taiwan’s NTHU finish in a virtual tie at 80% and 79% (and some change) separating them.

The rest of the teams had at least some trouble with this task as witnessed by the median being significantly higher than the mean score. This indicates that there are several teams who encountered difficulties completing this task. But, hey, who said this was going to be easy?

Speaking of things that are difficult, how about that MrBayes? This year, the students were using MrBayes to examine how viruses transmitted by white flies are impacting cassava production in Africa.

This wasn’t an easy application for most of the teams. While Tsinghua pulled down a 99% score, closely trailed by Nanyang with 98%, the average score on this app was only 67% and the median was 64%.

This was a great app for NEU, however, with their 96% score putting them in the winners circle. Team Chicago Fusion was just a few fractions of a point behind NEU, nabbing the Honorable Mention.

The most difficult application in this edition of the cluster competition looks to be Born, a seismic imaging app used to identify oil and gas reserves. It’s not that this was necessarily the most complicated or difficult to understand application, it’s that it was so damned time consuming. And it’s the time consuming nature of Born that separated the teams in the final accounting.

The teams had to try to process 1,136 Born “shots.” Each shot is independent of the others, which makes for an embarrassingly parallel application – great, right? Well, no. Running on CPUs alone, each Born shot takes somewhere between two and three hours. Ouch.

Several of the teams decided to use their cloud budget and run a bunch of Born instances in the cloud. While this was a good idea, the teams didn’t have enough cloud capacity to run all that much Born – particularly since each shot took so long to complete.

The best approach was to port Born onto GPUs, as four or five teams proved. The top teams on our leaderboard all ported Born and realized great dividends. Tsinghua completed the entire 1,136 slate of datasets and posted a score of 99%. Nanyang also completed all of the datasets and took home second place with their score of 90%. NTHU was a nanometer behind and grabbed third place. USTC gets an honorable mention for posting a score of 83%.

The rest of the teams didn’t do so hot on this one. The average score was competition low of 63% with the median score at 55%. This was a tough mountain to climb and if you didn’t port Born over to GPUs, you didn’t have a chance to complete all of the datasets, even if you were able to devote your entire cluster to it.

Looking at the final stats, Nanyang was the clear winner with an astounding 95 out of a possible 100 points. NTHU and Tsinghua finished very close together, with NTHU nabbing second place by fractions of a percent. Team Peking, a relative newbie with this being their second appearance in the competition, takes home an honorable mention.

These teams finished above the rest of the pack by a respectable margin, as shown by the average score of 70% and the median score of 71%. But, at the end of the day, all of the teams were winners. Everyone showed up, no one gave up, and everyone learned a lot (including me, in fact).

So that’s another student cluster competition in the books. If you’ve missed any of our coverage, you can catch up on it using the following links.

For an intro into the high-stakes world of student cluster competitions, look here.

If you want to see what kind of hardware the teams are driving, here are their configs.

If you want to see the applications and tasks for this year’s event, click your clicker here.

To meet this year’s teams via our video interviews, click here for the American teams, here for the European teams, and here for the Asian teams.

One final note: the bettors in our betting pool were woefully uninformed. The ‘smart money’ was pretty dumb this year, given that the winning Nanyang team was placed as a 35-1 underdog. Wow, if this was a real pool, anyone betting on Nanyang would have really cleaned up!

We’ll be back with more Student Cluster Competition features, more competitions, and even better coverage in 2018. Stay tuned.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

IBM Launches Commercial Quantum Network with Samsung, ORNL

December 14, 2017

In the race to commercialize quantum computing, IBM is one of several companies leading the pack. Today, IBM announced it had signed JPMorgan Chase, Daimler AG, Samsung and a number of other corporations to its IBM Q Net Read more…

By Tiffany Trader

TACC Researchers Test AI Traffic Monitoring Tool in Austin

December 13, 2017

Traffic jams and mishaps are often painful and sometimes dangerous facts of life. At this week’s IEEE International Conference on Big Data being held in Boston, researchers from TACC and colleagues will present a new Read more…

By HPCwire Staff

AMD Wins Another: Baidu to Deploy EPYC on Single Socket Servers

December 13, 2017

When AMD introduced its EPYC chip line in June, the company said a portion of the line was specifically designed to re-invigorate a single socket segment in what has become an overwhelmingly two-socket landscape in the d Read more…

By John Russell

HPE Extreme Performance Solutions

Explore the Origins of Space with COSMOS and Memory-Driven Computing

From the formation of black holes to the origins of space, data is the key to unlocking the secrets of the early universe. Read more…

Microsoft Wants to Speed Quantum Development

December 12, 2017

Quantum computing continues to make headlines in what remains of 2017 as several tech giants jockey to establish a pole position in the race toward commercialization of quantum. This week, Microsoft took the next step in Read more…

By Tiffany Trader

IBM Launches Commercial Quantum Network with Samsung, ORNL

December 14, 2017

In the race to commercialize quantum computing, IBM is one of several companies leading the pack. Today, IBM announced it had signed JPMorgan Chase, Daimler AG, Read more…

By Tiffany Trader

AMD Wins Another: Baidu to Deploy EPYC on Single Socket Servers

December 13, 2017

When AMD introduced its EPYC chip line in June, the company said a portion of the line was specifically designed to re-invigorate a single socket segment in wha Read more…

By John Russell

Microsoft Wants to Speed Quantum Development

December 12, 2017

Quantum computing continues to make headlines in what remains of 2017 as several tech giants jockey to establish a pole position in the race toward commercializ Read more…

By Tiffany Trader

HPC Iron, Soft, Data, People – It Takes an Ecosystem!

December 11, 2017

Cutting edge advanced computing hardware (aka big iron) does not stand by itself. These computers are the pinnacle of a myriad of technologies that must be care Read more…

By Alex R. Larzelere

IBM Begins Power9 Rollout with Backing from DOE, Google

December 6, 2017

After over a year of buildup, IBM is unveiling its first Power9 system based on the same architecture as the Department of Energy CORAL supercomputers, Summit a Read more…

By Tiffany Trader

Microsoft Spins Cycle Computing into Core Azure Product

December 5, 2017

Last August, cloud giant Microsoft acquired HPC cloud orchestration pioneer Cycle Computing. Since then the focus has been on integrating Cycle’s organization Read more…

By John Russell

GlobalFoundries, Ayar Labs Team Up to Commercialize Optical I/O

December 4, 2017

GlobalFoundries (GF) and Ayar Labs, a startup focused on using light, instead of electricity, to transfer data between chips, today announced they've entered in Read more…

By Tiffany Trader

HPE In-Memory Platform Comes to COSMOS

November 30, 2017

Hewlett Packard Enterprise is on a mission to accelerate space research. In August, it sent the first commercial-off-the-shelf HPC system into space for testing Read more…

By Tiffany Trader

US Coalesces Plans for First Exascale Supercomputer: Aurora in 2021

September 27, 2017

At the Advanced Scientific Computing Advisory Committee (ASCAC) meeting, in Arlington, Va., yesterday (Sept. 26), it was revealed that the "Aurora" supercompute Read more…

By Tiffany Trader

NERSC Scales Scientific Deep Learning to 15 Petaflops

August 28, 2017

A collaborative effort between Intel, NERSC and Stanford has delivered the first 15-petaflops deep learning software running on HPC platforms and is, according Read more…

By Rob Farber

Oracle Layoffs Reportedly Hit SPARC and Solaris Hard

September 7, 2017

Oracle’s latest layoffs have many wondering if this is the end of the line for the SPARC processor and Solaris OS development. As reported by multiple sources Read more…

By John Russell

AMD Showcases Growing Portfolio of EPYC and Radeon-based Systems at SC17

November 13, 2017

AMD’s charge back into HPC and the datacenter is on full display at SC17. Having launched the EPYC processor line in June along with its MI25 GPU the focus he Read more…

By John Russell

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Japan Unveils Quantum Neural Network

November 22, 2017

The U.S. and China are leading the race toward productive quantum computing, but it's early enough that ultimate leadership is still something of an open questi Read more…

By Tiffany Trader

GlobalFoundries Puts Wind in AMD’s Sails with 12nm FinFET

September 24, 2017

From its annual tech conference last week (Sept. 20), where GlobalFoundries welcomed more than 600 semiconductor professionals (reaching the Santa Clara venue Read more…

By Tiffany Trader

Amazon Debuts New AMD-based GPU Instances for Graphics Acceleration

September 12, 2017

Last week Amazon Web Services (AWS) streaming service, AppStream 2.0, introduced a new GPU instance called Graphics Design intended to accelerate graphics. The Read more…

By John Russell

Leading Solution Providers

Google Releases Deeplearn.js to Further Democratize Machine Learning

August 17, 2017

Spreading the use of machine learning tools is one of the goals of Google’s PAIR (People + AI Research) initiative, which was introduced in early July. Last w Read more…

By John Russell

Perspective: What Really Happened at SC17?

November 22, 2017

SC is over. Now comes the myriad of follow-ups. Inboxes are filled with templated emails from vendors and other exhibitors hoping to win a place in the post-SC thinking of booth visitors. Attendees of tutorials, workshops and other technical sessions will be inundated with requests for feedback. Read more…

By Andrew Jones

IBM Begins Power9 Rollout with Backing from DOE, Google

December 6, 2017

After over a year of buildup, IBM is unveiling its first Power9 system based on the same architecture as the Department of Energy CORAL supercomputers, Summit a Read more…

By Tiffany Trader

EU Funds 20 Million Euro ARM+FPGA Exascale Project

September 7, 2017

At the Barcelona Supercomputer Centre on Wednesday (Sept. 6), 16 partners gathered to launch the EuroEXA project, which invests €20 million over three-and-a-half years into exascale-focused research and development. Led by the Horizon 2020 program, EuroEXA picks up the banner of a triad of partner projects — ExaNeSt, EcoScale and ExaNoDe — building on their work... Read more…

By Tiffany Trader

Delays, Smoke, Records & Markets – A Candid Conversation with Cray CEO Peter Ungaro

October 5, 2017

Earlier this month, Tom Tabor, publisher of HPCwire and I had a very personal conversation with Cray CEO Peter Ungaro. Cray has been on something of a Cinderell Read more…

By Tiffany Trader & Tom Tabor

Tensors Come of Age: Why the AI Revolution Will Help HPC

November 13, 2017

Thirty years ago, parallel computing was coming of age. A bitter battle began between stalwart vector computing supporters and advocates of various approaches to parallel computing. IBM skeptic Alan Karp, reacting to announcements of nCUBE’s 1024-microprocessor system and Thinking Machines’ 65,536-element array, made a public $100 wager that no one could get a parallel speedup of over 200 on real HPC workloads. Read more…

By John Gustafson & Lenore Mullin

Flipping the Flops and Reading the Top500 Tea Leaves

November 13, 2017

The 50th edition of the Top500 list, the biannual publication of the world’s fastest supercomputers based on public Linpack benchmarking results, was released Read more…

By Tiffany Trader

Intel Launches Software Tools to Ease FPGA Programming

September 5, 2017

Field Programmable Gate Arrays (FPGAs) have a reputation for being difficult to program, requiring expertise in specialty languages, like Verilog or VHDL. Easin Read more…

By Tiffany Trader

  • arrow
  • Click Here for More Headlines
  • arrow
Share This