Study reproducibility has long been a challenge in research, and science’s increasing dependence on high performance computing has worsened the problem. Data sets and software are often critical by-products of studies but are typically missing from published reports – this hobbles efforts to repeat studies and sometimes lessens confidence in the results.
In a blog post today on the SC16 web site, researcher Michela Taufer of the University of Delaware outlined an effort to address the reproducibility challenge. The idea is to ensure computational results are as reproducible as ‘traditional’ experimental results.
Taufer wrote she is part of a small team at SC16, “that over the next several years will work to promote and support replication and reproducibility of computational results. One of the approaches we are exploring is to use SC’s Student Cluster Competition (SCC) to test the reproducibility of results presented in papers at previous SC conferences. This is the first time that students have been challenged to reproduce results from a recent computational paper rather than run more traditional benchmark applications.”
After evaluating submissions from past SC paper authors, the SCC selected “A parallel connectivity algorithm for de Bruijn graphs in metagenomic applications” by Flick, Jain, Pan, and Aluru for the inaugural reproducibility initiative at SC16 (the paper is available from the ACM Digital Library). Results from this paper will be reproduced by the 14 teams taking part in the cluster competition in November.
“We hope our student teams will be especially engaged by the idea that they aren’t running just a sterile benchmark application but rather, by attempting to reproduce previously reported application performance, they are actively taking part in the scientific process. We believe this kind of early engagement will be very important to efforts to make reproducibility a standard part of the computational research process,” wrote Taufer, SC16 Panels Chair and a SC16 SCC Reproducibility Committee Member
One of the paper’s co-authors, Chirag Jain, will be helping the SCC committee to create a challenging competition task for the students and will serve as a judge during the competition. SC16 is in Salt Lake City, November 13-18. Here is a link to the full blog: http://sc16.supercomputing.org/2016/03/16/sc16-explores-reproducibility-advanced-computing-student-competition-michela-taufer/