Sept. 28 — Authors of a paper judged at SC15 in Austin, Texas, have transformed a portion of their experiment into an application challenge for the SC16 Student Cluster Competition (SCC) as part of a larger effort to raise the bar for experimental research in computing.

SC is responding to the growing need for scientific repeatability, replicability and reproducibility (the three Rs) of work in scientific papers to safeguard the integrity of the research ecosystem. SC has joined with the Association for Computing Machinery (ACM) to award papers judged at the annual conference with a badge to acknowledge the level of the three Rs in the paper’s artifacts—digital objects that were either created by the authors to be used as part of the study or generated by the experiment itself.

“SC is in complete agreement with ACM regarding the need to increase the integrity of scholarly work being conducted in the high performance computing ecosystem, and the Three Rs initiative is an essential piece of that effort,” said Cherri Pancake, ACM vice-president, SC16 vice-chair and professor and Intel Faculty Fellow in EECS at Oregon State University.

In addition to SC, a number of ACM conferences and journals have already instituted formal processes for artifact review. ACM provides the terminology and standards for review processes in order to promote a base level of uniformity that will enable labeling of successfully reviewed papers across ACM publications choosing to adopt such practices.

The SC15 paper being highlighted this year was selected following a call for application proposal that was distributed across authors of papers at SC13, SC14 and SC15. The paper, “A parallel connectivity algorithm for de Bruijn graphs in metagenomic applications,” was authored by Patrick Flick, Chirag Jain, Tony Pan and Srinivas Aluru, all from the Georgia Institute of Technology in Atlanta.

The authors of this paper worked with the SCC organizers to transform their artifact into an application challenge at the SC16 competition. The authors will receive a certificate of appreciation from SIGHPC at the SC16 Award Ceremony, and the paper will receive the Results Replicated badge in the ACM Digital Library.

“A component of the SCC is for students to compete by running a real scientific workload using real applications,” explained Stephen Harrell, SC16 SCC chair and a scientific applications analyst at Purdue University. “This effort provides students with a workload associated with a scientific paper, where they now are expected to extend beyond running given data sets to using the paper and artifacts to participate in scientific reproducibility.”

At SC16, authors of submitted papers were asked whether they want to participate in the next step of the SC initiative. The next step is to add a two-page artifact descriptor to the paper documenting the code, data and workflow in the paper. Authors of 34 submissions expressed interest, and authors of nine took part by adding a two-page artifact descriptor at the end of the paper:

1. Pierre Jolivet and Pierre-Henri: “Tournier Block Iterative Methods and Recycling for Improved Scalability of Linear Solvers”
2. Duane Merrill and Michael Garland: “Merge-based Parallel Sparse Matrix-Vector Multiplication”
3. Sangeetha Abdu Jyothi, Ankit Singla, P. Brighten Godfrey and Alexandra Kolla: “Measuring and Understanding Throughput of Network Topologies”
4. W. Michael Brown, Andrey Semin, Michael Hebenstreit, Sergey Khvostov, Karthik Raman and Steven J. Plimpton: “Increasing Molecular Dynamics Simulation Rates with an 8-Fold Increase in Electrical Power Efficiency”
5. Ehab Abdelhamid, Ibrahim Abdelaziz, Panos Kalnis, Zuhair Khayyat and Fuad Jamour: “ScaleMine: Scalable Parallel Frequent Subgraph Mining in a Single Large Graph”
6. Markus Hoehnerbach, Ahmed E. Ismail and Paolo Bientinesi: “The Vectorization of the Tersoff Multi-Body Potential: An Exercise in Performance Portability”
7. Shaden Smith, Jongsoo Parky and George Karypis: “An Exploration of Optimization Algorithms for High Performance Tensor Completion”
8. Maria Dimakopoulou, St´ephane Eranian, Nectarios Koziris and Nicholas Bambos: “Reliable and Efficient Performance Monitoring in Linux”
9. Preeti Malakar, Venkatram Vishwanath, Christopher Knight, Todd Munson and Michael E. Papka: “Optimal Execution of Co-analysis for Large-scale Molecular Dynamics Simulations”

These papers will include the two-page appendix in the proceedings and have the Artifacts Available badge attached in the ACM Digital Library. One of the nine papers—to be announced in December 2016—will serve as the SC17 SCC application.

“SC is looking at extending the artifact descriptor to all the papers and make it a valuable support towards repeatability, replicability and reproducibility of the research presented at the conference,” said John West, SC16 General Chair from the Texas Advanced Computing Center.

---

Source: SC16