[Paper Review] CGO/PPoPP'17 Artifact Evaluation Discussion (enabling open and reproducible research)

Grigori Fursin|arXiv (Cornell University)|Feb 19, 2017

Scientific Computing and Data Management3 references3 citations

TL;DR

This paper proposes a community-driven, open-access model for research publication and artifact evaluation to enhance reproducibility and collaboration in computer systems research. By sharing full experimental artifacts—including code, data, and metadata—via public repositories like cTuning.org, the authors enable crowd-sourced validation, public discussion, and iterative improvement, significantly reducing review burden and improving fairness and transparency in academic evaluation.

ABSTRACT

<em>14 March 2017, CNRS webinar, Grenoble, France</em> A decade ago my research nearly stalled. I was investigating how to crowdsource performance analysis and optimization of realistic workloads across diverse hardware provided by volunteers and combine it with machine learning [1]. Often, it was simply impossible to reproduce crowdsourced empirical results and build predictive models due to continuously changing software and hardware stacks. Worse still, lack of realistic workloads and representative data sets in our community severely limited the usefulness of such models.<br> <br> All these problems motivated me to create a public portal (cTuning.org) to share, validate and reuse workloads, data sets, tools, experimental results, and predictive models while involving the community in this effort [2]. This experience, in turn, helped us to initiate the so-called Artifact Evaluation (AE) at ACM conferences on parallel programming, architecture and code generation (ASPLOS, CGO, PPoPP, PACT, SC and MLSys). AE aims to independently validate experimental results reported in the publications and to encourage code and data sharing. These slides are from my webinar <em>“Enabling open and reproducible research at computer systems conferences: the good, the bad and the ugly”</em> at CNRS Grenoble (14 March 2017). I shared my practical experience organizing Artifact Evaluation over the past years, along with encountered problems and possible solutions.<br> <br> On the one hand, we have received incredible support from the research community, ACM, universities, and companies. We have even received a record number of artifact submissions at the CGO/PPoPP'17 AE (27 vs 17 two years ago) sponsored by NVIDIA and the cTuning foundation. We have also introduced Artifact Appendices and co-authored the new ACM Result and Artifact Review and Badging policy now used at Supercomputing. <br> <br> On the other hand, the use of proprietary benchmarks, rare hardware platforms, and totally ad-hoc scripts to set up, run and process experiments all place a huge burden on evaluators. It is simply too difficult and time-consuming to customize and rebuild experimental setups, reuse artifacts and eventually build upon others’ efforts - the main pillars of open science!<br> <br> I then present Collective Knowledge (CK), my attempt to introduce a customizable workflow framework with a unified JSON API and a cross-platform package manager, that can automate ML&amp;systems R&amp;D and enable live papers while automatically adapting to continuously evolving software and hardware [3]. I also demonstrate a practical CK workflow to collaboratively optimize deep learning across different compilers, libraries, data sets and diverse platforms from resource-constrained mobile devices to data centers (see our Android app to crowdsource DNN optimization across diverse mobile devices provided by volunteers, and the public repository with results) [4].<br> <br> Finally, I describe our novel publication model to reproduce results from published papers with the help of the community [5]. <br> <br> <em>Please feel free to contact me at [e-mail] if you have any questions or comments! I am looking forward to your feedback! </em> <strong>References</strong> “Milepost GCC: Machine learning enabled self-tuning compiler”, International journal of parallel programming, Volume 39, Issue 3, pp.296-327, 2009 “Collective Tuning Initiative: automating and accelerating development and optimization of computing systems”, GCC Developers' Summit, Montreal, Canada. 2009 “Collective Knowledge: towards R&amp;D sustainability”, Proceedings of the Conference on Design, Automation, and Test in Europe (DATE), 2016 “Optimizing Convolutional Neural Networks on Embedded Platforms with OpenCL”, IWOCL'16, Vienna, Austria, 2016 “Community-driven reviewing and validation of publications”, Proceedings of the 1st ACM SIGPLAN Workshop on Reproducible Research Methodologies and New Publication Models in Computer Engineering @ PLDI’14, Edinburgh, UK

Motivation & Objective

• Address the widespread reproducibility crisis in computer systems research due to missing or incomplete experimental artifacts.
• Overcome the limitations of traditional peer review, which is often slow, biased, or unfair, especially under high submission pressure.
• Foster a culture of sharing and collaboration by enabling public, transparent, and time-stamped evaluation of research artifacts and publications.
• Reduce the burden on reviewers by leveraging community feedback and pre-validated artifacts before formal review.
• Demonstrate that open access and collaborative validation can coexist with traditional publishing while enhancing scientific integrity and impact.

Proposed method

• Deploy a public web portal (cTuning.org) to host and share research artifacts, including benchmarks, datasets, tools, and predictive models.
• Integrate experimental results with detailed meta-information (features) to support machine learning and data mining on optimization techniques.
• Use open access publication to enable public discussion and community review, exemplified by high-visibility exposure on platforms like Slashdot.
• Implement a model where public feedback and community validation precede or supplement formal peer review, improving review quality and relevance.
• Leverage containerization (e.g., Docker), provenance tracking (e.g., CARE, Collective Mind), and versioned repositories to ensure reproducibility of complex experimental setups.
• Support coexistence with traditional publishing by allowing open access pre-prints and extended versions, while preserving options for patenting or commercialization.

Experimental results

Research questions

• RQ1 How can community-driven artifact evaluation improve the reproducibility and reliability of experimental results in computer systems research?
• RQ2 What mechanisms can reduce the burden on traditional reviewers while enhancing the fairness and transparency of publication evaluation?
• RQ3 Can public, open-access platforms for research artifacts and results lead to faster validation, improved model accuracy, and broader scientific impact?
• RQ4 How can negative or failed results be meaningfully shared and used to prevent redundant research and accelerate progress?
• RQ5 What technical and cultural changes are needed to support sustainable, collaborative, and reproducible research in computer engineering?

Key findings

• The community served as a de facto reviewer for the authors’ open-access publication, providing valuable feedback, identifying missing citations, and improving the work through public discussion.
• Public exposure on platforms like Slashdot generated significant community engagement, including 150+ comments, demonstrating the potential for public validation and rapid feedback loops.
• The approach enabled early adoption and recognition by industry leaders: IBM referenced the work in 2009 for embedded systems optimization, and Fujitsu cited it in 2014 for Exascale systems.
• Open access and shared artifacts allowed the authors to later extend their work into a special issue of the International Journal of Parallel Programming (IJPP), showing compatibility with traditional publishing.
• The model naturally supports reproducibility as a byproduct, reducing reliance on manual verification and improving the quality and fairness of peer review.
• The authors observed that negative results and failed experiments could be meaningfully shared and discussed, preventing redundant work and enabling collective improvement—despite this practice remaining underutilized in the community.