[论文解读] CGO/PPoPP'17 Artifact Evaluation Discussion (enabling open and reproducible research)

Grigori Fursin|arXiv (Cornell University)|Feb 19, 2017

Scientific Computing and Data Management参考文献 3被引用 3

一句话总结

本文提出了一种由社区驱动的、开放获取的研究出版与成果物评估模型，旨在提升计算机系统研究中的可重现性与协作性。通过将完整的实验成果物（包括代码、数据和元数据）通过 cTuning.org 等公共仓库共享，作者实现了众包验证、公开讨论与迭代改进，显著减轻了评审负担，并提升了学术评价的公平性与透明度。

ABSTRACT

<em>14 March 2017, CNRS webinar, Grenoble, France</em> A decade ago my research nearly stalled. I was investigating how to crowdsource performance analysis and optimization of realistic workloads across diverse hardware provided by volunteers and combine it with machine learning [1]. Often, it was simply impossible to reproduce crowdsourced empirical results and build predictive models due to continuously changing software and hardware stacks. Worse still, lack of realistic workloads and representative data sets in our community severely limited the usefulness of such models.<br> <br> All these problems motivated me to create a public portal (cTuning.org) to share, validate and reuse workloads, data sets, tools, experimental results, and predictive models while involving the community in this effort [2]. This experience, in turn, helped us to initiate the so-called Artifact Evaluation (AE) at ACM conferences on parallel programming, architecture and code generation (ASPLOS, CGO, PPoPP, PACT, SC and MLSys). AE aims to independently validate experimental results reported in the publications and to encourage code and data sharing. These slides are from my webinar <em>“Enabling open and reproducible research at computer systems conferences: the good, the bad and the ugly”</em> at CNRS Grenoble (14 March 2017). I shared my practical experience organizing Artifact Evaluation over the past years, along with encountered problems and possible solutions.<br> <br> On the one hand, we have received incredible support from the research community, ACM, universities, and companies. We have even received a record number of artifact submissions at the CGO/PPoPP'17 AE (27 vs 17 two years ago) sponsored by NVIDIA and the cTuning foundation. We have also introduced Artifact Appendices and co-authored the new ACM Result and Artifact Review and Badging policy now used at Supercomputing. <br> <br> On the other hand, the use of proprietary benchmarks, rare hardware platforms, and totally ad-hoc scripts to set up, run and process experiments all place a huge burden on evaluators. It is simply too difficult and time-consuming to customize and rebuild experimental setups, reuse artifacts and eventually build upon others’ efforts - the main pillars of open science!<br> <br> I then present Collective Knowledge (CK), my attempt to introduce a customizable workflow framework with a unified JSON API and a cross-platform package manager, that can automate ML&amp;systems R&amp;D and enable live papers while automatically adapting to continuously evolving software and hardware [3]. I also demonstrate a practical CK workflow to collaboratively optimize deep learning across different compilers, libraries, data sets and diverse platforms from resource-constrained mobile devices to data centers (see our Android app to crowdsource DNN optimization across diverse mobile devices provided by volunteers, and the public repository with results) [4].<br> <br> Finally, I describe our novel publication model to reproduce results from published papers with the help of the community [5]. <br> <br> <em>Please feel free to contact me at [e-mail] if you have any questions or comments! I am looking forward to your feedback! </em> <strong>References</strong> “Milepost GCC: Machine learning enabled self-tuning compiler”, International journal of parallel programming, Volume 39, Issue 3, pp.296-327, 2009 “Collective Tuning Initiative: automating and accelerating development and optimization of computing systems”, GCC Developers' Summit, Montreal, Canada. 2009 “Collective Knowledge: towards R&amp;D sustainability”, Proceedings of the Conference on Design, Automation, and Test in Europe (DATE), 2016 “Optimizing Convolutional Neural Networks on Embedded Platforms with OpenCL”, IWOCL'16, Vienna, Austria, 2016 “Community-driven reviewing and validation of publications”, Proceedings of the 1st ACM SIGPLAN Workshop on Reproducible Research Methodologies and New Publication Models in Computer Engineering @ PLDI’14, Edinburgh, UK

研究动机与目标

• 解决因实验成果物缺失或不完整而导致的计算机系统研究中广泛存在的可重现性危机。
• 克服传统同行评审的局限性，后者在高投稿压力下往往显得缓慢、存在偏见或不公平。
• 通过实现研究成果物与出版物的公开、透明、带时间戳的评估，促进共享与协作的文化。
• 通过利用社区反馈和预先验证的成果物，在正式评审前减轻评审者的负担。
• 证明开放获取与协作验证可与传统出版共存，同时增强科学严谨性与影响力。

提出的方法

• 部署公开网络门户（cTuning.org）以托管和共享研究成果物，包括基准测试、数据集、工具和预测模型。
• 将实验结果与详细元信息（特征）结合，以支持优化技术的机器学习与数据挖掘。
• 通过开放获取出版实现公开讨论与社区评审，典型例证为在 Slashdot 等平台获得高曝光度。
• 实施一种模式，即公众反馈与社区验证在正式同行评审之前或作为补充，以提升评审质量与相关性。
• 利用容器化技术（如 Docker）、溯源追踪（如 CARE、Collective Mind）以及版本化仓库，确保复杂实验设置的可重现性。
• 通过允许开放获取的预印本和扩展版本，支持与传统出版的共存，同时保留专利或商业化选项。

实验结果

研究问题

• RQ1 社区驱动的成果物评估如何提升计算机系统研究中实验结果的可重现性与可靠性？
• RQ2 哪些机制可在减轻传统评审者负担的同时，增强出版评估的公平性与透明度？
• RQ3 面向研究成果物与结果的公开、开放获取平台能否实现更快的验证、提升模型准确率并扩大科学影响力？
• RQ4 如何有意义地共享和利用负面结果或失败实验，以避免重复研究并加速科学进展？
• RQ5 为支持计算机工程领域可持续的协作与可重现研究，需要哪些技术与文化变革？

主要发现

• 社区在事实上充当了作者开放获取出版物的评审者，提供了宝贵反馈，识别出缺失的引用，并通过公开讨论改进了工作。
• 在 Slashdot 等平台的公开曝光引发了显著的社区参与，包括超过 150 条评论，展示了公众验证与快速反馈回路的潜力。
• 该方法促成了产业领袖的早期采纳与认可：IBM 在 2009 年引用该工作用于嵌入式系统优化，Fujitsu 在 2014 年引用该工作用于 Exascale 系统。
• 开放获取与共享成果物使作者得以将工作扩展为《国际并行编程杂志》（IJPP）的特刊，表明其与传统出版的兼容性。
• 该模型自然地将可重现性作为副产品，减少了对人工验证的依赖，提升了同行评审的质量与公平性。
• 作者观察到，负面结果与失败实验可被有意义地共享与讨论，从而避免重复工作并推动集体改进——尽管该实践在社区中仍使用不足。