[Paper Review] "Share and Enjoy": Publishing Useful and Usable Scientific Models

Tom Crick, Benjamin A. Hall+2 more|arXiv (Cornell University)|Sep 1, 2014

Scientific Computing and Data Management11 references4 citations

TL;DR

This paper proposes an integrated cloud-based infrastructure to automate the sharing, testing, and validation of scientific models and algorithms, using continuous integration (CI) systems to ensure reproducibility. By treating benchmarks as version-controlled artifacts and triggering automated execution upon code commits, the system enables real-time, authoritative performance comparisons, reducing overhead and accelerating scientific discovery in computational science.

ABSTRACT

The reproduction and replication of reported scientific results is a hot topic within the academic community. The retraction of numerous studies from a wide range of disciplines, from climate science to bioscience, has drawn the focus of many commentators, but there exists a wider socio-cultural problem that pervades the scientific community. Sharing code, data and models often requires extra effort; this is currently seen as a significant overhead that may not be worth the time investment. Automated systems, which allow easy reproduction of results, offer the potential to incentivise a culture change and drive the adoption of new techniques to improve the efficiency of scientific exploration. In this paper, we discuss the value of improved access and sharing of the two key types of results arising from work done in the computational sciences: models and algorithms. We propose the development of an integrated cloud-based system underpinning computational science, linking together software and data repositories, toolchains, workflows and outputs, providing a seamless automated infrastructure for the verification and validation of scientific models and in particular, performance benchmarks.

Motivation & Objective

• To address the widespread reproducibility crisis in computational science by improving access to and sharing of scientific models and algorithms.
• To reduce the cultural and technical barriers that hinder code, data, and model reuse in scientific research.
• To automate the verification and validation of scientific models and algorithms through integrated cloud-based continuous integration (CI) systems.
• To create a sustainable, scalable infrastructure where models and algorithms evolve together with verifiable, publicly accessible results.
• To reposition scientific models and benchmarks as first-class outputs of research, linked directly to publications and implementations.

Proposed method

• Leverage cloud platforms (e.g., AWS, Google Cloud, Azure) to host and execute scientific workloads on demand.
• Use version-controlled repositories (e.g., Git) to manage models, algorithms, and benchmark suites as first-class research artifacts.
• Integrate continuous integration (CI) systems (e.g., Jenkins, Visual Studio Online) to automatically trigger execution upon code or model updates.
• Define standardized performance metrics beyond wall-clock time, including algorithmic operations count and structural properties like termination and stability.
• Publish results from all algorithm runs on all benchmarks as authoritative, time-stamped outputs, analogous to kernel archives.
• Enable automated testing and validation of new algorithms against a shared, evolving set of benchmarks, ensuring reproducibility from the outset.

Experimental results

Research questions

• RQ1 How can scientific models and algorithms be shared and validated in a way that reduces the perceived overhead of reproducibility?
• RQ2 What infrastructure is needed to enable automated, continuous verification of scientific models and algorithms?
• RQ3 How can performance metrics be standardized and made meaningful across diverse computational science domains?
• RQ4 What role can cloud-based continuous integration play in improving the reliability and reusability of scientific software?
• RQ5 How can the scientific community be incentivized to adopt a model where benchmarks and algorithms are co-developed and versioned in a shared, automated system?

Key findings

• An integrated cloud-based continuous integration system can automate the execution and validation of scientific algorithms on shared benchmarks, significantly reducing manual effort and error.
• Benchmarking results are more reliable and comparable when performance metrics such as operation counts and structural properties (e.g., termination, stability) are prioritized over wall-clock time.
• The system enables real-time, authoritative publication of algorithm results on benchmarks, creating a trusted, versioned source of truth for scientific comparisons.
• By linking papers, models, and implementations through automated CI pipelines, the system enhances external validation and supports the reuse of scientific outputs.
• The proposed infrastructure reduces the perceived overhead of reproducibility, positioning it as a productivity-enhancing tool rather than a burden.
• The model demonstrates that a 'fourth paradigm' of data-intensive scientific discovery is feasible when models and algorithms are developed and validated in a shared, automated environment.