[Paper Review] Complications for Computational Experiments from Modern Processors

Ian P. Gent, Hecher, Markus+2 more|arXiv (Cornell University)|Apr 12, 2013

Scientific Computing and Data Management15 references35 citations

TL;DR

This paper proposes a 'recomputation manifesto' to ensure long-term reproducibility of computational experiments in computer science by mandating the preservation of exact virtual machine environments. It argues that only by providing complete, emulatable virtual machines—regardless of source code availability—can experiments be reliably recomputed decades later, with the goal of making all experiments recomputable for all time.

ABSTRACT

In this paper, we revisit the approach to empirical experiments for combinatorial solvers. We provide a brief survey on tools that can help to make empirical work easier. We illustrate origins of uncertainty in modern hardware and show how strong the influence of certain aspects of modern hardware and its experimental setup can be in an actual experimental evaluation. More specifically, there can be situations where (i) two different researchers run a reasonable-looking experiment comparing the same solvers and come to different conclusions and (ii) one researcher runs the same experiment twice on the same hardware and reaches different conclusions based upon how the hardware is configured and used. We investigate these situations from a hardware perspective. Furthermore, we provide an overview on standard measures, detailed explanations on effects, potential errors, and biased suggestions for useful tools. Alongside the tools, we discuss their feasibility as experiments often run on clusters to which the experimentalist has only limited access. Our work sheds light on a number of benchmarking-related issues which could be considered to be folklore or even myths.

Motivation & Objective

• To address the widespread lack of reproducibility in computational experiments within computer science.
• To establish recomputability as a standard practice, ensuring experiments can be exactly replicated over time.
• To overcome the limitations of source code sharing alone, which fails to guarantee consistent execution environments.
• To promote the use of virtual machines as the only reliable method for ensuring long-term recomputability.
• To create a sustainable infrastructure—tools and repositories—for hosting and accessing recomputable experiments.

Proposed method

• Propose that computational experiments must be hosted with complete virtual machines (VMs) that exactly replicate the original execution environment.
• Advocate for the use of virtualization technology to encapsulate all system dependencies, including OS, libraries, and runtime state.
• Mandate that repositories store and preserve VM images to ensure that experiments can be rerun identically in the future.
• Encourage the development of tools and platforms (e.g., recomputation.org) to simplify the creation, upload, and retrieval of recomputable experiments.
• Promote open access to VMs and associated tools under open-source licenses to ensure long-term availability and reuse.
• Position virtual machines as the gold standard for recomputability, even for closed-source experiments, by preserving the exact execution environment.

Experimental results

Research questions

• RQ1 What is the most reliable method to ensure that computational experiments can be exactly recomputed decades after their original execution?
• RQ2 Why is source code availability alone insufficient for meaningful reproducibility of computational experiments?
• RQ3 How can the scientific community standardize and institutionalize the practice of recomputing experiments over time?
• RQ4 What infrastructure and tools are needed to make recomputability as easy and accessible as version control for software?
• RQ5 What are the long-term technical and economic feasibility considerations for preserving computational experiments in virtual machine form?

Key findings

• The only way to guarantee recomputability of computational experiments is to preserve the exact virtual machine environment in which they were originally validated.
• Even with source code available, experiments often fail to rebuild or run due to changes in system libraries, compilers, or operating systems over time.
• Virtual machines eliminate environmental variability and ensure deterministic, repeatable execution across different hardware and time periods.
• The cost of storing data and VMs is decreasing exponentially, making long-term preservation of experiments economically feasible.
• Tools like Sumatra, CDE, and Burrito show early promise, but a dedicated ecosystem for recomputable experiments is still lacking.
• Recomputing all experiments from the past five years annually would require only 10% of new CPU resources, demonstrating the practicality of large-scale recomputation.