[Paper Review] CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark

Zachary S. Siegel, Sayash Kapoor+3 more|arXiv (Cornell University)|Sep 17, 2024

Scientific Computing and Data ManagementDecision Sciences3 citations

TL;DR

CORE-Bench introduces a benchmark of 270 computational reproducibility tasks derived from 90 scientific papers across computer science, social science, and medicine, evaluating AI agents' ability to reproduce research results using provided code and data. The best-performing agent achieved only 21% accuracy on the hardest task level, highlighting significant room for improvement in automating scientific reproducibility.

ABSTRACT

AI agents have the potential to aid users on a variety of consequential tasks, including conducting scientific research. To spur the development of useful agents, we need benchmarks that are challenging, but more crucially, directly correspond to real-world tasks of interest. This paper introduces such a benchmark, designed to measure the accuracy of AI agents in tackling a crucial yet surprisingly challenging aspect of scientific research: computational reproducibility. This task, fundamental to the scientific process, involves reproducing the results of a study using the provided code and data. We introduce CORE-Bench (Computational Reproducibility Agent Benchmark), a benchmark consisting of 270 tasks based on 90 scientific papers across three disciplines (computer science, social science, and medicine). Tasks in CORE-Bench consist of three difficulty levels and include both language-only and vision-language tasks. We provide an evaluation system to measure the accuracy of agents in a fast and parallelizable way, saving days of evaluation time for each run compared to a sequential implementation. We evaluated two baseline agents: the general-purpose AutoGPT and a task-specific agent called CORE-Agent. We tested both variants using two underlying language models: GPT-4o and GPT-4o-mini. The best agent achieved an accuracy of 21% on the hardest task, showing the vast scope for improvement in automating routine scientific tasks. Having agents that can reproduce existing work is a necessary step towards building agents that can conduct novel research and could verify and improve the performance of other research agents. We hope that CORE-Bench can improve the state of reproducibility and spur the development of future research agents.

Motivation & Objective

• To address the critical but underexplored challenge of computational reproducibility in scientific research, where published results often cannot be reproduced due to missing dependencies, version mismatches, or environment issues.
• To develop a benchmark that reflects real-world scientific workflows, requiring agents to install dependencies, run code, and extract answers from outputs—mirroring actual research practice.
• To enable fast, parallelizable evaluation of AI agents on reproducibility tasks, reducing evaluation time from days to hours compared to sequential execution.
• To assess the performance of both general-purpose and task-specific agents in reproducing results from real scientific repositories, using GPT-4o and GPT-4o-mini as underlying models.
• To foster the development of AI agents capable of verifying and improving scientific research by first mastering the foundational task of reproducibility.

Proposed method

• CORE-Bench constructs 270 tasks from 90 real scientific papers across three disciplines, with three difficulty levels: low (language-only), medium (vision-language), and high (complex code execution and reasoning).
• Each task requires the agent to access a CodeOcean capsule containing the paper’s code, data, Dockerfile, and README, then install dependencies and execute the code in a sandboxed environment.
• The evaluation system automates the execution and comparison of agent outputs against ground-truth results, validating correctness through a fast, parallelizable pipeline.
• Agents are evaluated on their ability to correctly answer all task-specific questions after successful code execution, with success defined as accurate response to all queries.
• Baseline agents include AutoGPT (general-purpose) and a task-specific agent called CORE-Agent, both tested with GPT-4o and GPT-4o-mini to assess performance differences.
• The benchmark includes error recovery and web search capabilities to simulate real-world debugging, such as resolving missing modules or locating external libraries.

Figure 1 : Overview of CORE-Bench . Each task in CORE-Bench requires an agent to reproduce the results of a research paper given its repository. The agent must install libraries, packages, and dependencies and run the code. If the code runs successfully, the agent needs to search through all outputs

Experimental results

Research questions

• RQ1 To what extent can current AI agents reproduce the results of published scientific studies using only the provided code and data?
• RQ2 How does the performance of general-purpose agents like AutoGPT compare to task-specific agents designed for computational reproducibility?
• RQ3 What are the primary failure modes in AI agent execution when reproducing scientific code, such as dependency resolution, environment setup, or code interpretation?
• RQ4 How does the difficulty level of tasks (low, medium, high) affect agent accuracy, and what bottlenecks emerge at higher levels?
• RQ5 Can a scalable, parallelizable evaluation framework significantly reduce the time required to benchmark AI agents on reproducibility tasks compared to sequential execution?

Key findings

• The best-performing agent, using the CORE-Agent with GPT-4o, achieved 21% accuracy on the hardest level of tasks, indicating substantial room for improvement in automating scientific reproducibility.
• General-purpose agents like AutoGPT struggled significantly, especially on high-difficulty tasks, failing to resolve dependency issues or correctly interpret complex codebases.
• A major failure mode was the inability to resolve missing Python modules (e.g., 'network_diffusion'), even after multiple web searches, highlighting limitations in agent reasoning and tool use.
• Many agents failed to properly interpret or execute code due to incorrect dependency installation, environment misconfiguration, or misreading of configuration files like 'config.ini'.
• The evaluation system reduced agent evaluation time from days to hours by enabling parallel execution across all tasks, demonstrating scalability for future benchmarking.
• Even when agents successfully ran the code, they often failed to extract or interpret the correct results from outputs, indicating a gap in downstream reasoning and answer extraction.

Figure 2 : Files and folders in each CodeOcean capsule. Each capsule contains a Readme, Dockerfile, and instructions on how to use Docker, which we selectively provide to the agent depending on the difficulty of the task.