[Paper Review] DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents

Peter Jansen, Marc-Alexandre Côté+6 more|arXiv (Cornell University)|Jun 10, 2024

Semantic Web and Ontologies4 citations

TL;DR

DiscoveryWorld is a text-based, simulated virtual environment designed to train and evaluate AI agents in end-to-end scientific discovery across 120 diverse, realistic tasks spanning topics like radioisotope dating, rocket science, and proteomics. The environment supports hypothesis generation, experiment design, data analysis, and conclusion drawing, with automatic evaluation metrics for task completion, procedural actions, and discovered knowledge—demonstrating that even strong baseline agents struggle, indicating the environment captures novel discovery challenges.

ABSTRACT

Automated scientific discovery promises to accelerate progress across scientific domains. However, developing and evaluating an AI agent's capacity for end-to-end scientific reasoning is challenging as running real-world experiments is often prohibitively expensive or infeasible. In this work we introduce DISCOVERYWORLD, the first virtual environment for developing and benchmarking an agent's ability to perform complete cycles of novel scientific discovery. DISCOVERYWORLD contains a variety of different challenges, covering topics as diverse as radioisotope dating, rocket science, and proteomics, to encourage development of general discovery skills rather than task-specific solutions. DISCOVERYWORLD itself is an inexpensive, simulated, text-based environment (with optional 2D visual overlay). It includes 120 different challenge tasks, spanning eight topics each with three levels of difficulty and several parametric variations. Each task requires an agent to form hypotheses, design and run experiments, analyze results, and act on conclusions. DISCOVERYWORLD further provides three automatic metrics for evaluating performance, based on (a) task completion, (b) task-relevant actions taken, and (c) the discovered explanatory knowledge. We find that strong baseline agents, that perform well in prior published environments, struggle on most DISCOVERYWORLD tasks, suggesting that DISCOVERYWORLD captures some of the novel challenges of discovery, and thus that DISCOVERYWORLD may help accelerate near-term development and assessment of scientific discovery competency in agents. Code available at: www.github.com/allenai/discoveryworld

Motivation & Objective

• To address the challenge of developing and evaluating general-purpose scientific discovery agents, which are hindered by the high cost and complexity of real-world experimentation.
• To create a scalable, low-cost simulation environment that supports full cycles of scientific reasoning—hypothesis formation, experimentation, analysis, and conclusion—without relying on physical labs.
• To encourage the development of generalizable discovery skills rather than task-specific solutions by covering eight diverse scientific domains with increasing difficulty.
• To provide a standardized benchmark with automatic evaluation metrics to measure agent performance across multiple dimensions: task completion, relevant actions, and explanatory knowledge discovery.
• To enable zero-shot evaluation of AI agents and human scientists on identical tasks, offering a direct comparison of performance and insight into the current limitations of AI in scientific reasoning.

Proposed method

• DiscoveryWorld is a text-based simulated environment with optional 2D visual overlay, enabling agents to navigate, interact with objects, use scientific instruments, and make observations.
• The environment hosts 120 challenge tasks across eight scientific topics—each with three difficulty levels and parametric variations—grounded in real scientific domains such as proteomics and nuclear decay.
• Tasks require agents to perform end-to-end discovery: form hypotheses, design and run experiments, analyze results, and derive explanatory knowledge without pre-specified solution paths.
• A three-part automatic evaluation framework measures performance based on (1) task completion, (2) number of task-relevant actions taken, and (3) quality of discovered explanatory knowledge.
• The environment is built to be realistic but simplified, allowing agents to apply scientific and commonsense knowledge, and is released under Apache-2.0 for reproducibility and community use.
• Human participants (11 Ph.D./M.Sc. scientists) were recruited to complete the same tasks under controlled conditions, with performance evaluated via manual scoring of hypotheses and evidence, and automated metrics for completion and procedural actions.

Figure 1: DiscoveryWorld is a virtual environment for developing and evaluating discovery agents, with challenge tasks covering a broad variety of different topics such as those shown above.

Experimental results

Research questions

• RQ1 Can a simulated, text-based virtual environment effectively support and evaluate end-to-end scientific discovery in AI agents across diverse scientific domains?
• RQ2 To what extent do strong baseline agents, pre-trained on prior environments, succeed in DiscoveryWorld tasks that require full discovery cycles?
• RQ3 How does human performance on DiscoveryWorld tasks compare to zero-shot AI agent performance, particularly in hypothesis generation and knowledge discovery?
• RQ4 Do the evaluation metrics—task completion, procedural actions, and discovered knowledge—provide a reliable and informative assessment of agent capabilities?
• RQ5 Can a unified, general-purpose discovery environment encourage the development of transferable discovery skills rather than task-specific optimization?

Key findings

• Strong baseline agents that perform well in prior environments struggle significantly on most DiscoveryWorld tasks, indicating that the environment captures novel and challenging aspects of scientific discovery.
• The evaluation framework successfully measures agent performance across three dimensions: task completion, relevant actions, and discovered explanatory knowledge, with the latter being particularly sensitive to true scientific insight.
• Human scientists achieved higher average scores than baseline agents on most tasks, especially in discovering explanatory knowledge, suggesting that current AI agents still lack robust hypothesis generation and reasoning capabilities.
• The inclusion of parametric variations and multiple difficulty levels across eight topics ensures that agents must generalize beyond narrow patterns, promoting development of transferable discovery skills.
• Despite the virtual nature of the environment, the tasks are grounded in realistic scientific problems, enabling meaningful application of domain knowledge and commonsense reasoning in agents.
• A small amount of data loss occurred due to save failures on the Windows platform, but the overall dataset remains robust and representative for evaluation purposes.

Figure 2: DiscoveryWorld tasks require end-to-end scientific discovery, from ideation, hypothesis formation, experiment design, data collection and analysis, forming conclusions, and acting on results. Distractors and task solutions that provide only descriptive discoveries require agents to frequen