[Paper Review] SciCode: A Research Coding Benchmark Curated by Scientists

Minyang Tian, Lin Gao+28 more|arXiv (Cornell University)|Jul 18, 2024

Scientific Computing and Data ManagementDecision Sciences3 citations

TL;DR

SciCode is a scientist-curated benchmark of 80 complex scientific coding problems across diverse natural science fields, decomposed into 338 subproblems requiring knowledge recall, reasoning, and code synthesis. In the most realistic evaluation, even the best model, Claude3.5-Sonnet, solved only 4.6% of main problems, highlighting the benchmark's rigor and the current limitations of large language models in scientific code generation.

ABSTRACT

Since language models (LMs) now outperform average humans on many challenging tasks, it has become increasingly difficult to develop challenging, high-quality, and realistic evaluations. We address this issue by examining LMs' capabilities to generate code for solving real scientific research problems. Incorporating input from scientists and AI researchers in 16 diverse natural science sub-fields, including mathematics, physics, chemistry, biology, and materials science, we created a scientist-curated coding benchmark, SciCode. The problems in SciCode naturally factorize into multiple subproblems, each involving knowledge recall, reasoning, and code synthesis. In total, SciCode contains 338 subproblems decomposed from 80 challenging main problems. It offers optional descriptions specifying useful scientific background information and scientist-annotated gold-standard solutions and test cases for evaluation. Claude3.5-Sonnet, the best-performing model among those tested, can solve only 4.6% of the problems in the most realistic setting. We believe that SciCode demonstrates both contemporary LMs' progress towards becoming helpful scientific assistants and sheds light on the development and evaluation of scientific AI in the future.

Motivation & Objective

• To address the saturation of existing language model benchmarks by creating a high-quality, realistic evaluation suite for scientific code generation.
• To develop a benchmark that reflects real scientific research tasks, ensuring relevance to practical applications in natural sciences.
• To overcome data contamination and model overfitting by curating problems from original research and ensuring zero overlap with public datasets.
• To evaluate the comprehensive capabilities of LMs—knowledge recall, reasoning, and integration—across complex, multi-step scientific problems.
• To motivate new AI research in scientific discovery by providing a challenging, well-structured benchmark with full annotation and test cases.

Proposed method

• Curated 80 main scientific problems from 16 natural science subfields, including physics, chemistry, biology, and materials science, through collaboration with domain scientists.
• Decomposed each main problem into multiple subproblems requiring individual function implementations, enabling modular evaluation and step-by-step reasoning.
• Provided detailed scientific background, gold-standard solutions, and multiple test cases for each subproblem, annotated and verified by at least two senior researchers per domain.
• Designed evaluation settings with toggles for background information and conditioning on prior subproblem outputs to assess model robustness under varying realism levels.
• Ensured dataset integrity by excluding overlap with existing public benchmarks and simplifying complex problems only when necessary to maintain scientific accuracy.
• Implemented a leaderboard and open-source codebase at https://scicode-bench.github.io/ to support reproducible evaluation and community engagement.

Figure 1: A SciCode main problem is decomposed into multiple smaller and easier subproblems. Docstrings specify the requirements and input-output formats. When necessary, scientific background knowledge is provided, written by our scientist annotators. The full problem is shown in subsection A.3

Experimental results

Research questions

• RQ1 To what extent can state-of-the-art language models generate correct, integrated scientific code for complex, multi-step problems drawn from real research?
• RQ2 How does the inclusion of scientific background knowledge affect model performance in solving domain-specific coding tasks?
• RQ3 How do proprietary and open-source language models compare in solving scientifically grounded, reasoning-intensive code generation tasks?
• RQ4 What is the impact of model architecture and training data on the ability to generalize to novel scientific problems not seen during pretraining?
• RQ5 Can a benchmark with high scientific fidelity and minimal data leakage serve as a reliable proxy for evaluating AI's role in accelerating scientific discovery?

Key findings

• Claude3.5-Sonnet, the best-performing model tested, solved only 4.6% of the main problems in the most realistic evaluation setting, indicating significant room for improvement.
• Claude3-Opus and GPT-4o each solved only 1.5% of the main problems, demonstrating that even top-tier proprietary models struggle with complex scientific code synthesis.
• The best open-source model, Deepseek-Coder-v2, solved only 3.1% of the problems, underscoring the performance gap between closed and open models in scientific coding tasks.
• Other open-source models, including Llama-3-70B-Instruct and Mixtral-8x7b, solved less than 1% of the problems, highlighting the current limitations of open models in scientific reasoning and code generation.
• The benchmark’s design, including scientific background and test cases, significantly improves evaluation reliability and enables fine-grained analysis of model failures.
• The high difficulty of the problems—rooted in real research and requiring deep domain knowledge—confirms that SciCode is a challenging and realistic benchmark for evaluating scientific AI.

(a)