[Paper Review] LAB-Bench: Measuring Capabilities of Language Models for Biology Research

Jon M. Laurent, Joseph D. Janizek+7 more|arXiv (Cornell University)|Jul 14, 2024

Biomedical Text Mining and Ontologies14 citations

TL;DR

LAB-Bench is a large, multi-task benchmark (over 2,400 MCQs) to evaluate frontier language models on practical biology research tasks, including literature recall, figure/table interpretation, database access, protocol writing, and DNA/protein sequence manipulation, with human comparisons and a public subset available.

ABSTRACT

There is widespread optimism that frontier Large Language Models (LLMs) and LLM-augmented systems have the potential to rapidly accelerate scientific discovery across disciplines. Today, many benchmarks exist to measure LLM knowledge and reasoning on textbook-style science questions, but few if any benchmarks are designed to evaluate language model performance on practical tasks required for scientific research, such as literature search, protocol planning, and data analysis. As a step toward building such benchmarks, we introduce the Language Agent Biology Benchmark (LAB-Bench), a broad dataset of over 2,400 multiple choice questions for evaluating AI systems on a range of practical biology research capabilities, including recall and reasoning over literature, interpretation of figures, access and navigation of databases, and comprehension and manipulation of DNA and protein sequences. Importantly, in contrast to previous scientific benchmarks, we expect that an AI system that can achieve consistently high scores on the more difficult LAB-Bench tasks would serve as a useful assistant for researchers in areas such as literature search and molecular cloning. As an initial assessment of the emergent scientific task capabilities of frontier language models, we measure performance of several against our benchmark and report results compared to human expert biology researchers. We will continue to update and expand LAB-Bench over time, and expect it to serve as a useful tool in the development of automated research systems going forward. A public subset of LAB-Bench is available for use at the following URL: https://huggingface.co/datasets/futurehouse/lab-bench

Motivation & Objective

• Assess frontier LLMs and their ability to perform practical biology research tasks beyond textbook questions.
• Evaluate recall, reasoning, and manipulation across literature, figures, tables, databases, protocols, and sequences.
• Compare model performance to PhD-level biologists and identify gaps needing tool integration or better distractor design.
• Provide a public subset for community use and outline benchmarks for future AI-assisted biology workflows.

Proposed method

• Construct a dataset with over 2,400 multiple-choice questions spanning LitQA2, SuppQA, FigQA, TableQA, DbQA, ProtocolQA, SeqQA, and CloningScenarios.
• Combine manual expert generation for hard categories with programmatic generation for scalable tasks.
• Evaluate frontier models without tools using 0-shot chain-of-thought prompting and allow models to decline answering when information is lacking.
• Compare model performance to human biology PhDs on selected subsets and report accuracy and precision metrics.
• Provide prompts, code, and a public data subset to enable reproducible benchmarking.

Figure 1: Sample questions for each of the categories provided in this work. Note that DbQA and SeqQA consist of many different subtasks, and only one task is presented here. The font size of the distance annotations in the FigQA example have been increased for legibility here.

Experimental results

Research questions

• RQ1 How do frontier language models perform across practical biology research tasks without external tools?
• RQ2 What is the gap between model and human expert performance on LitQA2, SuppQA, FigQA, TableQA, DbQA, ProtocolQA, SeqQA, and CloningScenarios?
• RQ3 To what extent do models rely on retrieval, reasoning, or test-taking strategies across these tasks?
• RQ4 How does model performance compare to human laboratories when evaluating sequence manipulation and cloning workflows?

Key findings

• Models show wide variance across LAB-Bench tasks, with higher willingness to answer in some categories and substantial refusals in lookup-heavy tasks.
• LitQA2 questions show models performing above random with retrieval-augmented setups; without retrieval, performance can drop below chance on some frontier models.
• FigQA and DbQA remain particularly challenging, with near-random precision for most models except a few (e.g., Claude 3.5 Sonnet for TableQA).
• SeqQA performance is 40-50% precision overall, with some subtasks exceeding 90% precision for simpler primer-design tasks.
• Cloning Scenarios remain well below human performance, indicating substantial gaps in complex, real-world molecular cloning reasoning.
• Humans consistently outperform models across most tasks, though some tasks show closer gaps (e.g., TableQA with Claude 3.5 Sonnet).

(a)