[Paper Review] BioAgent Bench: An AI Agent Evaluation Suite for Bioinformatics

Dionizije Fa, Marko Čuljak+2 more|arXiv (Cornell University)|Jan 29, 2026

Cancer Genomics and Diagnostics0 citations

TL;DR

BioAgent Bench provides a benchmark dataset and evaluation suite to measure AI agents’ ability to perform end-to-end bioinformatics pipelines, assess robustness with perturbations, and compare open-weight versus closed-weight models across multiple harnesses.

ABSTRACT

This paper introduces BioAgent Bench, a benchmark dataset and an evaluation suite designed for measuring the performance and robustness of AI agents in common bioinformatics tasks. The benchmark contains curated end-to-end tasks (e.g., RNA-seq, variant calling, metagenomics) with prompts that specify concrete output artifacts to support automated assessment, including stress testing under controlled perturbations. We evaluate frontier closed-source and open-weight models across multiple agent harnesses, and use an LLM-based grader to score pipeline progress and outcome validity. We find that frontier agents can complete multi-step bioinformatics pipelines without elaborate custom scaffolding, often producing the requested final artifacts reliably. However, robustness tests reveal failure modes under controlled perturbations (corrupted inputs, decoy files, and prompt bloat), indicating that correct high-level pipeline construction does not guarantee reliable step-level reasoning. Finally, because bioinformatics workflows may involve sensitive patient data, proprietary references, or unpublished IP, closed-source models can be unsuitable under strict privacy constraints; in such settings, open-weight models may be preferable despite lower completion rates. We release the dataset and evaluation suite publicly.

Motivation & Objective

• Provide a benchmark dataset of end-to-end bioinformatics tasks suited for AI agents.
• Compare frontier closed-source and open-weight models in agent-enabled workflows.
• Evaluate robustness of agent pipelines under controlled perturbations and data corruption.
• Offer an evaluation harness that records transcripts, assesses progress, and scores outcomes.
• Promote privacy-conscious deployment by highlighting scenarios favoring open-weight models.

Proposed method

• Define end-to-end bioinformatics tasks spanning RNA-seq, variant calling, metagenomics, etc.
• Use task prompts and required input/reference data to form evaluation units with concrete output formats (e.g., CSV).
• Evaluate agents via harnesses (Claude Code, Codex CLI, OpenCode) and an LLM grader for step completion and final artifacts.
• Incorporate perturbation tests (corrupted inputs, decoys, prompt bloat) to assess robustness.
• Measure completion rate as the primary metric, with analysis of planning quality and failure modes.
• Report results with task- and model-level heatmaps and robustness statistics.

Figure 1: An overview of BioAgent Bench. Inputs to LLM agents consist of a task prompt, input data, and reference data. While solving the provided task, an agent can use general-purpose packages or specialized bioinformatics tools. After the agent finishes generation, LLM judge compares its outputs

Experimental results

Research questions

• RQ1 Can frontier closed-source models complete multi-step bioinformatics pipelines end-to-end with minimal scaffolding?
• RQ2 How do open-weight models compare to closed-source models in completion rates and robustness across bioinformatics tasks?
• RQ3 What is the relationship between planning quality and pipeline completion in agent-based bioinformatics workflows?
• RQ4 What failure modes arise under input corruption, decoys, or prompt bloat in bioinformatics agent pipelines?
• RQ5 How does robustness to perturbations vary across tasks and harnesses?

Key findings

• Frontier models achieve high pipeline completion rates, with Claude Opus 4.5 at 100% and Gemini 3 Pro, GPT-5.2, Sonnet 4.5 above 90%.
• Open-weight models trail on average, best being GLM-4.7 at 82.5% completion in Codex CLI; others around 65%.
• Planning quality correlates with completion rate (Pearson r = 0.61) but is not deterministically predictive of success across all models.
• Robustness tests reveal brittle step-level reasoning, e.g., sensitivity to corrupted inputs, decoys, and prompt bloat, with prompt bloat reducing completion across tasks by 28% on average.
• Closed-source models can be prone to error-correction loops, while frontier models more often recover and complete pipelines.
• Open-weight models may be preferable in privacy-constrained settings despite lower completion rates.

Figure 2: Model-task completion heatmap. The left panel shows a pairwise completion matrix: rows and columns correspond to models and tasks, respectively, and each cell reports the completion rate (in %) for each model and task pair. Cell color encodes the completion rate, with numeric annotations s