[Paper Review] GenoTEX: An LLM Agent Benchmark for Automated Gene Expression Data Analysis

Haoyang Liu, Chen, Shuyu+2 more|arXiv (Cornell University)|Jun 21, 2024

Gene expression and cancer classificationBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

GenoTEX introduces a benchmark for evaluating LLM-based agents in automated gene expression data analysis, covering dataset selection, preprocessing, and statistical analysis with human-curated code and results. GenoAgents, a team of context-aware, collaborative LLM agents with iterative correction and expert consultation, achieve strong performance but reveal persistent challenges in consistency and error handling, highlighting the need for improved multi-agent coordination in genomics AI.

ABSTRACT

Recent advancements in machine learning have significantly improved the identification of disease-associated genes from gene expression datasets. However, these processes often require extensive expertise and manual effort, limiting their scalability. Large Language Model (LLM)-based agents have shown promise in automating these tasks due to their increasing problem-solving abilities. To support the evaluation and development of such methods, we introduce GenoTEX, a benchmark dataset for the automated analysis of gene expression data. GenoTEX provides analysis code and results for solving a wide range of gene-trait association problems, encompassing dataset selection, preprocessing, and statistical analysis, in a pipeline that follows computational genomics standards. The benchmark includes expert-curated annotations from bioinformaticians to ensure accuracy and reliability. To provide baselines for these tasks, we present GenoAgent, a team of LLM-based agents that adopt a multi-step programming workflow with flexible self-correction, to collaboratively analyze gene expression datasets. Our experiments demonstrate the potential of LLM-based methods in analyzing genomic data, while error analysis highlights the challenges and areas for future improvement. We propose GenoTEX as a promising resource for benchmarking and enhancing automated methods for gene expression data analysis. The benchmark is available at https://github.com/Liu-Hy/GenoTEX.

Motivation & Objective

• To address the high cost and inefficiency of manual gene expression data analysis, which consumes up to 45% of bioinformaticians' time and costs $848.3M annually.
• To develop a standardized, reproducible pipeline for gene identification tasks aligned with expert bioinformatician practices in computational genomics.
• To evaluate the effectiveness of LLM-based agents in automating complex, multi-step gene expression analysis workflows.
• To identify key failure modes and challenges in LLM agent collaboration for genomics, particularly in consistency and error handling.
• To provide a publicly available benchmark (GenoTEX) and baseline (GenoAgents) for advancing AI-driven genomics research.

Proposed method

• GenoTEX is constructed by training human bioinformaticians to follow standardized guidelines for dataset selection, preprocessing, and statistical analysis of real-world gene expression data.
• The benchmark includes input datasets, annotated code, and intermediate and final results, ensuring accuracy and reproducibility.
• GenoAgents is a team of LLM-based agents using context-aware planning, iterative correction, and domain expert consultation to simulate human bioinformatician workflows.
• Agents collaborate through a structured pipeline: dataset selection → preprocessing → statistical analysis, with feedback loops for error correction.
• The system employs prompt engineering to limit feedback to three key suggestions and encourage critical evaluation of reviewer input.
• Evaluation uses task-specific metrics for dataset selection, preprocessing, and statistical analysis, with F1 scores and code correctness as key performance indicators.

Experimental results

Research questions

• RQ1 Can LLM-based agents automate the full gene expression data analysis pipeline with accuracy comparable to human bioinformaticians?
• RQ2 How do collaborative LLM agents perform in context-aware planning, iterative correction, and expert consultation during complex genomics tasks?
• RQ3 What are the main failure modes and error patterns in LLM agent workflows, particularly in statistical analysis and code generation?
• RQ4 How does the consistency of agent feedback affect the reliability of automated analysis pipelines?
• RQ5 To what extent can prompt engineering mitigate instability and inconsistency in LLM agent collaboration for genomics?

Key findings

• GenoAgents achieved strong overall performance in automating gene expression data analysis, demonstrating the feasibility of LLM-based agents in genomics workflows.
• The statistical analysis task had the lowest F1 score at 67.08%, indicating significant challenges in model selection, parameter passing, and code execution.
• Code review agents provided inconsistent feedback across identical runs, with one instance rejecting correct code and another approving incorrect code, highlighting instability in LLM-based feedback.
• A critical error occurred when the Statistician agent passed an unexpected 'fit_intercept' argument to a model constructor, causing a runtime error despite correct logic.
• Despite prompt engineering to limit feedback and encourage critical evaluation, persistent randomness and inconsistency in agent behavior remain key challenges.
• Error analysis revealed that cumulative error risk in multi-step pipelines significantly impacts performance, underscoring the need for robust iterative correction and consensus mechanisms in future agent designs.