[Paper Review] BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models

Xiangru Tang, Bill Qian+4 more|arXiv (Cornell University)|Aug 31, 2023

Genetics, Bioinformatics, and Biomedical Research4 citations

TL;DR

BioCoder introduces a comprehensive benchmark for evaluating large language models (LLMs) in generating bioinformatics-specific code, using 1,026 Python functions and 1,243 Java methods from GitHub and 253 Rosalind Project examples. The benchmark reveals that models requiring long context (≥2600 tokens) and possessing domain-specific bioinformatics knowledge outperform general models, with GPT-4 achieving up to 25% higher pass@K scores than smaller models.

ABSTRACT

Pre-trained large language models (LLMs) have significantly improved code generation. As these models scale up, there is an increasing need for the output to handle more intricate tasks and to be appropriately specialized to particular domains. Here, we target bioinformatics due to the amount of domain knowledge, algorithms, and data operations this discipline requires. We present BioCoder, a benchmark developed to evaluate LLMs in generating bioinformatics-specific code. BioCoder spans much of the field, covering cross-file dependencies, class declarations, and global variables. It incorporates 1,026 Python functions and 1,243 Java methods extracted from GitHub, along with 253 examples from the Rosalind Project, all pertaining to bioinformatics. Using topic modeling, we show that the overall coverage of the included code is representative of the full spectrum of bioinformatics calculations. BioCoder incorporates a fuzz-testing framework for evaluation. We have applied it to evaluate various models including InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder, StarCoder+, InstructCodeT5+, GPT-3.5, and GPT- 4. Furthermore, we fine-tuned one model (StarCoder), demonstrating that our training dataset can enhance the performance on our testing benchmark (by >15% in terms of Pass@K under certain prompt configurations and always >3%). The results highlight two key aspects of successful models: (1) Successful models accommodate a long prompt (> 2,600 tokens) with full context, including functional dependencies. (2) They contain domain-specific knowledge of bioinformatics, beyond just general coding capability. This is evident from the performance gain of GPT-3.5/4 compared to the smaller models on our benchmark (50% vs. up to 25%). Availability and implementation: Code is available at: https://github.com/gersteinlab/biocoder and https://biocoder-benchmark. github.io/.

Motivation & Objective

• Address the lack of comprehensive, domain-specific benchmarks for evaluating LLMs in bioinformatics code generation.
• Create a high-quality, representative dataset of real-world bioinformatics code from GitHub repositories and the Rosalind Project.
• Evaluate the impact of contextual length and domain-specific knowledge on LLM performance in bioinformatics tasks.
• Develop a fuzzer testing framework to ensure reliable and scalable evaluation of code generation models.
• Demonstrate that fine-tuning with BioCoder improves LLM performance, especially in handling complex, multi-file dependencies and domain-specific logic.

Proposed method

• Curated 1,720 bioinformatics repositories from peer-reviewed articles to extract 1,026 Python functions and 1,243 Java methods with full context, including class declarations and global variables.
• Integrated 253 bioinformatics problems from the Rosalind Project, focusing on core topics like DNA/RNA analysis and sequence processing.
• Designed a parsing tool to extract function specifications, comments, and dependencies from large codebases, preserving cross-file and cross-package relationships.
• Built a fuzzer testing framework to evaluate code correctness through automated, scalable execution of generated functions.
• Applied Pass@K evaluation across 1,000 iterations to measure functional correctness and robustness of generated code.
• Conducted ablation studies to distinguish between memorization and true code generation capability, ensuring model generalization.

Figure 1: Overview of our contribution in BioCoder .

Experimental results

Research questions

• RQ1 To what extent do LLMs with long context windows (≥2600 tokens) outperform those with shorter context in generating correct bioinformatics code?
• RQ2 How does domain-specific knowledge in bioinformatics influence LLM performance compared to general coding knowledge?
• RQ3 Can fine-tuning on the BioCoder dataset significantly improve LLMs' code generation accuracy on bioinformatics tasks?
• RQ4 What role do prompt formatting elements—such as comment symbols (#)—play in improving code generation reliability?
• RQ5 How representative is the BioCoder benchmark of the full spectrum of bioinformatics code generation tasks?

Key findings

• Models with long context (≥2600 tokens) significantly outperform those with shorter context, especially when handling cross-file dependencies and complex function calls.
• GPT-4 achieved up to 25% higher Pass@K scores than smaller models like InCoder and CodeGen, indicating a strong performance gap due to domain-specific knowledge.
• Fine-tuning StarCoder on BioCoder improved Pass@K by over 15% in certain prompt configurations and by at least 3% in all configurations.
• The inclusion of comment symbols (#) in prompts led to cleaner, runnable code outputs, reducing extraneous text and runtime errors.
• The benchmark demonstrated high coverage of bioinformatics topics through topic modeling, confirming its representativeness across core bioinformatics workflows.
• Ablation studies confirmed that models were not simply memorizing solutions, but actively generating correct code, validating the benchmark’s reliability for evaluation.

Figure 2: A diagram of the BioCoder construction process involving custom GitHub repository cleaning, parsing, and function selection, as well as context and test case creation and a massively dockerized testing framework.