[Paper Review] BEND: Benchmarking DNA Language Models on biologically meaningful tasks

Frederikke Isa Marin, Felix Teufel+5 more|arXiv (Cornell University)|Nov 21, 2023

RNA and protein synthesis mechanisms51 references27 citations

TL;DR

BEND proposes a standardized benchmark for DNA language models across seven biologically meaningful tasks on the human genome, evaluating multiple LMs and baselines to assess long-range and sparse-signal reasoning in genomic data.

ABSTRACT

The genome sequence contains the blueprint for governing cellular processes. While the availability of genomes has vastly increased over the last decades, experimental annotation of the various functional, non-coding and regulatory elements encoded in the DNA sequence remains both expensive and challenging. This has sparked interest in unsupervised language modeling of genomic DNA, a paradigm that has seen great success for protein sequence data. Although various DNA language models have been proposed, evaluation tasks often differ between individual works, and might not fully recapitulate the fundamental challenges of genome annotation, including the length, scale and sparsity of the data. In this study, we introduce BEND, a Benchmark for DNA language models, featuring a collection of realistic and biologically meaningful downstream tasks defined on the human genome. We find that embeddings from current DNA LMs can approach performance of expert methods on some tasks, but only capture limited information about long-range features. BEND is available at https://github.com/frederikkemarin/BEND.

Motivation & Objective

• Motivate a standardized, biologically grounded evaluation suite for DNA language models (LMs).
• Assess how well current DNA LMs capture long-range contextual information and sparse regulatory signals.
• Evaluate a broad set of LM architectures and tokenizations on realistic genomic tasks of varying length scales.
• Provide insights into what genomic features LMs learn during pre-training and how embeddings aid downstream tasks.

Proposed method

• Curate seven biologically meaningful downstream tasks defined on the human genome with varying length scales.
• Benchmark six public self-supervised DNA LMs plus simple baselines (AWD-LSTM, dilated CNN) on all tasks.
• Attach a lightweight downstream CNN on top of frozen LM embeddings to perform task-specific predictions; for variant effects, use zero-shot cosine similarity in embedding space.
• Provide data splits by whole-chromosome holdout (except gene finding, where sequence-identity splits are recommended).
• Offer an adaptable benchmarking framework to prepare embeddings and train lightweight classifiers, and compare to task-specific supervised baselines.

Figure 1: The organization of eukaryotic genomic DNA. The numbers are indicative examples for the human genome. Genes are structured as alternating introns (average: 5,400 bp) and exons (average: 170 bp), and have a promoter regulatory element before their transcription start site. Enhancer regulato

Experimental results

Research questions

• RQ1 Can current DNA LMs capture long-range genomic context necessary for downstream annotation tasks?
• RQ2 How do different pre-training objectives and tokenization strategies affect what genomic features LMs learn?
• RQ3 Do LM embeddings provide consistent improvements over task-specific baselines across a spectrum of genomic tasks?
• RQ4 To what extent can LM-based representations support tasks with sparse signals and long-range dependencies (e.g., enhancer annotation)?

Key findings

• DNA LMs show promising performance on some tasks and can approach expert methods, but no LM consistently outperforms all baselines across tasks.
• Long-range reasoning remains challenging, particularly for sparse and distant regulatory elements like enhancers.
• Embeddings from different LMs learn distinct genomic features; some capture gene structure information, others focus on noncoding regions.
• NT-MS emerges as a strong default LM, but shorter, smaller models (e.g., DNABERT) can outperform it on certain tasks.
• LM embeddings often underperform task-specific supervised methods in modeling functional genomics data when used with a simple downstream CNN.
• Gene finding benefits from LM embeddings in combination with a simple CNN, approaching but not surpassing specialized AUGUSTUS performance; enhancer annotation remains difficult.

Figure A1: Length distribution of samples in the gene finding dataset.