[Paper Review] HybriDNA: A Hybrid Transformer-Mamba2 Long-Range DNA Language Model

Mingqian Ma, Guoqing Liu+13 more|ArXiv.org|Feb 15, 2025

Environmental DNA in Biodiversity Studies5 citations

TL;DR

HybriDNA introduces a decoder-only hybrid Transformer-Mamba2 DNA language model that processes ultra-long DNA sequences at single-nucleotide resolution and achieves state-of-the-art results across DNA understanding benchmarks and generative design tasks, with scaling benefits observed from 300M to 7B parameters and context lengths up to 131k tokens.

ABSTRACT

Advances in natural language processing and large language models have sparked growing interest in modeling DNA, often referred to as the "language of life". However, DNA modeling poses unique challenges. First, it requires the ability to process ultra-long DNA sequences while preserving single-nucleotide resolution, as individual nucleotides play a critical role in DNA function. Second, success in this domain requires excelling at both generative and understanding tasks: generative tasks hold potential for therapeutic and industrial applications, while understanding tasks provide crucial insights into biological mechanisms and diseases. To address these challenges, we propose HybriDNA, a decoder-only DNA language model that incorporates a hybrid Transformer-Mamba2 architecture, seamlessly integrating the strengths of attention mechanisms with selective state-space models. This hybrid design enables HybriDNA to efficiently process DNA sequences up to 131kb in length with single-nucleotide resolution. HybriDNA achieves state-of-the-art performance across 33 DNA understanding datasets curated from the BEND, GUE, and LRB benchmarks, and demonstrates exceptional capability in generating synthetic cis-regulatory elements (CREs) with desired properties. Furthermore, we show that HybriDNA adheres to expected scaling laws, with performance improving consistently as the model scales from 300M to 3B and 7B parameters. These findings underscore HybriDNA's versatility and its potential to advance DNA research and applications, paving the way for innovations in understanding and engineering the "language of life".

Motivation & Objective

• Develop a decoder-only DNA foundation model combining Transformer attention with Selective State Space Models (Mamba2) to handle ultra-long sequences at single-nucleotide resolution.
• Pretrain on a large multi-species genomic corpus to learn general genomic representations.
• Demonstrate discriminative fine-tuning for understanding tasks and generative fine-tuning for designing synthetic regulatory DNA elements.
• Investigate scaling laws and the impact of increasing context length on performance.

Proposed method

• Hybrid architecture: alternating HybriDNA Mamba2 blocks and HybriDNA Transformer blocks in a 7:1 ratio.
• HybriDNA Mamba2 blocks use State-Space Duality (SSD) with A_t simplified to a_t I for efficiency, processed in parallel with tensor parallelism.
• Pretraining on multi-species genomes with next-token prediction on base-level tokens (A, C, G, T) across 845 species totaling 160.75B nucleotides (train) and 13.25B nucleotides (valid).
• Echo embedding discriminative fine-tuning to inject future-context information for understanding tasks.
• Generative fine-tuning with task-specific prompt tokens enabling controlled sequence generation.
• Two-stage pretraining with context length warm-up from 8k to 32k to 131k tokens.

Experimental results

Research questions

• RQ1 Can HybriDNA achieve state-of-the-art performance on short-range and long-range DNA understanding benchmarks across multiple species?
• RQ2 Does the hybrid Transformer-Mamba2 architecture improve efficiency and scalability for ultra-long DNA sequences compared to pure transformer or pure SSM approaches?
• RQ3 How do model size and context length affect performance, and do scaling laws hold for HybriDNA?
• RQ4 Can HybriDNA generate realistic, desirable cis-regulatory elements across species?

Key findings

• HybriDNA achieves state-of-the-art performance across 33 DNA understanding datasets and up to 35 with echo embedding fine-tuning.
• Scaling from 300M to 3B and 7B parameters yields consistent performance gains on benchmarks.
• Long-context processing (up to 131k tokens) enhances performance on long-range tasks.
• HybriDNA demonstrates strong generative capabilities in designing synthetic CREs such as yeast promoters and cell type–specific human enhancers.
• Echo embeddings improve discriminative performance on understanding tasks.
• Model variants with echo embedding show improved metrics on several tasks.