[Paper Review] GENERator: A Long-Context Generative Genomic Foundation Model

Wei Wu, Qiuyi Li+16 more|ArXiv.org|Feb 11, 2025

Gene expression and cancer classification6 citations

TL;DR

GENERator is a 1.2B-parameter generative genomic foundation model with a 98k base-pair context, trained on 386B bp of eukaryotic DNA, achieving state-of-the-art on genomic benchmarks and enabling central dogma-aligned protein coding and promoter design.

ABSTRACT

The rapid advancement of DNA sequencing has produced vast genomic datasets, yet interpreting and engineering genomic function remain fundamental challenges. Recent large language models have opened new avenues for genomic analysis, but existing approaches are often limited by restricted training scope, constrained generative capability, or prohibitive computational cost. We introduce GENErator, a generative genomic foundation model for long-context DNA modeling, with a context length of 98k nucleotides, pre-trained on 386 billion nucleotides of eukaryotic DNA. Without task-specific fine-tuning, GENERator exhibits strong intrinsic capabilities: unsupervised embedding analyses reveal phylogenetically coherent structure, and sequence recovery benchmarks demonstrate generative accuracy comparable to or exceeding state-of-the-art models with substantially improved computational efficiency. In a zero-shot setting, GENERator achieves competitive variant effect prediction performance relative to alignment-based methods, while remaining fully alignment-free and broadly applicable across species. With task-specific fine-tuning, the model attains leading performance on established genomic benchmarks. We further demonstrate practical generative applications. GENERator can generate protein-coding DNA sequences that translate into structurally plausible proteins and, through a prompt-guided design framework, design cis-regulatory elements with targeted activity profiles, including synthetic super-enhancers validated by high-throughput UMI-STARR-seq assays. Together, these results establish GENERator as an efficient and biologically grounded framework for genomic interpretation and programmable sequence design. Code and supplementary resources are available at https://github.com/GenerTeam/GENERator.

Motivation & Objective

• Advance genomic sequence modeling with a long-context generative foundation model tailored for DNA data.
• Demonstrate state-of-the-art performance on established and newly proposed genomic benchmarks.
• Show alignment with the central dogma by generating protein-coding sequences that translate to known protein families.
• Explore sequence design capabilities, including prompt-responsive promoter design with activity targeting.
• Investigate training strategies and tokenizer choices that maximize long-range genomic understanding.

Proposed method

• Use a transformer decoder architecture inspired by Llama with 26 layers and 2,048 hidden size.
• Pre-train on 386B nucleotides of eukaryotic DNA from RefSeq using a 6-mer tokenizer for next-token prediction (NTP).
• Compare gene-sequence training versus whole-sequence training and identify semantically rich regions as more effective for downstream tasks.
• Employ techniques to handle long-context data efficiently (Flash Attention, Zero Redundancy Optimizer) and introduce randomized tokenization starting points to improve robustness.
• Evaluate on Genomic Benchmarks, NT tasks, and new Gener tasks including gene/taxonomic classification and next-K-mer prediction; analyze central dogma and promoter design tasks.
• Provide detailed architectural specs (e.g., 26 layers, hidden size 2048, vocab 4128, context length 16384 tokens corresponding to 98,304 bp) and training setup (batch size 2M tokens, 6 epochs, AdamW, cosine warmup).

Figure 1: Overview of the Gener ator . (A) The pre-training dataset of the Gener ator encompasses a diverse range of eukaryotic organisms and gene types, totaling 386B nucleotides. (B) The pre-training employs the next token prediction (NTP) task, utilizing a 6-mer tokenizer. (C) Model comparison re

Experimental results

Research questions

• RQ1 Can GENERator achieve state-of-the-art performance across multiple genomic benchmarks and tasks?
• RQ2 How does the choice of tokenizer (6-mer) impact next-token prediction in a causal DNA language model compared to BPE or single-nucleotide tokenizers?
• RQ3 Does training on gene regions (semantically rich data) outperform whole-genome training for downstream genomic tasks?
• RQ4 Can the model generate protein-coding DNA sequences that translate to proteins within the target families (central dogma alignment)?
• RQ5 To what extent can GENERator assist in sequence design, such as promoter design with targeted activity profiles?

Key findings

• Achieves state-of-the-art performance across Genomic Benchmarks, NT tasks, and newly proposed Gener tasks.
• Uses a 98k bp context with 1.2B parameters and outperforms baselines like NT-multi, Enformer, GROVER, HyenaDNA, and Caduceus on key tasks.
• Gene-sequence training (focusing on semantically rich regions) outperforms whole-sequence training for downstream tasks across multiple taxonomic groups.
• Demonstrates central dogma alignment by generating protein-coding DNA sequences that translate into proteins structurally analogous to known families, and evaluates their foldability (AlphaFold) and distributional perplexity (Progen2).
• Shows promoter-design capability with prompt-responsive activity targeting using the DeepSTARR promoter dataset, enabling controlled sequence optimization.

Figure 2: Evaluation of next K-mer prediction. (A) Accuracy of the next K-mer prediction task across various tokenizers and input token lengths. (B) Comparison of the Gener ator against baseline models on a dataset comprised exclusively mammalian DNA.