[Paper Review] HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution

Éric Nguyen, Michael Poli+11 more|arXiv (Cornell University)|Jun 27, 2023

RNA and protein synthesis mechanisms148 citations

TL;DR

HyenaDNA pretrained on the human reference genome with up to 1 million-token context at single-nucleotide resolution, achieving state-of-the-art results across multiple genomic benchmarks with far fewer parameters and enabling in-context learning and ultralong-range tasks.

ABSTRACT

Genomic (DNA) sequences encode an enormous amount of information for gene regulation and protein synthesis. Similar to natural language models, researchers have proposed foundation models in genomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such as identifying regulatory elements. Due to the quadratic scaling of attention, previous Transformer-based genomic models have used 512 to 4k tokens as context (<0.001% of the human genome), significantly limiting the modeling of long-range interactions in DNA. In addition, these methods rely on tokenizers or fixed k-mers to aggregate meaningful DNA units, losing single nucleotide resolution where subtle genetic variations can completely alter protein function via single nucleotide polymorphisms (SNPs). Recently, Hyena, a large language model based on implicit convolutions was shown to match attention in quality while allowing longer context lengths and lower time complexity. Leveraging Hyena's new long-range capabilities, we present HyenaDNA, a genomic foundation model pretrained on the human reference genome with context lengths of up to 1 million tokens at the single nucleotide-level - an up to 500x increase over previous dense attention-based models. HyenaDNA scales sub-quadratically in sequence length (training up to 160x faster than Transformer), uses single nucleotide tokens, and has full global context at each layer. We explore what longer context enables - including the first use of in-context learning in genomics. On fine-tuned benchmarks from the Nucleotide Transformer, HyenaDNA reaches state-of-the-art (SotA) on 12 of 18 datasets using a model with orders of magnitude less parameters and pretraining data. On the GenomicBenchmarks, HyenaDNA surpasses SotA on 7 of 8 datasets on average by +10 accuracy points. Code at https://github.com/HazyResearch/hyena-dna.

Motivation & Objective

• Motivate long-context genomic modeling that preserves single-nucleotide resolution to capture SNP-level effects.
• Develop a scalable, attention-free model (HyenaDNA) able to process ultralong DNA sequences (up to 1M tokens).
• Demonstrate that longer context improves pretraining perplexity and enables novel downstream capabilities like in-context learning.
• Show that HyenaDNA reaches SotA performance on multiple genomic benchmarks with far fewer parameters and data.

Proposed method

• Use a decoder-only Hyena architecture as a drop-in replacement for attention with long convolutions and data-controlled gating.
• Train on the human reference genome with single-nucleotide tokens (A, C, G, T, N, plus special tokens) and context lengths up to 1,000,000.
• Introduce a sequence length warm-up scheduler to stabilize ultralong sequence training and reduce time.
• Employ an implicit convolution (Hyena) with FFT-based implementation to achieve O(L log L) computation per layer.
• Apply soft prompt tuning to enable downstream adaptation without updating pretrained weights (in-context learning).
• Evaluate on diverse downstream genomic tasks and compare against Transformer-based baselines and GenomicBenchmarks/Nucleotide Transformer.

Experimental results

Research questions

• RQ1 Can a convolution-based long-context model achieve single-nucleotide resolution on genomic sequences longer than previous dense-attention models?
• RQ2 What new capabilities emerge when leveraging ultralong-context genomic foundations models, including in-context learning and soft prompting?
• RQ3 How does HyenaDNA perform on standard genomic benchmarks with far fewer parameters and data than prior models?
• RQ4 To what extent does longer context improve perplexity and enable ultralong-range tasks like species classification?

Key findings

• HyenaDNA achieves up to 1,000,000 token context with sub-quadratic scaling and 160x faster training than a Transformer at 1M context.
• On GenomicBenchmarks, HyenaDNA sets SotA on 7 of 8 datasets, averaging +10 accuracy points and up to +20 on enhancer identification.
• On Nucleotide Transformer benchmarks, HyenaDNA surpasses SotA on 12 of 18 datasets using a model with orders of magnitude fewer parameters and pretraining data.
• On 919-chromatin features (DeepSEA) task, HyenaDNA is competitive with far fewer parameters than baselines.
• HyenaDNA demonstrates effective in-context learning via soft prompting and few-shot demonstrations, with performance improving as more prompt tokens are added.
• Embeddings from HyenaDNA cluster by biotype and achieve highest biotype F1 score with a much smaller model (72.0 F1 with 7M params).
• In a 5-way species classification task, HyenaDNA achieves 99.5% accuracy at 1M context, solving the task where Transformers are infeasible at such lengths.