[Paper Review] METAGENE-1: Metagenomic Foundation Model for Pandemic Monitoring

Ollie Liu, Sami Jaghouar+5 more|arXiv (Cornell University)|Jan 3, 2025

Genetics, Bioinformatics, and Biomedical ResearchBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

METAGENE-1 pretrained a 7B decoder-only transformer on 1.5 trillion base pairs of metagenomic wastewater sequences to enable pathogen detection, metagenomic embedding, and anomaly detection for pandemic monitoring. It achieves state-of-the-art results on genomic benchmarks and downstream public-health tasks.

ABSTRACT

We pretrain METAGENE-1, a 7-billion-parameter autoregressive transformer model, which we refer to as a metagenomic foundation model, on a novel corpus of diverse metagenomic DNA and RNA sequences comprising over 1.5 trillion base pairs. This dataset is sourced from a large collection of human wastewater samples, processed and sequenced using deep metagenomic (next-generation) sequencing methods. Unlike genomic models that focus on individual genomes or curated sets of specific species, the aim of METAGENE-1 is to capture the full distribution of genomic information present within this wastewater, to aid in tasks relevant to pandemic monitoring and pathogen detection. We carry out byte-pair encoding (BPE) tokenization on our dataset, tailored for metagenomic sequences, and then pretrain our model. In this paper, we first detail the pretraining dataset, tokenization strategy, and model architecture, highlighting the considerations and design choices that enable the effective modeling of metagenomic data. We then show results of pretraining this model on our metagenomic dataset, providing details about our losses, system metrics, and training stability over the course of pretraining. Finally, we demonstrate the performance of METAGENE-1, which achieves state-of-the-art results on a set of genomic benchmarks and new evaluations focused on human-pathogen detection and genomic sequence embedding, showcasing its potential for public health applications in pandemic monitoring, biosurveillance, and early detection of emerging health threats.

Motivation & Objective

• Motivate the use of a metagenomic foundation model trained on diverse wastewater sequencing data to capture broad microbiome distributions.
• Describe dataset creation, tokenization, and decoder-only transformer architecture tailored to metagenomic data.
• Evaluate METAGENE-1 on pathogen detection, genomic embeddings, and genome-wide benchmarks.
• Demonstrate downstream applications including anomaly detection in wastewater and potential public health use cases.

Proposed method

• Pretrain a 7-billion-parameter autoregressive transformer (decoder-only) on a metagenomic DNA/RNA corpus totaling over 1.5 trillion base pairs.
• Tokenize sequences with byte-pair encoding (BPE) with a 1,024-token vocabulary to produce ~370 billion tokens.
• Train using a 512-token context with packed reads and an attention mask that prevents cross-read attention.
• Use a dense transformer architecture with 32 layers, 32 heads, embedding size 4096, and RMSNorm, optimizing with Adam-like settings and a cosine learning-rate schedule.
• Perform continual pretraining by mixing in known-species genomic data at a 1:8 ratio to broaden generalization.
• Evaluate with pathogen-detection MCC benchmarks, Genomic Embedding (Gene-MTEB) tasks, and Genome Understanding Evaluation (GUE) subtasks; assess embedding quality and anomaly detection.

Figure 2 : Overview of the metagenomic data collection and sequencing pipeline for model pretraining . The process begins with the collection of wastewater (left), which contains genomic fragments from a diverse collection ( e.g. , tens of thousands) of constituent organisms (center). These samples

Experimental results

Research questions

• RQ1 Can METAGENE-1 reliably detect human pathogens across diverse sequencing deliveries?
• RQ2 How well do metagenomic representations learned from wastewater generalize to embedding and classification tasks across species?
• RQ3 What is METAGENE-1’s performance on standard genomic benchmarks compared to prior multi-species models?
• RQ4 Can METAGENE-1 support anomaly detection for wastewater-based surveillance and early threat detection?

Key findings

• METAGENE-1 consistently outperforms competing models on pathogen-detection benchmarks across four datasets, achieving MCC gains of 3–17 points.
• On Genomic Embedding tasks, METAGENE-1 achieves the highest global average and strong performance across Human-Virus and related sub-tasks.
• In Genome Understanding Evaluation, METAGENE-1 tops 13 of 28 subtasks, notably excelling in Epigenetic Marks Prediction (EMP) while noting room for improvement in promoter-related tasks.
• Gene-MTEB embedding results show robust zero-shot representations, especially for Human-Virus tasks, outperforming baseline models by more than 6 accuracy points on several metrics.
• An anomaly-detection experiment using length-normalized cross-entropy losses demonstrates clear separation between metagenomic and non-metagenomic data, indicating potential for wastewater OOD detection.

Figure 3 : Metagenomic composition of the METAGENE-1 pretraining dataset, estimated via Kraken 2 (Wood et al., 2019 ) sequence classification, and visualized via Krona (Ondov et al., 2011 ) . See Figure 7 for a more-detailed view.