[Paper Review] DNAGPT: A Generalized Pre-trained Tool for Versatile DNA Sequence Analysis Tasks

Daoan Zhang, Weitong Zhang+5 more|arXiv (Cornell University)|Jul 11, 2023

Genomics and Phylogenetic Studies12 citations

TL;DR

DNAGPT is a generalized pre-training model for DNA that handles both sequence and numerical data through a multi-task training regime, enabling diverse downstream tasks across species.

ABSTRACT

Pre-trained large language models demonstrate potential in extracting information from DNA sequences, yet adapting to a variety of tasks and data modalities remains a challenge. To address this, we propose DNAGPT, a generalized DNA pre-training model trained on over 200 billion base pairs from all mammals. By enhancing the classic GPT model with a binary classification task (DNA sequence order), a numerical regression task (guanine-cytosine content prediction), and a comprehensive token language, DNAGPT can handle versatile DNA analysis tasks while processing both sequence and numerical data. Our evaluation of genomic signal and region recognition, mRNA abundance regression, and artificial genomes generation tasks demonstrates DNAGPT's superior performance compared to existing models designed for specific downstream tasks, benefiting from pre-training using the newly designed model structure.

Motivation & Objective

• Motivate the need for a unified model that can handle both DNA sequences and numerical outputs.
• Design a multi-task pre-training framework that includes sequence, numerical, and binary tasks.
• Develop a token language and architecture that jointly processes sequence and numeric data.
• Evaluate DNAGPT on genomic signals and regions recognition, mRNA abundance prediction, and artificial genome generation across species.

Proposed method

• Extend a GPT-like transformer with sequence tokens and number tokens.
• Introduce three pre-training tasks: next-token prediction, GC content prediction, and sequence order prediction.
• Incorporate a hierarchical DNA token language with non-overlapped k-mers to form words, sentences, and paragraphs.
• Train on over 200 billion base pairs from mammalian reference genomes across multiple model variants (DNAGPT-H, DNAGPT-M, DNAGPT-S-512, DNAGPT-B-512).
• Fine-tune DNAGPT for downstream tasks with task-specific heads for classification and regression while preserving the base model for zero-shot potential.

Experimental results

Research questions

• RQ1 Can a single generalized pre-training model jointly handle DNA sequence and numerical data for diverse tasks?
• RQ2 How does cross-species pre-training influence performance on GSR recognition, mRNA abundance, and genome generation?
• RQ3 What are the benefits and limitations of the designed token language and multi-task pre-training for DNA analysis?

Key findings

• DNAGPT achieves superior performance on genomic signals and regions recognition across human, mouse, bovine, and fruit fly compared with state-of-the-art task-specific methods.
• DNAGPT outperforms specialized models in mRNA abundance prediction when combining DNA sequences with mRNA half-life data.
• DNAGPT generates artificial human genomes with higher alignment to real genome statistics (PCs, allele frequencies, LD) than GAN/RBM baselines.
• Attention visualizations indicate DNAGPT focuses on non-coding regions to identify genomic signals like TIS and PAS.
• Across organismal tasks, larger pre-training data improves downstream performance (H, M, S-512 variants).
• Generation temperature analysis shows controllable diversity vs. fidelity in synthetic genomes.