[论文解读] DNAGPT: A Generalized Pre-trained Tool for Versatile DNA Sequence Analysis Tasks

Daoan Zhang, Weitong Zhang+5 more|arXiv (Cornell University)|Jul 11, 2023

Genomics and Phylogenetic Studies被引用 12

一句话总结

DNAGPT 是一个通用的DNA预训练模型，能够通过多任务训练范式同时处理序列和数值数据，从而在跨物种的多种下游任务中具备多样性。

ABSTRACT

Pre-trained large language models demonstrate potential in extracting information from DNA sequences, yet adapting to a variety of tasks and data modalities remains a challenge. To address this, we propose DNAGPT, a generalized DNA pre-training model trained on over 200 billion base pairs from all mammals. By enhancing the classic GPT model with a binary classification task (DNA sequence order), a numerical regression task (guanine-cytosine content prediction), and a comprehensive token language, DNAGPT can handle versatile DNA analysis tasks while processing both sequence and numerical data. Our evaluation of genomic signal and region recognition, mRNA abundance regression, and artificial genomes generation tasks demonstrates DNAGPT's superior performance compared to existing models designed for specific downstream tasks, benefiting from pre-training using the newly designed model structure.

研究动机与目标

• Motivate the need for a unified model that can handle both DNA sequences and numerical outputs.
• Design a multi-task pre-training framework that includes sequence, numerical, and binary tasks.
• Develop a token language and architecture that jointly processes sequence and numeric data.
• Evaluate DNAGPT on genomic signals and regions recognition, mRNA abundance prediction, and artificial genome generation across species.

提出的方法

• Extend a GPT-like transformer with sequence tokens and number tokens.
• Introduce three pre-training tasks: next-token prediction, GC content prediction, and sequence order prediction.
• Incorporate a hierarchical DNA token language with non-overlapped k-mers to form words, sentences, and paragraphs.
• Train on over 200 billion base pairs from mammalian reference genomes across multiple model variants (DNAGPT-H, DNAGPT-M, DNAGPT-S-512, DNAGPT-B-512).
• Fine-tune DNAGPT for downstream tasks with task-specific heads for classification and regression while preserving the base model for zero-shot potential.

实验结果

研究问题

• RQ1 Can a single generalized pre-training model jointly handle DNA sequence and numerical data for diverse tasks?
• RQ2 How does cross-species pre-training influence performance on GSR recognition, mRNA abundance, and genome generation?
• RQ3 What are the benefits and limitations of the designed token language and multi-task pre-training for DNA analysis?

主要发现

• DNAGPT achieves superior performance on genomic signals and regions recognition across human, mouse, bovine, and fruit fly compared with state-of-the-art task-specific methods.
• DNAGPT outperforms specialized models in mRNA abundance prediction when combining DNA sequences with mRNA half-life data.
• DNAGPT generates artificial human genomes with higher alignment to real genome statistics (PCs, allele frequencies, LD) than GAN/RBM baselines.
• Attention visualizations indicate DNAGPT focuses on non-coding regions to identify genomic signals like TIS and PAS.
• Across organismal tasks, larger pre-training data improves downstream performance (H, M, S-512 variants).
• Generation temperature analysis shows controllable diversity vs. fidelity in synthetic genomes.]
• table_headers: []
• table_rows: []} }# 清单: 这里只返回结构化字段的翻译文本，表头与表行保持为空数组的原样。