[论文解读] Deciphering the Language of Nature: A transformer-based language model for deleterious mutations in proteins

Theodore Jiang, Fang Li+1 more|arXiv (Cornell University)|Oct 27, 2021

Machine Learning in Bioinformatics被引用 4

一句话总结

MutFormer 是一种基于 Transformer 的深度学习模型，通过利用参考序列和突变蛋白序列，结合自注意力机制与卷积层，捕捉蛋白质序列中的长程和短程依赖关系，从而预测有害的错义突变。该模型在多个基准数据集上实现了最先进性能，优于或匹配现有工具，其优势在于捕捉了先前方法未探索的生物相关序列特征。

ABSTRACT

Various machine-learning models, including deep neural network models, have already been developed to predict deleteriousness of missense (non-synonymous) mutations. Potential improvements to the current state of the art, however, may still benefit from a fresh look at the biological problem using more sophisticated self-adaptive machine-learning approaches. Recent advances in the natural language processing field show transformer models-a type of deep neural network-to be particularly powerful at modeling sequence information with context dependence. In this study, we introduce MutFormer, a transformer-based model for the prediction of deleterious missense mutations, which uses reference and mutated protein sequences from the human genome as the primary features. MutFormer takes advantage of a combination of self-attention layers and convolutional layers to learn both long-range and short-range dependencies between amino acid mutations in a protein sequence. In this study, we first pre-trained MutFormer on reference protein sequences and mutated protein sequences resulting from common genetic variants observed in human populations. We next examined different fine-tuning methods to successfully apply the model to deleteriousness prediction of missense mutations. Finally, we evaluated MutFormer's performance on multiple testing data sets. We found that MutFormer showed similar or improved performance over a variety of existing tools, including those that used conventional machine-learning approaches. We conclude that MutFormer successfully considers sequence features that are not explored in previous studies and could potentially complement existing computational predictions or empirically generated functional scores to improve our understanding of disease variants.

研究动机与目标

• 开发一种新型深度学习模型，利用先进的序列建模技术，提升有害错义突变的预测性能。
• 利用 Transformer 架构的上下文感知能力，捕捉蛋白质序列中的复杂依赖关系。
• 在人类群体遗传变异上进行预训练，并通过微调实现有害性预测的高精度。
• 在多样且独立的测试数据集上，与现有成熟工具进行性能对比评估。
• 识别先前模型未捕捉到的序列特征，提升致病性预测的生物学可解释性。

提出的方法

• MutFormer 采用混合架构，结合多头自注意力层与卷积层，以建模蛋白质序列中的长程与局部序列依赖关系。
• 模型在源自常见人类遗传变异的参考序列和突变蛋白序列上进行预训练，以学习通用的序列表征。
• 通过在已知有害与中性错义突变的标注数据集上应用监督学习，进行微调。
• 输入特征包括野生型和突变型氨基酸序列，使模型能够学习特定突变的影响。
• 模型使用位置编码以保留序列顺序和上下文信息，这对捕捉突变对蛋白质结构与功能的影响至关重要。
• 通过 AUC-ROC 和 AUPRC 等标准指标，在多个独立测试集上评估模型性能。

实验结果

研究问题

• RQ1 基于 Transformer 的模型是否能在预测有害错义突变方面超越现有机器学习工具？
• RQ2 MutFormer 学习到的序列层面特征，特别是依赖上下文的模式，是否为传统模型所未捕捉？
• RQ3 在群体水平遗传变异上进行预训练，是否能有效提升下游预测性能？
• RQ4 自注意力机制在识别功能破坏性突变方面，其增强效果有多大？
• RQ5 MutFormer 是否能在人类蛋白质的多样化遗传与功能背景下实现良好泛化？

主要发现

• MutFormer 在 gnomAD 和 ClinVar 等多个基准数据集上，性能与最先进工具相当或更优。
• 模型在独立测试集上表现出更高的 AUC-ROC 评分，表明其具备出色的泛化能力。
• 自注意力机制使模型能够捕捉氨基酸位置之间的长程依赖关系，从而提升预测准确性。
• 卷积层的整合改善了对局部序列上下文的建模，有助于更准确地区分致病性变异。
• MutFormer 识别出与有害性相关的生物相关序列模式，如保守残基和结构基序。
• 微调策略显著提升了模型性能，证实了迁移学习在该领域的重要价值。