[Paper Review] Deciphering the Language of Nature: A transformer-based language model for deleterious mutations in proteins

Theodore Jiang, Fang Li+1 more|arXiv (Cornell University)|Oct 27, 2021

Machine Learning in Bioinformatics4 citations

TL;DR

MutFormer is a transformer-based deep learning model that predicts deleterious missense mutations by leveraging both reference and mutated protein sequences, using self-attention and convolutional layers to capture long- and short-range sequence dependencies. It achieves state-of-the-art performance on multiple benchmark datasets, outperforming or matching existing tools by capturing biologically relevant sequence features not explored in prior methods.

ABSTRACT

Various machine-learning models, including deep neural network models, have already been developed to predict deleteriousness of missense (non-synonymous) mutations. Potential improvements to the current state of the art, however, may still benefit from a fresh look at the biological problem using more sophisticated self-adaptive machine-learning approaches. Recent advances in the natural language processing field show transformer models-a type of deep neural network-to be particularly powerful at modeling sequence information with context dependence. In this study, we introduce MutFormer, a transformer-based model for the prediction of deleterious missense mutations, which uses reference and mutated protein sequences from the human genome as the primary features. MutFormer takes advantage of a combination of self-attention layers and convolutional layers to learn both long-range and short-range dependencies between amino acid mutations in a protein sequence. In this study, we first pre-trained MutFormer on reference protein sequences and mutated protein sequences resulting from common genetic variants observed in human populations. We next examined different fine-tuning methods to successfully apply the model to deleteriousness prediction of missense mutations. Finally, we evaluated MutFormer's performance on multiple testing data sets. We found that MutFormer showed similar or improved performance over a variety of existing tools, including those that used conventional machine-learning approaches. We conclude that MutFormer successfully considers sequence features that are not explored in previous studies and could potentially complement existing computational predictions or empirically generated functional scores to improve our understanding of disease variants.

Motivation & Objective

• To develop a novel deep learning model that improves prediction of deleterious missense mutations using advanced sequence modeling techniques.
• To leverage the context-aware capabilities of transformer architectures to capture complex dependencies in protein sequences.
• To pre-train on human population genetic variants and fine-tune for accurate deleteriousness prediction.
• To evaluate performance against established tools on diverse, independent test datasets.
• To identify sequence features not captured by prior models, enhancing biological interpretability of pathogenicity predictions.

Proposed method

• MutFormer employs a hybrid architecture combining multi-head self-attention layers and convolutional layers to model both long-range and local sequence dependencies in protein sequences.
• The model is pre-trained on reference and mutated protein sequences derived from common human genetic variants to learn general sequence representations.
• Fine-tuning is applied using supervised learning on labeled datasets of known deleterious and neutral missense mutations.
• Input features consist of both wild-type and mutant amino acid sequences, enabling the model to learn the impact of specific substitutions.
• The model uses positional encoding to preserve sequence order and context, critical for capturing mutation effects in protein structure and function.
• Performance is evaluated using standard metrics such as AUC-ROC and AUPRC on multiple independent test sets.

Experimental results

Research questions

• RQ1 Can a transformer-based model outperform existing machine learning tools in predicting deleterious missense mutations?
• RQ2 What sequence-level features, particularly context-dependent patterns, does MutFormer learn that are not captured by conventional models?
• RQ3 How effective is pre-training on population-level genetic variants for improving downstream prediction performance?
• RQ4 To what extent do self-attention mechanisms enhance the detection of functionally disruptive mutations?
• RQ5 Can MutFormer generalize across diverse genetic and functional contexts in human proteins?

Key findings

• MutFormer achieves comparable or superior performance to state-of-the-art tools on multiple benchmark datasets, including gnomAD and ClinVar.
• The model demonstrates improved AUC-ROC scores on independent test sets, indicating strong generalization ability.
• Self-attention mechanisms enable the model to capture long-range dependencies between amino acid positions, enhancing prediction accuracy.
• The integration of convolutional layers improves modeling of local sequence context, contributing to better discrimination of pathogenic variants.
• MutFormer identifies biologically relevant sequence patterns, such as conserved residues and structural motifs, that correlate with deleteriousness.
• Fine-tuning strategies significantly enhance performance, confirming the value of transfer learning in this domain.