[Paper Review] Retrieval-Enhanced Mutation Mastery: Augmenting Zero-Shot Prediction of Protein Language Model

Yang Tan, Ruilin Wang+3 more|arXiv (Cornell University)|Oct 28, 2024

Topic Modeling4 citations

TL;DR

This paper proposes ProtREM, a retrieval-enhanced protein language model that integrates sequence, local structural, and evolutionary information via disentangled cross-attention and homologous sequence retrieval to improve zero-shot mutation effect prediction. It achieves state-of-the-art performance on over 2 million mutants across 217 assays in ProteinGym and enables successful in silico and wet-lab validation of enhanced DNA polymerase and VHH antibody mutants.

ABSTRACT

Enzyme engineering enables the modification of wild-type proteins to meet industrial and research demands by enhancing catalytic activity, stability, binding affinities, and other properties. The emergence of deep learning methods for protein modeling has demonstrated superior results at lower costs compared to traditional approaches such as directed evolution and rational design. In mutation effect prediction, the key to pre-training deep learning models lies in accurately interpreting the complex relationships among protein sequence, structure, and function. This study introduces a retrieval-enhanced protein language model for comprehensive analysis of native properties from sequence and local structural interactions, as well as evolutionary properties from retrieved homologous sequences. The state-of-the-art performance of the proposed ProtREM is validated on over 2 million mutants across 217 assays from an open benchmark (ProteinGym). We also conducted post-hoc analyses of the model's ability to improve the stability and binding affinity of a VHH antibody. Additionally, we designed 10 new mutants on a DNA polymerase and conducted wet-lab experiments to evaluate their enhanced activity at higher temperatures. Both in silico and experimental evaluations confirmed that our method provides reliable predictions of mutation effects, offering an auxiliary tool for biologists aiming to evolve existing enzymes. The implementation is publicly available at https://github.com/tyang816/ProtREM.

Motivation & Objective

• Address the gap in existing protein language models that fail to jointly integrate sequence, structure, and evolutionary information for mutation effect prediction.
• Improve the reliability of zero-shot prediction for single- and multi-site protein mutations in low-throughput experimental settings.
• Develop a unified framework that leverages retrieved homologous sequences to enhance evolutionary representation and mitigate inaccuracies from predicted 3D structures.
• Validate the model’s predictive power through both high-throughput benchmark evaluation and targeted wet-lab experiments on functional enzymes.
• Provide a publicly available tool to support biologists in designing functionally improved protein variants with minimal experimental cost.

Proposed method

• ProtREM encodes protein sequences and local 3D structural features into sequence and structure tokens, with local structures clustered into 2048 distinct structural tokens.
• Disentangled multi-head cross-attention layers integrate sequence and structural representations in a BERT-style pre-training paradigm.
• Homologous sequences are retrieved via Jackhmmer and converted into evolutionary logits using alignment-based tokenization.
• The model fuses structural, sequence, and evolutionary representations to generate fitness scores for each residue mutation.
• Pre-training is conducted using masked language modeling objectives on full-length protein sequences with joint optimization of sequence, structure, and evolutionary signals.
• The model is fine-tuned on high-throughput DMS data from ProteinGym and evaluated on both benchmark and low-throughput experimental datasets.

Experimental results

Research questions

• RQ1 Can ProtREM consistently outperform existing models in predicting mutation effects across diverse proteins and assays using large-scale DMS data?
• RQ2 How well does ProtREM generalize to low-throughput experimental settings where precise functional measurements are available?
• RQ3 To what extent can ProtREM guide the design of functionally improved mutants in real-world enzyme engineering applications?
• RQ4 Does the integration of retrieved homologous sequences significantly improve prediction reliability compared to models relying only on sequence or structure?
• RQ5 Can ProtREM identify novel, high-performing mutants that are experimentally validated for enhanced thermostability or activity?

Key findings

• ProtREM achieves state-of-the-art performance on the ProteinGym benchmark, with Spearman’s ρ exceeding prior methods across 217 assays and over 2 million mutants.
• The model demonstrates higher predictive reliability on low-throughput experimental data, successfully identifying beneficial mutations in a VHH antibody for improved stability and binding affinity.
• Wet-lab validation of 10 designed mutants in phi29 DNA polymerase confirmed enhanced thermostability, with increased Tm values measured via differential scanning fluorimetry.
• The integration of retrieved homologous sequences significantly improves evolutionary representation, reducing reliance on potentially inaccurate predicted 3D structures.
• Post-hoc analyses show that ProtREM effectively identifies conserved, non-mutatable regions and prioritizes functionally relevant mutations.
• The implementation is publicly available at https://github.com/tyang816/ProtREM, enabling reproducibility and community adoption.