[Paper Review] Fine-tuning Protein Language Models with Deep Mutational Scanning improves Variant Effect Prediction

Aleix Lafita, Ferran Gonzalez Hernandez+6 more|arXiv (Cornell University)|May 10, 2024

Machine Learning in Bioinformatics10 citations

TL;DR

The paper introduces Normalised Log-odds Ratio (NLR), a lightweight fine-tuning head for Protein Language Models (PLMs) trained on Deep Mutational Scanning (DMS) data to improve missense variant effect predictions across benchmarks.

ABSTRACT

Protein Language Models (PLMs) have emerged as performant and scalable tools for predicting the functional impact and clinical significance of protein-coding variants, but they still lag experimental accuracy. Here, we present a novel fine-tuning approach to improve the performance of PLMs with experimental maps of variant effects from Deep Mutational Scanning (DMS) assays using a Normalised Log-odds Ratio (NLR) head. We find consistent improvements in a held-out protein test set, and on independent DMS and clinical variant annotation benchmarks from ProteinGym and ClinVar. These findings demonstrate that DMS is a promising source of sequence diversity and supervised training data for improving the performance of PLMs for variant effect prediction.

Motivation & Objective

• Motivate improving functional impact prediction of missense variants beyond zero-shot PLM performance.
• Propose a normalisation and fine-tuning pipeline to leverage DMS scores from multiple assays.
• Demonstrate improvements across held-out proteins and independent benchmarks.
• Assess generalisation when training data is limited and analyse model performance across proteins.
• Discuss scalability and future directions for integrating more DMS data and MSAs-based PLMs.

Proposed method

• Normalise DMS scores across assays by rescaling so synonymous means 0 and nonsense means -1, then cap to [-2, 2].
• Introduce Normalised Log-odds Ratio (NLR) head that computes a matrix of log-odds ratios for all substitutions per wildtype sequence.
• Fine-tune the ESM-1v encoder (and compare ESM-1b/ESM-2) with the NLR head; average predictions across five model checkpoints at inference.
• Train with DMS data from 25 proteins (109,215 variants) using 5-fold cross-validation and full training thereafter.
• Evaluate on held-out MaveDB test proteins, ProteinGym DMS assays, and ClinVar pathogenic/benign variants; analyse per-protein performance and baseline zero-shot results.

Figure 1: Methods overview. A) Preparation of normalised DMS functional scores from a subset of MaveDB experiments. The mean scores of synonymous and nonsense variants are used to create a common scale across assays and proteins. B) Fine-tuning pipeline for ESM-1v models using the Normalised Log-odd

Experimental results

Research questions

• RQ1 Can NLR fine-tuning improve PLM-based variant effect prediction beyond zero-shot performance?
• RQ2 Does integrating diverse DMS datasets on a common scale enhance predictions across independent benchmarks?
• RQ3 How does NLR affect performance across different PLM architectures (ESM-1v, ESM-1b, ESM-2) and how does training data size impact gains?
• RQ4 Is there improved performance for proteins with lower baseline zero-shot accuracy or limited representation in pre-training data?

Key findings

• NLR fine-tuning improves micro-averaged Spearman correlation on MaveDB test proteins from 0.478 to 0.503 (+5.2%).
• NLR fine-tuning improves ProteinGym average Spearman correlation from 0.331 to 0.396 (+19.6%).
• NLR fine-tuning increases ClinVar auROC from 0.891 to 0.902 (+1.23%).
• Per-protein ClinVar analysis shows consistent improvements, especially for proteins with lower baseline auROCs.
• ESM-1b and ESM-2 also benefit from NLR fine-tuning, with up to 25.6% relative Spearman gain on ProteinGym in some setups; benefits extend beyond a single architecture.
• Improvements scale with more DMS data and slightly more for proteins not represented in pre-training (e.g., viral proteins).

Figure 2: Results after NLR fine-tuning of ESM-1v models across benchmarks. A) Performance in the five MaveDB test proteins. ProteinGym DMS assays and ClinVar pathogenic variants. B) Spearman correlation in MaveDB test proteins. Mean $\pm$ standard deviation (std) of 50 bootstrapped samples. C) Spea