[Paper Review] Enhancing the efficiency of protein language models with minimal wet-lab data through few-shot learning

Ziyi Zhou, Liang Zhang+4 more|arXiv (Cornell University)|Feb 3, 2024

Machine Learning in BioinformaticsBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

This paper proposes FSFP, a few-shot fine-tuning strategy that enhances protein language models using only tens of wet-lab measurements of single-site mutants. By combining meta-transfer learning, learning-to-rank, and parameter-efficient fine-tuning, FSFP significantly improves prediction accuracy across 87 deep mutational scanning datasets, outperforming both unsupervised and supervised baselines under extreme data scarcity.

ABSTRACT

Accurately modeling the protein fitness landscapes holds great importance for protein engineering. Recently, due to their capacity and representation ability, pre-trained protein language models have achieved state-of-the-art performance in predicting protein fitness without experimental data. However, their predictions are limited in accuracy as well as interpretability. Furthermore, such deep learning models require abundant labeled training examples for performance improvements, posing a practical barrier. In this work, we introduce FSFP, a training strategy that can effectively optimize protein language models under extreme data scarcity. By combining the techniques of meta-transfer learning, learning to rank, and parameter-efficient fine-tuning, FSFP can significantly boost the performance of various protein language models using merely tens of labeled single-site mutants from the target protein. The experiments across 87 deep mutational scanning datasets underscore its superiority over both unsupervised and supervised approaches, revealing its potential in facilitating AI-guided protein design.

Motivation & Objective

• To address the limited performance and interpretability of pre-trained protein language models in predicting protein fitness landscapes.
• To overcome the practical barrier of requiring large amounts of labeled experimental data for model fine-tuning.
• To develop a training strategy that achieves high performance with minimal wet-lab data—specifically, only tens of single-site mutant measurements.
• To enhance the efficiency and generalization of protein language models in AI-guided protein design under data scarcity.
• To integrate meta-learning, ranking loss, and parameter-efficient fine-tuning for optimal adaptation to target proteins with minimal data.

Proposed method

• FSFP employs meta-transfer learning to enable fast adaptation of protein language models to new proteins using few labeled examples.
• It incorporates a learning-to-rank loss function to align model predictions with the relative fitness order of protein variants.
• Parameter-efficient fine-tuning techniques are used to update only a small subset of model parameters, reducing computational cost and preventing overfitting.
• The method is trained on a diverse set of protein families to generalize across different target proteins and mutation types.
• It leverages pre-trained protein language models as the backbone, fine-tuning them with minimal labeled data from deep mutational scanning experiments.
• The framework is designed to be scalable and applicable to various protein engineering tasks with limited experimental validation.

Experimental results

Research questions

• RQ1 Can protein language models achieve high prediction accuracy for fitness landscapes using only tens of labeled single-site mutants?
• RQ2 How does combining meta-learning, ranking loss, and parameter-efficient fine-tuning improve model generalization under data scarcity?
• RQ3 Does FSFP outperform both unsupervised pre-training and fully supervised fine-tuning in few-shot protein fitness prediction?
• RQ4 To what extent can FSFP generalize across diverse protein families and mutation types with minimal experimental data?
• RQ5 Can the integration of learning-to-rank loss enhance model interpretability and ranking performance on protein fitness scores?

Key findings

• FSFP significantly improves protein fitness prediction accuracy across 87 deep mutational scanning datasets, even with only 10–50 labeled single-site mutants.
• The method outperforms both unsupervised pre-training and standard supervised fine-tuning, especially in low-data regimes.
• FSFP achieves state-of-the-art performance on multiple benchmark datasets, demonstrating superior generalization across diverse protein families.
• The use of learning-to-rank loss leads to better alignment with biological fitness order, improving interpretability of model predictions.
• Parameter-efficient fine-tuning enables effective adaptation with minimal computational overhead and reduced risk of overfitting.
• Meta-learning enables rapid adaptation to new proteins, confirming the method's robustness in few-shot scenarios.