[Paper Review] Reprogramming Pretrained Language Models for Protein Sequence Representation Learning

Ria Vinod, Pin‐Yu Chen+1 more|arXiv (Cornell University)|Jan 5, 2023

Machine Learning in Bioinformatics4 citations

TL;DR

This paper introduces R2DL, a framework that reprograms pretrained English language models for protein sequence representation learning by learning a sparse linear mapping between English and protein vocabulary embeddings. R2DL achieves state-of-the-art performance with up to 10⁵× higher data efficiency than standard supervised and pretrained baselines across diverse protein property and function prediction tasks.

ABSTRACT

Machine Learning-guided solutions for protein learning tasks have made significant headway in recent years. However, success in scientific discovery tasks is limited by the accessibility of well-defined and labeled in-domain data. To tackle the low-data constraint, recent adaptions of deep learning models pretrained on millions of protein sequences have shown promise; however, the construction of such domain-specific large-scale model is computationally expensive. Here, we propose Representation Learning via Dictionary Learning (R2DL), an end-to-end representation learning framework in which we reprogram deep models for alternate-domain tasks that can perform well on protein property prediction with significantly fewer training samples. R2DL reprograms a pretrained English language model to learn the embeddings of protein sequences, by learning a sparse linear mapping between English and protein sequence vocabulary embeddings. Our model can attain better accuracy and significantly improve the data efficiency by up to $10^5$ times over the baselines set by pretrained and standard supervised methods. To this end, we reprogram an off-the-shelf pre-trained English language transformer and benchmark it on a set of protein physicochemical prediction tasks (secondary structure, stability, homology, stability) as well as on a biomedically relevant set of protein function prediction tasks (antimicrobial, toxicity, antibody affinity).

Motivation & Objective

• To address the challenge of limited labeled data in protein machine learning by enabling data-efficient representation learning.
• To develop a lightweight alternative to computationally expensive, domain-specific pretraining of large protein language models.
• To explore cross-domain transfer learning by repurposing high-capacity, off-the-shelf English language models for protein sequences.
• To improve data efficiency in protein representation learning without training models from scratch.
• To enable high-performance protein property and function prediction using minimal labeled data.

Proposed method

• R2DL reprograms a pretrained English transformer by learning a sparse linear transformation between its token embeddings and protein sequence embeddings.
• The method uses k-SVD-based dictionary learning to learn a sparse mapping from English language model embeddings to protein sequence embeddings.
• A regularized objective function with L₀ norm enforces sparsity in the learned transformation matrix to improve generalization and data efficiency.
• The framework is trained end-to-end on protein sequences using the learned mapping, without fine-tuning the original language model.
• The model leverages the pre-learned linguistic inductive biases of the English language model to capture biologically meaningful representations in protein sequences.
• The approach is evaluated across multiple protein tasks, including secondary structure, stability, solubility, homology, and function prediction (antimicrobial, toxicity, antibody affinity).

Figure 1 : Left: Descriptions of considered predictive tasks. We select the set of physicochemical property prediction tasks from the well-studied domains in [ 6 ] , and the biomedical function prediction tasks from works with biomedically relevant small-szied labeled datasets [ 3 ; 14 ] . Center: W

Experimental results

Research questions

• RQ1 Can a pretrained English language model be effectively reprogrammed to learn meaningful protein sequence representations without fine-tuning?
• RQ2 To what extent can cross-domain transfer from natural language to protein sequences improve data efficiency in protein machine learning tasks?
• RQ3 How does the performance of R2DL compare to standard supervised training and domain-specific pretraining in low-data regimes?
• RQ4 Can sparse linear mapping between language model embeddings and protein sequences yield competitive results across diverse biological tasks?
• RQ5 What is the upper bound of data efficiency achievable with reprogrammed foundation models in protein representation learning?

Key findings

• R2DL achieves state-of-the-art performance on all evaluated protein prediction tasks, including secondary structure, stability, solubility, homology, antimicrobial activity, toxicity, and antibody affinity prediction.
• The model improves data efficiency by up to 10⁵× compared to standard supervised training and pretrained baselines, enabling high accuracy with minimal labeled data.
• On the antimicrobial prediction task, R2DL achieves 88% accuracy with only 6,489 training samples, matching the performance of a pretrained transformer but with significantly fewer data requirements.
• On the toxicity prediction task, R2DL attains 93.78% accuracy using just 8,153 labeled samples, outperforming standard supervised methods in low-data settings.
• For the homology classification task, R2DL achieves 26% Top-1 accuracy with 10,438 training samples, matching the performance of an LSTM baseline but with a much more efficient framework.
• The framework demonstrates broad adaptability across diverse biological tasks, including physicochemical property prediction and biomedically relevant function prediction, with consistent gains in data efficiency.

Figure 2 : System illustration of the Representation Reprogramming via Dictionary Learning (R2DL) framework. In Step 1, R2DL loads a pretrained language model (source), obtains the source vocabulary embeddings, and specifies the protein tokens (target). In Step 2, R2DL learns a sparse linear mapping