[Paper Review] PepDoRA: A Unified Peptide Language Model via Weight-Decomposed Low-Rank Adaptation

Leyao Wang, Rishab Pulugurta+4 more|arXiv (Cornell University)|Oct 28, 2024

Machine Learning in Bioinformatics4 citations

TL;DR

PepDoRA is a unified peptide language model that leverages Weight-Decomposed Low-Rank Adaptation (DoRA) to fine-tune ChemBERTa-77M-MLM on 100,000 diverse, experimentally validated peptides, enabling accurate prediction of therapeutic properties such as membrane permeability, hemolysis, non-fouling propensity, and target-specific binding—outperforming both general-purpose and peptide-specific baselines.

ABSTRACT

Peptide therapeutics, including macrocycles, peptide inhibitors, and bioactive linear peptides, play a crucial role in therapeutic development due to their unique physicochemical properties. However, predicting these properties remains challenging. While structure-based models primarily focus on local interactions, language models are capable of capturing global therapeutic properties of both modified and linear peptides. Protein language models like ESM-2, though effective for natural peptides, cannot however encode chemical modifications. Conversely, pre-trained chemical language models excel in representing small molecule properties but are not optimized for peptides. To bridge this gap, we introduce PepDoRA, a unified peptide representation model. Leveraging Weight-Decomposed Low-Rank Adaptation (DoRA), PepDoRA efficiently fine-tunes the ChemBERTa-77M-MLM on a masked language model objective to generate optimized embeddings for downstream property prediction tasks involving both modified and unmodified peptides. By tuning on a diverse and experimentally valid set of 100,000 modified, bioactive, and binding peptides, we show that PepDoRA embeddings capture functional properties of input peptides, enabling the accurate prediction of membrane permeability, non-fouling and hemolysis propensity, and via contrastive learning, target protein-specific binding. Overall, by providing a unified representation for chemically and biologically diverse peptides, PepDoRA serves as a versatile tool for function and activity prediction, facilitating the development of peptide therapeutics across a broad spectrum of applications.

Motivation & Objective

• Address the gap in existing models that cannot effectively represent both natural and modified peptides, especially those with complex modifications like cyclization or unnatural amino acids.
• Overcome limitations of protein language models (e.g., ESM-2) and chemical language models (e.g., ChemBERTa) in capturing functional and physicochemical properties of bioactive peptides.
• Develop a unified representation framework that combines the robust chemical knowledge of pre-trained cLMs with efficient fine-tuning for peptide-specific tasks.
• Enable accurate prediction of key therapeutic properties—such as membrane permeability, hemolysis, and non-fouling—without requiring structural data.
• Facilitate the design of peptide therapeutics targeting challenging, conformationally diverse or undruggable proteins through joint peptide-protein embedding learning.

Proposed method

• Fine-tune the pre-trained ChemBERTa-77M-MLM model using Weight-Decomposed Low-Rank Adaptation (DoRA), a parameter-efficient fine-tuning method that decomposes weight updates into low-rank matrices.
• Train on a diverse, experimentally validated dataset of 100,000 peptides, including both modified and bioactive linear peptides, using a masked language modeling objective.
• Leverage DoRA to preserve the pre-trained chemical knowledge of ChemBERTa while adapting it to peptide-specific functional properties.
• Employ contrastive learning with a CLIP-inspired architecture to learn joint peptide-protein embeddings, maximizing cosine similarity for true binding pairs.
• Use the resulting peptide embeddings as input features for downstream regression and classification tasks, including membrane permeability, hemolysis, and non-fouling prediction.
• Compare performance against baseline models including full fine-tuning, LoRA, PeptideCLM, and PeptideBERT to validate the effectiveness of DoRA-based adaptation.

Experimental results

Research questions

• RQ1 Can DoRA-based fine-tuning effectively adapt a chemical language model (ChemBERTa) to capture functional and physicochemical properties of diverse peptides, including modified and bioactive sequences?
• RQ2 How does PepDoRA compare to specialized peptide models like PeptideBERT and PeptideCLM in predicting key therapeutic properties such as membrane permeability and hemolysis?
• RQ3 To what extent can PepDoRA embeddings enable accurate prediction of target-specific peptide-protein binding, especially for challenging or disordered targets?
• RQ4 Does the unified representation learned by PepDoRA generalize across both natural and modified peptides without requiring separate model architectures?
• RQ5 Can the contrastive learning framework using PepDoRA embeddings effectively distinguish true from non-binding peptide-protein pairs in a zero-shot or few-shot setting?

Key findings

• PepDoRA achieved 87% accuracy, 70% F1-score, and 71% recall on non-fouling classification, outperforming PeptideBERT (87% accuracy, 69% F1, 67% recall).
• On hemolysis prediction, PepDoRA achieved 80% accuracy and 37% F1-score, slightly outperforming PeptideBERT (81% accuracy, 35% F1), with significantly better recall (27% vs. 25%).
• In peptide-protein interaction prediction, PepDoRA achieved 95.9% binary accuracy, 62.1% Top-1 accuracy, and 86.1% Top 10% accuracy, surpassing all baselines including LoRA-fine-tuned ChemBERTa (95.1%, 60.0%, 84.9%).
• PepDoRA’s performance on target-specific binding prediction via contrastive learning demonstrates strong generalization, particularly for undruggable or conformationally flexible targets.
• The model’s DoRA-based fine-tuning strategy preserved ChemBERTa’s chemical knowledge while enabling high-fidelity adaptation to peptide-specific tasks, avoiding full fine-tuning costs.
• PepDoRA outperformed the specialized PeptideCLM-23M model (93.9% binary accuracy, 47.3% Top-1 accuracy) on peptide-protein interaction tasks, indicating broader applicability across peptide types.