[Paper Review] Improving Antibody Humanness Prediction using Patent Data

Talip Uçar, Aubin Ramon+4 more|arXiv (Cornell University)|Jan 25, 2024

Monoclonal and Polyclonal Antibodies ResearchMedicine3 citations

TL;DR

This paper proposes SelfPAD, a self-supervised learning framework that leverages patent data from the Patented Antibody Database (PAD) to improve antibody humanness prediction—a critical factor in reducing immunogenicity during therapeutic antibody development. By using a two-stage training process combining weakly-supervised contrastive learning on patent-associated functional targets followed by fine-tuning with cross-entropy loss for humanness scoring, the model achieves state-of-the-art performance on five out of six immunogenicity prediction benchmarks, demonstrating superior generalization and interpretability.

ABSTRACT

We investigate the potential of patent data for improving the antibody humanness prediction using a multi-stage, multi-loss training process. Humanness serves as a proxy for the immunogenic response to antibody therapeutics, one of the major causes of attrition in drug discovery and a challenging obstacle for their use in clinical settings. We pose the initial learning stage as a weakly-supervised contrastive-learning problem, where each antibody sequence is associated with possibly multiple identifiers of function and the objective is to learn an encoder that groups them according to their patented properties. We then freeze a part of the contrastive encoder and continue training it on the patent data using the cross-entropy loss to predict the humanness score of a given antibody sequence. We illustrate the utility of the patent data and our approach by performing inference on three different immunogenicity datasets, unseen during training. Our empirical results demonstrate that the learned model consistently outperforms the alternative baselines and establishes new state-of-the-art on five out of six inference tasks, irrespective of the used metric.

Motivation & Objective

• To address the challenge of predicting antibody humanness, a key factor in immunogenicity and drug development failure.
• To explore the utility of patent data—previously underutilized in antibody representation learning—by leveraging functional associations from over 16,000 patent filings.
• To develop a multi-stage learning framework that improves representation learning and predictive performance beyond existing methods using Observed Antibody Space (OAS).
• To enable interpretable humanness predictions by identifying key residues for humanization through attention analysis.
• To demonstrate transferability of the learned model to downstream tasks such as immunogenicity prediction and humanization guidance.

Proposed method

• The method employs a two-stage training process: first, a contrastive self-supervised pre-training stage using antibody sequences linked to multiple potential targets via patent filings.
• In the pre-training phase, sequences associated with the same target are treated as positive pairs, and the model learns to group them in latent space using contrastive loss.
• The encoder is then frozen and fine-tuned on the same patent data using cross-entropy loss to predict humanness scores, enabling end-to-end optimization for the downstream task.
• A multi-layer perceptron (MLP) head is added on top of the frozen encoder and jointly trained with the sequence encoder for improved regression performance.
• The model incorporates data augmentation techniques such as swap and shift noise during fine-tuning to improve robustness.
• Interpretability is enhanced by computing attention differences between parental and humanized sequences to highlight key residues for humanization.

(a) Pre-training

Experimental results

Research questions

• RQ1 Can patent data be effectively leveraged to improve antibody representation learning and humanness prediction beyond traditional sources like OAS?
• RQ2 Does a multi-stage, multi-loss training strategy—combining contrastive learning with supervised fine-tuning—lead to better performance on humanness prediction than single-stage or baseline approaches?
• RQ3 Can the model identify biologically relevant residues for humanization by analyzing attention maps in the context of known humanized sequences?
• RQ4 How does the inclusion of noise augmentation and layer reinitialization affect model generalization and performance on immunogenicity benchmarks?
• RQ5 To what extent does the model generalize to unseen datasets, and can it outperform existing state-of-the-art models across multiple evaluation metrics?

Key findings

• The FT-SelfPAD model achieved 100% F1, recall, precision, accuracy, ROC AUC, and PR AUC on the 25 humanization data benchmark, outperforming all baselines.
• On the 217 immunogenicity dataset, the model achieved an ROC AUC of 82.86% and PR AUC of 93.79% in the ablation study, demonstrating robust performance with optimal hyperparameter choices.
• The model correctly clustered parental antibody sequences (non-human) separately from humanized and human-origin sequences in PCA visualization, confirming its ability to detect immunogenicity risk.
• Attention analysis revealed that the model identifies mutation sites in humanized antibodies with high fidelity, often recommending fewer mutations than the original humanization process.
• The ablation study confirmed that combining swap and shift noise with reinitialization of four pre-trained layers significantly improves performance, with ROC AUC increasing from 80.74% to 82.86%.
• The model established new state-of-the-art performance on five out of six inference tasks across multiple metrics, including AUC, F1, and PR AUC, across three independent immunogenicity datasets.

(b) Fine-tuning