[Paper Review] Large scale paired antibody language models

Henry Kenlay, Frédéric A. Dreyer+4 more|arXiv (Cornell University)|Mar 26, 2024

Monoclonal and Polyclonal Antibodies Research6 citations

TL;DR

IgBert and IgT5 are large antibody-specific language models trained on paired and unpaired antibody sequences to improve sequence recovery and downstream predictive tasks, outperforming existing protein and antibody LMs on key benchmarks, and are publicly available for antibody engineering applications.

ABSTRACT

Antibodies are proteins produced by the immune system that can identify and neutralise a wide variety of antigens with high specificity and affinity, and constitute the most successful class of biotherapeutics. With the advent of next-generation sequencing, billions of antibody sequences have been collected in recent years, though their application in the design of better therapeutics has been constrained by the sheer volume and complexity of the data. To address this challenge, we present IgBert and IgT5, the best performing antibody-specific language models developed to date which can consistently handle both paired and unpaired variable region sequences as input. These models are trained comprehensively using the more than two billion unpaired sequences and two million paired sequences of light and heavy chains present in the Observed Antibody Space dataset. We show that our models outperform existing antibody and protein language models on a diverse range of design and regression tasks relevant to antibody engineering. This advancement marks a significant leap forward in leveraging machine learning, large scale data sets and high-performance computing for enhancing antibody design for therapeutic development.

Motivation & Objective

• Motivate leveraging massive antibody sequencing data to improve design and engineering of antibodies.
• Develop antibody-specific language models that can handle both paired and unpaired variable regions as input.
• Pre-train on extensive unpaired data and fine-tune on paired heavy/light chain data to learn cross-chain features.
• Evaluate models on sequence recovery, binding affinity, and expression predictions, comparing to existing antibody and protein LMs.

Proposed method

• Pre-train BERT- and T5-style models on over two billion unpaired antibody sequences from OAS, starting from ProtBert/ProtT5 weights.
• Fine-tune unpaired models on 2,038,528 unique paired heavy/light sequences from OAS, forming paired IgBert and IgT5.
• Use masked language modelling (MLM) objectives for both BERT and T5, with 15% masking and span-based masking for T5.
• Construct inputs by concatenating heavy and light chains with a separator token to learn cross-chain features.
• Fine-tune with mixed batches of unpaired and paired data to mitigate forgetting of unpaired pre-training.
• Evaluate using sequence recovery on test sets, downstream binding/expression prediction via linear models on embeddings, and perplexity/pseudo-perplexity measurements.

Figure 1: Overview of an antibody structure and its domains. The sequence of the variable region is used as input to the transformer encoder to obtain a residue-level embedding representation. Training is achieved through masked language modelling, where a random fraction of the input is replaced by

Experimental results

Research questions

• RQ1 Do antibody-specific language models trained on large-scale paired/unpaired data outperform general protein LMs on antibody design tasks?
• RQ2 Can paired training (heavy+light) yield cross-chain features that improve downstream predictions such as binding affinity and expression?
• RQ3 How do IgBert and IgT5 compare with existing AbLang, AntiBERTy, and ProtBert/ProtT5 in sequence recovery and perplexity?
• RQ4 What is the impact of data quality and pairing on model performance for antibody engineering tasks?
• RQ5 Can these models enable in silico affinity maturation and other therapeutic design workflows?

Key findings

• IgBert and IgT5 outperform existing antibody- and protein-language models on sequence recovery across antibody regions, especially in hypervariable CDRs.
• Paired models (IgBert, IgT5) yield the best performance for predicting binding affinity in linear downstream models, highlighting the value of cross-chain learning.
• General protein models (ProtBert, ProtT5) sometimes outperform antibody-specific models on expression prediction, suggesting broad evolutionary information aids certain tasks.
• Perplexity/pseudo-perplexity measurements show antibody-specific paired models achieve lower values than general protein models, indicating higher sequence “naturalness” for paired data.
• Fine-tuning on paired data provides substantial gains over pre-training on unpaired data alone, emphasizing the importance of native heavy/light chain pairing.
• The authors provide IgBert and IgT5 publicly, enabling use in antibody engineering and design workflows.

Figure 2: Data processing and training strategy. We further pre-train the ProtT5 and ProtBert models from Elnaggar et al. ( 2022 ) on unpaired antibody sequences from OAS after clustering them with Linclust. These unpaired models are then fine-tuned on paired sequences clustered with MMseqs2, combin