[Paper Review] Deciphering antibody affinity maturation with language models and weakly supervised learning

Jeffrey A. Ruffolo, Jeffrey J. Gray+1 more|arXiv (Cornell University)|Dec 14, 2021

Monoclonal and Polyclonal Antibodies Research91 citations

TL;DR

The paper introduces AntiBERTy, an antibody-specific language model trained on 558M sequences, and uses a weakly supervised MIL framework to identify binding residues within repertoires. It demonstrates that repertoire embeddings reveal affinity-maturation trajectories and that MIL attention localizes to paratope residues in VRC01-class antibodies.

ABSTRACT

In response to pathogens, the adaptive immune system generates specific antibodies that bind and neutralize foreign antigens. Understanding the composition of an individual's immune repertoire can provide insights into this process and reveal potential therapeutic antibodies. In this work, we explore the application of antibody-specific language models to aid understanding of immune repertoires. We introduce AntiBERTy, a language model trained on 558M natural antibody sequences. We find that within repertoires, our model clusters antibodies into trajectories resembling affinity maturation. Importantly, we show that models trained to predict highly redundant sequences under a multiple instance learning framework identify key binding residues in the process. With further development, the methods presented here will provide new insights into antigen binding from repertoire sequences alone.

Motivation & Objective

• Develop an antibody-focused language model to learn repertoire representations.
• Explore affinity maturation trajectories through embeddings of antibody sequences.
• Identify binding/paratope residues using a multiple instance learning framework.
• Demonstrate that MIL attention aligns with known paratope regions in VRC01-class antibodies.

Proposed method

• Train AntiBERTy (BERT-style transformer encoder) on 558M non-redundant antibody sequences using masked language modeling.
• Construct k-NN graphs and evo-velocity analyses from AntiBERTy embeddings to study repertoire evolution.
• Use a multiple instance learning framework to predict bags containing highly redundant (likely binding) sequences.
• Encode sequences with AntiBERTy to produce per-residue embeddings, followed by attention-based pooling and a MIL classifier.
• Train MIL models on bags of 64 sequences with equal positive/negative sampling and use cross-entropy loss.
• Analyze attention maps to identify residues contributing to binding, comparing with known VRC01 gp120 complex structures.

Experimental results

Research questions

• RQ1 Can an antibody-specific language model reveal affinity-maturation trajectories within immune repertoires?
• RQ2 Does a MIL framework using AntiBERTy embeddings identify paratope residues in antibodies from repertoire data?
• RQ3 Do MIL attention scores localize to known binding regions (paratopes) in HIV-1VRC01-class antibodies?
• RQ4 What are the quantitative performance characteristics of MIL models on donor-specific and combined repertoires?

Key findings

• AntiBERTy embeddings reveal continuous trajectories from germline to highly mutated derivatives in repertoire embeddings, consistent with affinity maturation.
• MIL models identify bags containing highly redundant sequences with reasonable accuracy across donors (AUROC up to 0.96 for some repertoires).
• Attention maps from MIL embeddings localize to known paratope regions (H2 loop and following beta strand) in several VRC01 antibodies.
• Single-instance MIL predictions correctly flag ten crystallized VRC01 antibodies in binding contexts, validating the approach.
• Combined repertoire MIL model achieves AUROC 0.84 and F1 up to 0.77, indicating transferable signals across donors.