[Paper Review] Generative Antibody Design for Complementary Chain Pairing Sequences through Encoder-Decoder Language Model

Simon K. S. Chu, Kathy Y. Wei|arXiv (Cornell University)|Jan 6, 2023

Monoclonal and Polyclonal Antibodies ResearchMedicine3 citations

TL;DR

This paper introduces pAbT5, an encoder-decoder protein language model that generates complementary antibody heavy or light chains conditioned on their pairing partner, using a sequence-to-sequence framework fine-tuned on 321k paired antibody sequences. The model successfully captures framework conservation and hypervariable domain diversity, and accurately recovers chain types and gene families, demonstrating its ability to incorporate biological constraints in generative antibody design.

ABSTRACT

Current protein language models (pLMs) predominantly focus on single-chain protein sequences and often have not accounted for constraints on generative design imposed by protein-protein interactions. To address this gap, we present paired Antibody T5 (pAbT5), an encoder-decoder model to generate complementary heavy or light chain from its pairing partner. We show that our model respects conservation in framework regions and variability in hypervariable domains, demonstrated by agreement with sequence alignment and variable-length CDR loops. We also show that our model captures chain pairing preferences through the recovery of ground-truth chain type and gene families. Our results showcase the potential of pAbT5 in generative antibody design, incorporating biological constraints from chain pairing preferences.

Motivation & Objective

• To address the limitation of existing protein language models that focus on single-chain sequences and ignore co-evolutionary constraints in protein-protein interactions.
• To develop a generative model that produces biologically plausible antibody chains conditioned on their pairing partner, preserving structural and functional constraints.
• To evaluate whether the model captures sequence conservation in framework regions and variability in hypervariable loops (CDRs).
• To assess the model's ability to recover ground-truth chain types and gene families during conditional generation.
• To test the model’s generalization to unseen sequences and pairings using rigorous node-based data splitting.

Proposed method

• The model uses a T5-based encoder-decoder architecture fine-tuned on paired VH-VL sequences from the OAS database, with encoder weights frozen during fine-tuning.
• Sequence-to-sequence generation is framed as forward-translation (light to heavy) and back-translation (heavy to light), without prefix or gap tokens for chain type or species.
• The model is trained on 321k translation samples derived from 239k distinct antibody sequences across human, mouse, and rat species using a 90-5-5 train/val/test split based on exclusive node partitioning.
• Fine-tuning uses a local batch size of 8, global batch size of 2048, a 5e-5 initial learning rate, AdaFactor optimizer with gradient clipping, and 2 days of training on eight A100 GPUs.
• Model performance is evaluated via next-word prediction confidence, sequence alignment, t-SNE visualization of embeddings, and zero-shot perplexity ranking on paired sequence recovery.
• Cross-attention maps and sequence identity/length comparisons are used to analyze attention patterns and generation fidelity across framework and CDR regions.

Experimental results

Research questions

• RQ1 Can the model generate antibody sequences that respect conservation in framework regions and variability in hypervariable CDR loops?
• RQ2 Does the model capture chain pairing preferences by recovering the correct chain type and gene family of the target sequence?
• RQ3 How well does the model generalize to unseen antibody sequences and pairings under rigorous node-based data splitting?
• RQ4 To what extent does the model’s attention mechanism reflect biological relevance, particularly in distinguishing framework from variable regions?
• RQ5 Can the model’s (pseudo-)perplexity rank paired sequences accurately in a zero-shot setting, indicating functional relevance?

Key findings

• The model exhibits higher next-word prediction confidence in conserved framework regions and increased uncertainty in hypervariable CDR loops, aligning with sequence conservation profiles from position-specific scoring matrices.
• Generated sequences show high sequence identity to observed sequences in framework regions (e.g., FR4 identity: 9.94±0.36 for light, 11.00±0.00 for heavy) and reasonable similarity in CDRs (e.g., CDR3 identity: 9.63±1.06 for light, 12.32±3.93 for heavy).
• The model recovers chain types with 55% accuracy and gene families with 62% AUROC in a classification task, indicating effective capture of pairing preferences.
• t-SNE visualization of encoder hidden states shows clear clustering by chain type, gene loci, and IGHV families, confirming the model learns biologically meaningful representations.
• In zero-shot perplexity ranking, the model outperforms baselines on multiple datasets, with a mean absolute Spearman rank correlation difference of -0.05 to -0.15 in favor of pAbT5 across 12 datasets.
• Ablation studies confirm that the model’s performance is robust to data clustering and that the encoder-decoder architecture is critical for accurate pairing prediction.