[Paper Review] Multimodal Pre-Training Model for Sequence-based Prediction of Protein-Protein Interaction

Yang Xue, Zijing Liu+2 more|arXiv (Cornell University)|Dec 9, 2021

Computational Drug Discovery Methods4 citations

TL;DR

This paper proposes S2F, a multimodal pre-training model that jointly learns protein sequence, 3D structure (via heavy atom point clouds and topology complexes), and functional descriptions to improve protein-protein interaction (PPI) prediction. By integrating structural flexibility and functional context beyond traditional contact maps, S2F achieves state-of-the-art performance across diverse PPI tasks, including cross-species PPI, antibody-antigen affinity prediction, SARS-CoV-2 neutralization, and mutation-driven binding changes.

ABSTRACT

Protein-protein interactions (PPIs) are essentials for many biological processes where two or more proteins physically bind together to achieve their functions. Modeling PPIs is useful for many biomedical applications, such as vaccine design, antibody therapeutics, and peptide drug discovery. Pre-training a protein model to learn effective representation is critical for PPIs. Most pre-training models for PPIs are sequence-based, which naively adopt the language models used in natural language processing to amino acid sequences. More advanced works utilize the structure-aware pre-training technique, taking advantage of the contact maps of known protein structures. However, neither sequences nor contact maps can fully characterize structures and functions of the proteins, which are closely related to the PPI problem. Inspired by this insight, we propose a multimodal protein pre-training model with three modalities: sequence, structure, and function (S2F). Notably, instead of using contact maps to learn the amino acid-level rigid structures, we encode the structure feature with the topology complex of point clouds of heavy atoms. It allows our model to learn structural information about not only the backbones but also the side chains. Moreover, our model incorporates the knowledge from the functional description of proteins extracted from literature or manual annotations. Our experiments show that the S2F learns protein embeddings that achieve good performances on a variety of PPIs tasks, including cross-species PPI, antibody-antigen affinity prediction, antibody neutralization prediction for SARS-CoV-2, and mutation-driven binding affinity change prediction.

Motivation & Objective

• To address the limitations of sequence-only or contact map-based protein representations in capturing full structural and functional context for PPI prediction.
• To improve PPI prediction by jointly modeling sequence, 3D structural topology (via heavy atom point clouds), and functional annotations.
• To develop a pre-training framework that generalizes across diverse PPI tasks, including cross-species interactions, antibody-antigen binding, and mutation effects.
• To enable better transfer learning for downstream biomedical applications such as vaccine design and drug discovery.

Proposed method

• The S2F model uses three modalities: amino acid sequences, 3D structural representations from heavy atom point clouds (C, N, O), and functional descriptions from literature or annotations.
• Structural features are extracted using topological data analysis (TDA) via the topology complex of point clouds, capturing both backbone and side-chain flexibility.
• The model employs a masked pre-training strategy, where input tokens from sequence, structure, and function modalities are masked and reconstructed.
• The architecture combines modality-specific encoders with cross-attention mechanisms to enable joint representation learning across modalities.
• Pre-training is conducted on a large-scale protein dataset, followed by fine-tuning on downstream PPI tasks with shared downstream networks.
• The model is evaluated using standard metrics: RMSE, Rp, AUROC, AUPR, F1, precision, and recall across multiple benchmarks.

Experimental results

Research questions

• RQ1 Can integrating structural topology from heavy atom point clouds improve protein representation for PPI prediction beyond contact maps?
• RQ2 Does incorporating functional descriptions from text enhance the model's ability to predict PPIs, especially in cases involving intrinsically disordered proteins?
• RQ3 How does the multimodal S2F model compare to sequence-only or structure-aware pre-training baselines across diverse PPI tasks?
• RQ4 Can the S2F model generalize to challenging tasks such as cross-species PPI and mutation-driven binding affinity changes?

Key findings

• On the SAbDab antibody-antigen affinity prediction task, S2F achieved an RMSE of 0.272 and Rp of 0.526, outperforming all baselines including PIPR+ProtTrans and PIPR+ProSE.
• For SARS-CoV-2 antibody neutralization prediction, S2F achieved an AUROC of 0.693, F1 score of 0.797, and AUPR of 0.836, significantly outperforming PIPR+ELMo and PIPR+TAPE.
• On the S645 mutation-driven affinity change task with Homology&Structure-split Cross Validation, S2F achieved an RMSE of 1.866 and Rp of 0.318, showing robust generalization.
• On the M1535 dataset for multiple-site mutations, S2F achieved an RMSE of 3.199 and Rp of 0.264, outperforming all compared models under the same challenging evaluation protocol.
• The ablation study confirms that all three modalities—sequence, structure, and function—contribute significantly to the final performance, with structure and function providing key gains over sequence-only models.