[Paper Review] A Systematic Study of Joint Representation Learning on Protein Sequences and Structures

Zuobai Zhang, Wang, Chuanrui+5 more|arXiv (Cornell University)|Mar 11, 2023

Machine Learning in Bioinformatics26 citations

TL;DR

The paper investigates combining a protein language model (ESM-2) with structure encoders (GVP, GearNet, CDConv) using three fusion strategies and six pre-training methods to achieve state-of-the-art protein function annotation and structure-related tasks.

ABSTRACT

Learning effective protein representations is critical in a variety of tasks in biology such as predicting protein functions. Recent sequence representation learning methods based on Protein Language Models (PLMs) excel in sequence-based tasks, but their direct adaptation to tasks involving protein structures remains a challenge. In contrast, structure-based methods leverage 3D structural information with graph neural networks and geometric pre-training methods show potential in function prediction tasks, but still suffers from the limited number of available structures. To bridge this gap, our study undertakes a comprehensive exploration of joint protein representation learning by integrating a state-of-the-art PLM (ESM-2) with distinct structure encoders (GVP, GearNet, CDConv). We introduce three representation fusion strategies and explore different pre-training techniques. Our method achieves significant improvements over existing sequence- and structure-based methods, setting new state-of-the-art for function annotation. This study underscores several important design choices for fusing protein sequence and structure information. Our implementation is available at https://github.com/DeepGraphLearning/ESM-GearNet.

Motivation & Objective

• Motivate and enable effective joint protein representation learning by integrating sequence and structure information.
• Systematically compare three fusion strategies (serial, parallel, cross) for combining PLM and structure encoders.
• Evaluate six pre-training techniques to leverage unlabeled sequence and structure data, including AlphaFold database pre-training.
• Demonstrate performance gains on protein function annotation (EC, GO) and structure-related tasks (PSR, MSP).

Proposed method

• Use a state-of-the-art PLM (ESM-2) with three structure encoders (GVP, GearNet, CDConv).
• Introduce three fusion schemes: serial fusion (structure encoder input initialized from PLM outputs), parallel fusion (concatenate sequence and structure representations), and cross fusion (cross-attention between modalities).
• Adopt a reduced learning rate for PLMs during training to protect pretrained representations.
• Pre-train the joint encoder on unlabeled data from AlphaFold Database using six objectives (residue type, distance, angle, dihedral, multiview contrast, SiamDiff).
• Explore six pre-training methods: residue type prediction, distance prediction, angle prediction, dihedral prediction, multiview contrast, SiamDiff (diffusion-based).
• Evaluate on EC, GO (BP/MF/CC), PSR, MSP tasks using protein-centric F-max, AUROC, and Spearman correlations; compare against SOTA baselines.

Experimental results

Research questions

• RQ1 Can integrating a PLM with structure encoders outperform single-modality models on protein function and structure tasks?
• RQ2 Which fusion strategy best leverages bimodal sequence-structure information across tasks?
• RQ3 What pre-training objectives best enhance joint sequence-structure representations for function and structure prediction?
• RQ4 How crucial is preserving PLM representations when jointly training with structure encoders?
• RQ5 Do joint representations achieve new state-of-the-art performance on EC and GO annotations?

Key findings

• Serial fusion consistently outperforms other fusion strategies across most tasks.
• Reducing the PLM learning rate is crucial to prevent forgetting and maintain performance gains.
• GearNet, though weaker standalone, benefits significantly from integration with PLMs, especially for EC and GO tasks.
• Joint pre-training with both sequence and structure objectives yields better performance than using either modality alone.
• Multiview Contrast and SiamDiff emerge as top pre-training methods, with Multiview Contrast aiding function annotation and SiamDiff aiding structure-related tasks.
• The proposed ESМ-GearNet approach achieves new state-of-the-art results on Enzyme Commission and Gene Ontology annotation tasks, and competitive results on PSR and MSP.