[Paper Review] PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding

Minghao Xu, Zuobai Zhang+6 more|arXiv (Cornell University)|Jun 5, 2022

Machine Learning in Bioinformatics59 citations

TL;DR

PEER proposes a comprehensive benchmark with 14 protein understanding tasks across function, localization, structure, PPI, and PLI, and evaluates single-task and multi-task learning with various baselines including pre-trained protein language models.

ABSTRACT

We are now witnessing significant progress of deep learning methods in a variety of tasks (or datasets) of proteins. However, there is a lack of a standard benchmark to evaluate the performance of different methods, which hinders the progress of deep learning in this field. In this paper, we propose such a benchmark called PEER, a comprehensive and multi-task benchmark for Protein sEquence undERstanding. PEER provides a set of diverse protein understanding tasks including protein function prediction, protein localization prediction, protein structure prediction, protein-protein interaction prediction, and protein-ligand interaction prediction. We evaluate different types of sequence-based methods for each task including traditional feature engineering approaches, different sequence encoding methods as well as large-scale pre-trained protein language models. In addition, we also investigate the performance of these methods under the multi-task learning setting. Experimental results show that large-scale pre-trained protein language models achieve the best performance for most individual tasks, and jointly training multiple tasks further boosts the performance. The datasets and source codes of this benchmark are all available at https://github.com/DeepGraphLearning/PEER_Benchmark

Motivation & Objective

• Motivate the need for a standardized, comprehensive benchmark for protein sequence understanding to accelerate progress in deep learning methods.
• Provide a multi-task benchmark (PEER) covering diverse protein understanding tasks and realistic splits to assess generalization.
• Evaluate a range of baselines from feature engineering to pre-trained protein language models, under single-task and multi-task settings.

Proposed method

• Define 14 tasks across five groups: function, localization, structure, PPI, and PLI.
• Represent proteins as sequences; represent ligands as molecular graphs for PLI tasks.
• Assess baseline models including feature engineers (DDE, Moran), sequence encoders (LSTM, Transformer, CNN, ResNet), and pre-trained models (ProtBert, ESM-1b).
• Explore three model pipelines: task-specific encoders with MLP predictors, siamese encoders for PPI, and protein–ligand encoders with a GIN for ligands.
• Investigate single-task learning versus multi-task learning with hard parameter sharing and a center/auxiliary task setup.
• Provide training, validation, and test splits designed to test generalization and out-of-distribution robustness.

Experimental results

Research questions

• RQ1 What is the performance of various sequence-based encoders and pre-trained protein language models on each PEER task?
• RQ2 Do multi-task learning setups improve performance over single-task baselines across diverse protein tasks?
• RQ3 Which task families benefit most from shared representations and knowledge transfer in a multi-task setting?
• RQ4 How do different data splits (e.g., handling high-order mutants, remote homology) impact generalization?
• RQ5 What is the relative benefit of fine-tuning pre-trained models versus freezing them across tasks?

Key findings

• Pre-trained protein language models (ESM-1b, ProtBert) achieve top performance on most tasks, especially when fine-tuned.
• Multi-task training with a shared encoder can further boost performance on center tasks compared to single-task baselines.
• Statistical features (DDE) can be competitive with learned encoders on certain tasks, highlighting complementary signals in sequences.
• Shallow CNNs can outperform deeper models when trained from scratch on several tasks.
• PPI and PLI tasks show strong sensitivity to data splits and require careful redundancy removal to assess generalization.