[Paper Review] ProtST: Multi-Modality Learning of Protein Sequences and Biomedical Texts

Minghao Xu, Xinyu Yuan+2 more|arXiv (Cornell University)|Jan 28, 2023

Machine Learning in BioinformaticsBiochemistry, Genetics and Molecular Biology30 citations

TL;DR

ProtST introduces ProtDescribe, a paired protein sequence-text dataset, and a multi-modal pre-training framework that aligns protein sequences with biomedical text to enhance protein representations and enable zero-shot prediction and text-to-protein retrieval.

ABSTRACT

Current protein language models (PLMs) learn protein representations mainly based on their sequences, thereby well capturing co-evolutionary information, but they are unable to explicitly acquire protein functions, which is the end goal of protein representation learning. Fortunately, for many proteins, their textual property descriptions are available, where their various functions are also described. Motivated by this fact, we first build the ProtDescribe dataset to augment protein sequences with text descriptions of their functions and other important properties. Based on this dataset, we propose the ProtST framework to enhance Protein Sequence pre-training and understanding by biomedical Texts. During pre-training, we design three types of tasks, i.e., unimodal mask prediction, multimodal representation alignment and multimodal mask prediction, to enhance a PLM with protein property information with different granularities and, at the same time, preserve the PLM's original representation power. On downstream tasks, ProtST enables both supervised learning and zero-shot prediction. We verify the superiority of ProtST-induced PLMs over previous ones on diverse representation learning benchmarks. Under the zero-shot setting, we show the effectiveness of ProtST on zero-shot protein classification, and ProtST also enables functional protein retrieval from a large-scale database without any function annotation.

Motivation & Objective

• Augment protein sequence representations with textual protein properties described in biomedical texts.
• Create ProtDescribe to pair sequences with rich property descriptions.
• Develop multi-modal pre-training tasks to inject property information while preserving sequence modeling power.
• Enable downstream supervised learning and zero-shot protein classification and retrieval.

Proposed method

• Use unimodal mask prediction to preserve co-evolutionary signals from protein sequences.
• Apply multimodal representation alignment with a contrastive InfoNCE loss between sequence and text representations.
• Introduce multimodal mask prediction to recover masked residues and words via a fusion module that models interdependencies between residues and text tokens.
• Pre-train PLMs (ProtBert/ESM variants) with a frozen biomedical language model (PubMedBERT) and a fusion module.
• Align representations to support zero-shot classification using label descriptions and zero-shot text-to-protein retrieval.

Experimental results

Research questions

• RQ1 Can textual protein property descriptions improve protein sequence representations beyond sequence-only PLMs?
• RQ2 To what extent do multimodal alignment and cross-modal masking enhance downstream protein localization, fitness, and function prediction?
• RQ3 Do ProtST-induced models enable effective zero-shot protein classification using label descriptions and zero-shot retrieval without function annotations?
• RQ4 How does ProtDescribe data quality (Swiss-Prot based) impact representation learning and generalization?

Key findings

• ProtST-induced PLMs consistently outperform vanilla PLMs on localization, fitness, and function annotation benchmarks.
• ProtST-ESM-2 achieves the highest localization and competitive function-annotation metrics across evaluated settings.
• Zero-shot ProtST classifiers match or surpass certain few-shot supervised baselines in localization and reaction classification tasks.
• Zero-shot text-to-protein retrieval demonstrates the ability to identify functional proteins from GO prompts without function annotations.
• Ensembling zero-shot and supervised models can further boost performance on downstream tasks.