[Paper Review] Exploring evolution-aware & -free protein language models as protein function predictors

Mingyang Hu, Fajie Yuan+6 more|arXiv (Cornell University)|Jun 14, 2022

Machine Learning in Bioinformatics4 citations

TL;DR

This study evaluates Evoformer, the attention-based architecture from AlphaFold, as a protein function predictor, comparing it with evolution-free (ESM-1b) and evolution-aware (MSA-Transformer) protein language models. Despite its superior performance in structure prediction, Evoformer underperforms ESM-1b on most function prediction tasks, particularly zero-shot fitness prediction, while showing strong performance in stability and secondary structure prediction. The study demonstrates that ESM-1b can efficiently generate high-quality MSAs usable by evolution-aware models, highlighting a complementary role between evolution-free and evolution-aware PLMs.

ABSTRACT

Large-scale Protein Language Models (PLMs) have improved performance in protein prediction tasks, ranging from 3D structure prediction to various function predictions. In particular, AlphaFold, a ground-breaking AI system, could potentially reshape structural biology. However, the utility of the PLM module in AlphaFold, Evoformer, has not been explored beyond structure prediction. In this paper, we investigate the representation ability of three popular PLMs: ESM-1b (single sequence), MSA-Transformer (multiple sequence alignment) and Evoformer (structural), with a special focus on Evoformer. Specifically, we aim to answer the following key questions: (i) Does the Evoformer trained as part of AlphaFold produce representations amenable to predicting protein function? (ii) If yes, can Evoformer replace ESM-1b and MSA-Transformer? (ii) How much do these PLMs rely on evolution-related protein data? In this regard, are they complementary to each other? We compare these models by empirical study along with new insights and conclusions. All code and datasets for reproducibility are available at https://github.com/elttaes/Revisiting-PLMs.

Motivation & Objective

• To assess whether the Evoformer module from AlphaFold, pre-trained on 3D structures, learns general-purpose protein representations suitable for function prediction.
• To compare the functional representation capacity of Evoformer, ESM-1b (evolution-free), and MSA-Transformer (evolution-aware) across multiple downstream tasks.
• To investigate the dependency of evolution-aware PLMs on multiple sequence alignment (MSA) quality and depth, and whether ESM-1b can generate effective MSAs for these models.
• To determine whether evolution-free PLMs like ESM-1b can serve as a fast, efficient alternative for MSA construction in evolution-aware models.

Proposed method

• Fine-tuned Evoformer, ESM-1b, and MSA-Transformer with a linear classification head on diverse protein function prediction tasks, including secondary structure, contact map, function annotation (CAFA), and fitness score prediction.
• Evaluated model performance using standard metrics such as F1-score, accuracy, and AUC across multiple datasets, including CAFA2 and CAFA3 for function annotation.
• Generated MSAs using both traditional tools (Jackhmmer/HHblits) and a fast, ESM-1b-based Siamese network framework called ESM-MSA for efficient homology detection.
• Conducted zero-shot evaluation on fitness prediction tasks to assess generalization and robustness of representations without fine-tuning.
• Compared performance using MSAs of varying depth and quality, including MSAs constructed by ESM-1b, to assess sensitivity to input MSA fidelity.
• Performed ablation studies by removing MSA input entirely to evaluate the necessity of evolutionary context for Evoformer and MSA-Transformer.

Experimental results

Research questions

• RQ1 Does the Evoformer module, pre-trained in AlphaFold, produce protein representations useful for predicting protein function beyond 3D structure?
• RQ2 Can Evoformer outperform ESM-1b and MSA-Transformer on downstream protein function prediction tasks, or is it limited to structure prediction?
• RQ3 How dependent are evolution-aware PLMs on the quality and depth of input MSAs, and can evolution-free ESM-1b generate MSAs that are effective for these models?
• RQ4 Is there a complementary relationship between evolution-free and evolution-aware PLMs, where ESM-1b can efficiently generate MSAs for use in Evoformer and MSA-Transformer?

Key findings

• Evoformer achieves strong performance on secondary structure prediction (F1-score: 0.785 with MSA input) and contact map prediction, indicating its utility for structure-related tasks.
• Evoformer outperforms ESM-1b and MSA-Transformer on miniprotein stability prediction and secondary structure prediction, but underperforms ESM-1b on most function annotation and fitness prediction tasks.
• Evoformer's performance drops significantly without MSAs (F1-score: 0.614 on secondary structure), confirming its heavy reliance on evolutionary context.
• MSA-Transformer shows similar sensitivity to MSA quality, with performance dropping from 0.748 (with MSA) to 0.634 (without MSA) on secondary structure prediction.
• ESM-1b-generated MSAs yield comparable performance to Jackhmmer/HHblits-derived MSAs for both Evoformer and MSA-Transformer, with ESM-MSA being significantly faster.
• In zero-shot fitness prediction, ESM-1b outperforms both Evoformer and MSA-Transformer, indicating that its representations are more robust for such tasks.