[Paper Review] A Comprehensive Review of Protein Language Models

Lei Wang, Xudong Li+5 more|ArXiv.org|Feb 8, 2025

Machine Learning in Bioinformatics6 citations

TL;DR

A macro-level survey of protein language models (PLMs), covering architectures, positional encoding, scaling laws, datasets, benchmarks, applications, tools, and challenges in the PLM landscape.

ABSTRACT

At the intersection of the rapidly growing biological data landscape and advancements in Natural Language Processing (NLP), protein language models (PLMs) have emerged as a transformative force in modern research. These models have achieved remarkable progress, highlighting the need for timely and comprehensive overviews. However, much of the existing literature focuses narrowly on specific domains, often missing a broader analysis of PLMs. This study provides a systematic review of PLMs from a macro perspective, covering key historical milestones and current mainstream trends. We focus on the models themselves and their evaluation metrics, exploring aspects such as model architectures, positional encoding, scaling laws, and datasets. In the evaluation section, we discuss benchmarks and downstream applications. To further support ongoing research, we introduce relevant mainstream tools. Lastly, we critically examine the key challenges and limitations in this rapidly evolving field.

Motivation & Objective

• Survey the historical milestones and current mainstream trends in PLMs.
• Analyze model architectures, positional encoding schemes, scaling laws, and pretraining datasets.
• Review evaluation benchmarks, downstream applications, and available tools for PLMs.
• Highlight key challenges, limitations, and future developmental directions in PLMs.

Proposed method

• Classify PLMs into non-transformer and Transformer-based architectures and summarize representative models.
• Discuss position encoding schemes (absolute and relative, including RoPE) and their impact on PLMs.
• Review scaling laws and their implications for model size, data, and compute in PLMs.
• Compile and categorize pretraining and structural/functional datasets and benchmarks used in PLMs.
• Provide a resource compilation of major PLMs, datasets, and tools with links.
• Critically analyze challenges such as reliance on sequence data vs. structural supervision and efficiency concerns.

Experimental results

Research questions

• RQ1 What are the predominant architectural trends and their evolution in PLMs?
• RQ2 How do positional encoding choices affect PLM performance and generalization in protein sequences?
• RQ3 What do scaling laws imply for the future growth of PLMs in terms of size, data, and compute?
• RQ4 What datasets and benchmarks most effectively support pretraining and evaluation of PLMs?
• RQ5 What are the major challenges and open questions facing PLMs and their downstream applications?

Key findings

• Transformer-based PLMs dominate current research and applications.
• RoPE-based and relative position encodings are commonly adopted to handle variable-length protein sequences.
• Scaling laws indicate performance improvements with larger models and data, with underfitting observed in some PLMs.
• A broad set of sequence, structure, and function datasets (e.g., UniProt, Pfam, AlphaFoldDB) and benchmarks (CASP, CAFA, GO, FLIP) are used for pretraining and evaluation.
• A wide range of downstream applications exist, including structure prediction, function prediction, protein design, and mutation effect prediction.
• The survey provides a consolidated resource of models, datasets, and tools to support ongoing PLM research.