[论文解读] Protein Language Models and Structure Prediction: Connection and Progression

Bozhen Hu, Jun Xia+5 more|arXiv (Cornell University)|Nov 30, 2022

Machine Learning in Bioinformatics被引用 21

一句话总结

一项系统性综述，将蛋白质语言模型（pLMs）与蛋白质结构预测（PSP）联系起来，详细介绍架构、预训练、数据库和应用，以及未来方向。

ABSTRACT

The prediction of protein structures from sequences is an important task for function prediction, drug design, and related biological processes understanding. Recent advances have proved the power of language models (LMs) in processing the protein sequence databases, which inherit the advantages of attention networks and capture useful information in learning representations for proteins. The past two years have witnessed remarkable success in tertiary protein structure prediction (PSP), including evolution-based and single-sequence-based PSP. It seems that instead of using energy-based models and sampling procedures, protein language model (pLM)-based pipelines have emerged as mainstream paradigms in PSP. Despite the fruitful progress, the PSP community needs a systematic and up-to-date survey to help bridge the gap between LMs in the natural language processing (NLP) and PSP domains and introduce their methodologies, advancements and practical applications. To this end, in this paper, we first introduce the similarities between protein and human languages that allow LMs extended to pLMs, and applied to protein databases. Then, we systematically review recent advances in LMs and pLMs from the perspectives of network architectures, pre-training strategies, applications, and commonly-used protein databases. Next, different types of methods for PSP are discussed, particularly how the pLM-based architectures function in the process of protein folding. Finally, we identify challenges faced by the PSP community and foresee promising research directions along with the advances of pLMs. This survey aims to be a hands-on guide for researchers to understand PSP methods, develop pLMs and tackle challenging problems in this field for practical purposes.

研究动机与目标

• 解释蛋白质序列如何被视为自然语言，以及为什么语言模型对PSP是适用的。
• 综述pLMs在PSP中的网络架构、预训练策略和应用。
• 总结常用的蛋白质数据库及其在预训练中的作用。
• 讨论基于pLM的架构如何整合到PSP流程和折叠机制中。
• 确定挑战并为pLM和PSP的未来研究提出有前景的方向。

提出的方法

• 回顾并分类现有的蛋白质语言模型（pLMs）及其预训练方法。
• 分析pLMs如何被纳入PSP流程和结构特征学习。
• 在pLMs背景下比较基于进化的和基于单序列的PSP方法。
• 总结用于pLM预训练的蛋白质数据库和数据资源。
• 突出局限性并预测该领域的未来趋势。

实验结果

研究问题

• RQ1 在数据、表示和目标方面，pLMs与传统PSP方法有何关系？
• RQ2 哪些架构选择和预训练策略对PSP任务最有效？
• RQ3 支撑PSP的pLM开发的关键数据库和数据资源有哪些？
• RQ4 基于pLM的PSP方法目前的局限性有哪些，未来哪些方向有前景？

主要发现

• pLMs利用大规模蛋白质序列数据来学习对结构与功能预测有用的表示。
• 基于Transformer的pLMs，结合在多样化数据库上的预训练，推动了PSP的进展，包括单序列和进化基于的方法。
• 多种预训练策略（如掩码语言建模）和架构（RNN/LSTM 与 Transformer）被用于捕捉蛋白质的依赖关系。
• 在某些情境中，pLMs使PSP流程能够在有限或无进化信息的情况下运行。
• 该综述综合了抗体、蛋白质复合体、蛋白质-配体和蛋白质-RNA结构相关任务的方法。
• 挑战包括蛋白质序列的分词、标注数据有限，以及需要整合多模态数据。