[论文解读] ProtChatGPT: Towards Understanding Proteins with Large Language Models

Chao Wang, Hehe Fan+2 more|arXiv (Cornell University)|Feb 15, 2024

Machine Learning in Bioinformatics被引用 7

一句话总结

ProtChatGPT 是一个类似 ChatGPT 的系统，通过 PLP-former 和多级适配器将蛋白质序列和结构嵌入对齐到冻结的 LLM，从而通过自然语言查询实现交互式蛋白质理解与设计。

ABSTRACT

Protein research is crucial in various fundamental disciplines, but understanding their intricate structure-function relationships remains challenging. Recent Large Language Models (LLMs) have made significant strides in comprehending task-specific knowledge, suggesting the potential for ChatGPT-like systems specialized in protein to facilitate basic research. In this work, we introduce ProtChatGPT, which aims at learning and understanding protein structures via natural languages. ProtChatGPT enables users to upload proteins, ask questions, and engage in interactive conversations to produce comprehensive answers. The system comprises protein encoders, a Protein-Language Pertaining Transformer (PLP-former), a projection adapter, and an LLM. The protein first undergoes protein encoders and PLP-former to produce protein embeddings, which are then projected by the adapter to conform with the LLM. The LLM finally combines user questions with projected embeddings to generate informative answers. Experiments show that ProtChatGPT can produce promising responses to proteins and their corresponding questions. We hope that ProtChatGPT could form the basis for further exploration and application in protein research. Code and our pre-trained model will be publicly available.

研究动机与目标

• 弥合蛋白质数据与自然语言在问答与设计任务中的模态差距。
• 利用预训练的蛋白质编码器和LLMs，实现在无需全端到端微调的情况下进行交互式蛋白质对话。
• 引入两阶段训练设置（PLP-former 和多级适配器），将蛋白质表示与文本描述对齐。
• 在蛋白质理解与设计任务上演示系统并分析各组件的影响。

提出的方法

• 使用两种预训练的蛋白质编码器获取序列和结构嵌入（1D 序列采用 ESM-1b，3D 结构采用 ESM-IF1）。
• 引入 PLP-former，使蛋白质嵌入与文本描述对齐，同时保持 LLM 冻结。
• 开发多级投影适配器，将对齐的蛋白质嵌入转换为与 LLM 兼容的提示。
• 在两阶段训练阶段，对 PLP-former 和适配器进行微调，使用蛋白质描述和蛋白质文本生成目标。
• 使用 Vicuna-13b 作为 LLM 解码器，并将蛋白质提示与用户问题提示拼接以进行生成。

Figure 1: An overview of the ProtChatGPT framework. The training process consists of two stages: (1) protein-description representation learning stage, and (2) protein-to-text generative learning stage. In the first stage, we enforce the PLP Transformer, a lightweight transformer with learnable quer

实验结果

研究问题

• RQ1 能否有效地用冻结的 LLM 在对齐的多级蛋白质嵌入下回答与蛋白质相关的问题？
• RQ2 仅序列编码与序列+结构编码在蛋白质到文本对齐质量和生成解释方面有何差异？
• RQ3 PLP-former 和多级适配器对生成的蛋白质描述的语义质量有何影响？
• RQ4 ProtChatGPT 是否能通过自然语言对话同时支持蛋白质的理解和设计任务？

主要发现

• ProtChatGPT 在多个指标上取得的语义评估分数高于缺少结构或 PLP-former 组件的消融版本。
• 两阶段训练（PLP-former 和适配器）有效地将蛋白质表示与 LLM 对齐，获得信息性强的回答。
• 定性对话显示 ProtChatGPT 能够讨论蛋白质的含义、功能与应用，包括设计考虑因素和变异。
• 案例研究展示了通过上下文提示区分同源蛋白并处理互斥功能。
• 对 1,000 对测试蛋白质的定量测试表明结构和 PLP-former 对性能都很重要。

Figure 2: Illustrations of the PLP-Former and protein-language representation learning. PLP-Former consists of two transformer submodules with shared self-attention: (1) a text transformer that performs encoding and decoding of protein descriptions, and (2) a protein transformer that interacts with