[Paper Review] ProtChatGPT: Towards Understanding Proteins with Large Language Models

Chao Wang, Hehe Fan+2 more|arXiv (Cornell University)|Feb 15, 2024

Machine Learning in Bioinformatics7 citations

TL;DR

ProtChatGPT is a ChatGPT-like system that aligns protein sequence and structure embeddings to a frozen LLM via a PLP-former and a multi-level adapter, enabling interactive protein understanding and design through natural language queries.

ABSTRACT

Protein research is crucial in various fundamental disciplines, but understanding their intricate structure-function relationships remains challenging. Recent Large Language Models (LLMs) have made significant strides in comprehending task-specific knowledge, suggesting the potential for ChatGPT-like systems specialized in protein to facilitate basic research. In this work, we introduce ProtChatGPT, which aims at learning and understanding protein structures via natural languages. ProtChatGPT enables users to upload proteins, ask questions, and engage in interactive conversations to produce comprehensive answers. The system comprises protein encoders, a Protein-Language Pertaining Transformer (PLP-former), a projection adapter, and an LLM. The protein first undergoes protein encoders and PLP-former to produce protein embeddings, which are then projected by the adapter to conform with the LLM. The LLM finally combines user questions with projected embeddings to generate informative answers. Experiments show that ProtChatGPT can produce promising responses to proteins and their corresponding questions. We hope that ProtChatGPT could form the basis for further exploration and application in protein research. Code and our pre-trained model will be publicly available.

Motivation & Objective

• Bridge the modality gap between protein data and natural language for Q&A and design tasks.
• Leverage pre-trained protein encoders and LLMs to enable interactive protein conversations without full end-to-end fine-tuning.
• Introduce a two-stage training setup (PLP-former and multi-level adapter) to align protein representations with text descriptions.
• Demonstrate the system on protein understanding and design tasks and analyze the impact of components.

Proposed method

• Use two pre-trained protein encoders to obtain sequence and structure embeddings (ESM-1b for 1D sequences and ESM-IF1 for 3D structures).
• Introduce PLP-former to align protein embeddings with textual descriptions while keeping the LLM frozen.
• Develop a multi-level projection adapter to translate aligned protein embeddings into prompts compatible with the LLM.
• Fine-tune the PLP-former and adapter in a two-stage training regime using protein-description and protein-text generation objectives.
• Employ Vicuna-13b as the LLM decoder and concatenate protein prompts with user question prompts for generation.

Figure 1: An overview of the ProtChatGPT framework. The training process consists of two stages: (1) protein-description representation learning stage, and (2) protein-to-text generative learning stage. In the first stage, we enforce the PLP Transformer, a lightweight transformer with learnable quer

Experimental results

Research questions

• RQ1 Can a frozen LLM be effectively guided to answer protein-related questions using aligned multi-level protein embeddings?
• RQ2 How do sequence-only versus sequence+structure encodings affect quality of protein-to-text alignment and generated explanations?
• RQ3 What is the impact of the PLP-former and multi-level adapter on semantic quality of generated protein descriptions?
• RQ4 Can ProtChatGPT support both understanding and design tasks for proteins through natural-language dialogue?

Key findings

• ProtChatGPT achieves higher semantic evaluation scores than ablations lacking structure or PLP-former components across multiple metrics.
• Two-stage training (PLP-former and adapter) effectively aligns protein representations with the LLM for informative responses.
• Qualitative conversations show ProtChatGPT can discuss protein meaning, functions, and applications, including design considerations and mutations.
• Case studies demonstrate differentiation of homologous proteins and handling of mutually exclusive functions through contextual prompts.
• Quantitative tests on 1,000 test protein pairs indicate the importance of both structure and PLP-former for performance.

Figure 2: Illustrations of the PLP-Former and protein-language representation learning. PLP-Former consists of two transformer submodules with shared self-attention: (1) a text transformer that performs encoding and decoding of protein descriptions, and (2) a protein transformer that interacts with