[Paper Review] ProteinGPT: Multimodal LLM for Protein Property Prediction and Structure Understanding

Yijia Xiao, Edward W. Sun+3 more|arXiv (Cornell University)|Aug 21, 2024

Machine Learning in Bioinformatics6 citations

TL;DR

ProteinGPT is a multimodal protein chat system that fuses protein sequences and structures into an LLM to enable interactive questions and concise QA, trained with a GPT-4o-guided instruction-tuning approach on a large ProteinQA dataset.

ABSTRACT

Understanding biological processes, drug development, and biotechnological advancements requires a detailed analysis of protein structures and functions, a task that is inherently complex and time-consuming in traditional protein research. To streamline this process, we introduce ProteinGPT, a state-of-the-art multimodal large language model for proteins that enables users to upload protein sequences and/or structures for comprehensive analysis and responsive inquiries. ProteinGPT integrates protein sequence and structure encoders with linear projection layers to ensure precise representation adaptation and leverages a large language model (LLM) to generate accurate, contextually relevant responses. To train ProteinGPT, we constructed a large-scale dataset of 132,092 proteins, each annotated with 20-30 property tags and 5-10 QA pairs per protein, and optimized the instruction-tuning process using GPT-4o. Experiments demonstrate that ProteinGPT effectively generates informative responses to protein-related questions, achieving high performance on both semantic and lexical metrics and significantly outperforming baseline models and general-purpose LLMs in understanding and responding to protein-related queries. Our code and data are available at https://github.com/ProteinGPT/ProteinGPT.

Motivation & Objective

• Motivate faster and more accurate protein analysis by combining sequence and structure data with natural language interactions.
• Develop a multi-modal pipeline that maps protein sequence and structure embeddings to an LLM-friendly representation.
• Create a large, richly annotated instruction-tuning dataset (ProteinQA) to teach the model to answer concise, relevant protein questions.

Proposed method

• Two frozen encoders (sequence: esm2_t36_3B_UR50D; structure: esm_if1_gvp4_t16_142M_UR50) generate embeddings.
• A linear projection layer aligns both embeddings with LLM embeddings to form soft prompts.
• Two-stage training: (1) modality alignment with the projection layer; (2) instruction tuning using a GPT-4o guided QA dataset.
• ProteinQA provides 132,092 protein samples with abstracts, 20–30 property tags, and 5–10 QA pairs per sample.
• QA augmentation with GPT-4o to create explicit Question-Answer pairs for instruction tuning.
• Evaluation includes semantic metrics (BERTScore, PubMedBERT-Score, GPT Score) and lexical metrics (ROUGE) across multiple base LLMs.

Figure 1: ProteinGPT Modality Fusion & Alignment Stage: we freeze the encoder blocks and train the linear project layer to learn how to align protein structure and protein sequence representations with text. In the alignment stage, the input to the training is only the projected protein representati

Experimental results

Research questions

• RQ1 Can multimodal fusion of protein sequence and structure improve LLM-based protein understanding and QA accuracy?
• RQ2 How does modality alignment followed by instruction tuning affect semantic and lexical QA performance for protein questions?
• RQ3 Does ProteinGPT outperform vanilla LLMs and GPT-based baselines on protein-focused inquiries?

Key findings

• ProteinGPT achieves high semantic similarity scores in evaluation, but lower lexical overlaps due to sentence-form outputs.
• Llama-3 and Mistral backbones yield the best performance among the tested models for ProteinGPT.
• ProteinGPT consistently outperforms vanilla LLM baselines and OpenAI GPT models on semantic evaluations across the tested tasks.
• Closed-ended QA accuracy reaches around 70–80% for some backbones, indicating reliable factual responses for single-word ground truths.
• Modality alignment improves performance over vanilla LLMs, and instruction tuning provides additional gains over modality alignment alone.

Figure 2: ProteinGPT Instruction Tuning Stage: we utilize the QA pairs and property tags in ProteinQA to tune the LLM to follow instructions and give concise responses. For instruction alignment, explicit prompts ( Questions on the protein) are included at the beginning of the prompt.