[Paper Review] BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine

Yizhen Luo, Jiahuan Zhang+5 more|arXiv (Cornell University)|Aug 18, 2023

Machine Learning in BioinformaticsBiochemistry, Genetics and Molecular Biology27 citations

TL;DR

BioMedGPT introduces an open multimodal generative transformer for biomedicine that aligns molecules, proteins, and natural language in a unified feature space. BioMedGPT-10B achieves strong biomedical QA performance and open-sources its models and multimodal datasets.

ABSTRACT

Foundation models (FMs) have exhibited remarkable performance across a wide range of downstream tasks in many domains. Nevertheless, general-purpose FMs often face challenges when confronted with domain-specific problems, due to their limited access to the proprietary training data in a particular domain. In biomedicine, there are various biological modalities, such as molecules, proteins, and cells, which are encoded by the language of life and exhibit significant modality gaps with human natural language. In this paper, we introduce BioMedGPT, an open multimodal generative pre-trained transformer (GPT) for biomedicine, to bridge the gap between the language of life and human natural language. BioMedGPT allows users to easily ``communicate'' with diverse biological modalities through free text, which is the first of its kind. BioMedGPT aligns different biological modalities with natural language via a large generative language model, namely, BioMedGPT-LM. We publish BioMedGPT-10B, which unifies the feature spaces of molecules, proteins, and natural language via encoding and alignment. Through fine-tuning, BioMedGPT-10B outperforms or is on par with human and significantly larger general-purpose foundation models on the biomedical QA task. It also demonstrates promising performance in the molecule QA and protein QA tasks, which could greatly accelerate the discovery of new drugs and therapeutic targets. In addition, BioMedGPT-LM-7B is the first large generative language model based on Llama2 in the biomedical domain, therefore is commercial friendly. Both BioMedGPT-10B and BioMedGPT-LM-7B are open-sourced to the research community. In addition, we publish the datasets that are meticulously curated for the alignment of multi-modalities, i.e., PubChemQA and UniProtQA. All the models, codes, and datasets are available at \url{https://github.com/PharMolix/OpenBioMed}.

Motivation & Objective

• Bridge the language of life with human natural language using a large language model fine-tuned on biomedical data.
• Unify textual, molecular, and protein modalities through independent encoders and alignment into a shared feature space.
• Demonstrate BioMedGPT-10B on biomedical QA, molecule QA, and protein QA tasks and release datasets for modality alignment.

Proposed method

• Fine-tune Llama2-Chat-7B on a large biomedical corpus to create BioMedGPT-LM-7B.
• Build BioMedGPT-10B by aligning 2D molecular graphs and protein sequences with the natural language space via modality adaptors.
• Use GraphMVP as the molecule encoder and ESM-2 as the protein encoder with independent modality adaptors.
• Perform multimodal fine-tuning using two curated datasets, PubChemQA and UniProtQA, with role-based prompts to guide the model.
• Freeze BioMedGPT-LM parameters and train molecule/protein encoders and adaptors (mPLUG-owl-like approach) to save compute and avoid forgetting.
• Evaluate using biomedical QA benchmarks (MedMCQA, PubMedQA, USMLE), molecule QA (ChEBI-20), and protein QA (UniProtQA), reporting BLEU/ROUGE/METEOR metrics.

Experimental results

Research questions

• RQ1 Can a single large biomedical language model effectively align and reason over multiple modalities (molecules, proteins, text)?
• RQ2 Does fine-tuning and dedicated multimodal alignment improve performance on biomedical QA tasks beyond general-purpose LLMs?
• RQ3 How do molecule QA and protein QA capabilities compare to baseline language models when data are presented in a unified multimodal space?
• RQ4 What datasets and prompting strategies best support multimodal alignment in biomedicine?

Key findings

• BioMedGPT-10B achieves state-of-the-art or on-par results with substantially larger models on biomedical QA benchmarks (MedMCQA, PubMedQA) and outperforms baselines on out-of-domain USMLE.
• On PubMedQA, BioMedGPT-10B reaches human-expert-like performance.
• In molecule QA, BioMedGPT-10B significantly outperforms ChatGPT and Llama2-7B-Chat in BLEU-2, BLEU-4, ROUGE metrics when using alignment.
• In protein QA, BioMedGPT-10B with alignment yields strong BLEU/ROUGE results and surpasses baselines notably, demonstrating effective integration of protein sequence data with natural language.
• BioMedGPT-LM-7B is the first biomedical variant of Llama2-based generative model and is open-sourced; BioMedGPT-10B is open-sourced as well.
• The authors publish PubChemQA and UniProtQA datasets to facilitate multimodal alignment research.