[Paper Review] BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks

Kai Zhang, Rong Zhou+21 more|arXiv (Cornell University)|May 26, 2023

Artificial Intelligence in Healthcare and EducationMedicine84 references51 citations

TL;DR

BiomedGPT is an open-source generalist vision-language model for diverse biomedical tasks, achieving SOTA on 15 of 25 experiments across 26 datasets with 182M parameters and enabling zero-shot transfer.

ABSTRACT

Traditional biomedical artificial intelligence (AI) models, designed for specific tasks or modalities, often exhibit limited flexibility in real-world deployment and struggle to utilize holistic information. Generalist AI holds the potential to address these limitations due to its versatility in interpreting different data types and generating tailored outputs for diverse needs. However, existing biomedical generalist AI solutions are typically heavyweight and closed source to researchers, practitioners, and patients. Here, we propose BiomedGPT, the first open-source and lightweight vision-language foundation model, designed as a generalist capable of performing various biomedical tasks. BiomedGPT achieved state-of-the-art results in 16 out of 25 experiments while maintaining a computing-friendly model scale. We also conducted human evaluations to assess the capabilities of BiomedGPT in radiology visual question answering, report generation, and summarization. BiomedGPT exhibits robust prediction ability with a low error rate of 3.8% in question answering, satisfactory performance with an error rate of 8.3% in writing complex radiology reports, and competitive summarization ability with a nearly equivalent preference score to human experts. Our method demonstrates that effective training with diverse data can lead to more practical biomedical AI for improving diagnosis and workflow efficiency.

Motivation & Objective

• Motivate a unified, generalist AI model for diverse biomedical tasks spanning imaging and text.
• Pretrain a single model on diverse multimodal biomedical data to learn universal representations.
• Fine-tune with task-specific instructions to handle vision-only, language, and multimodal tasks.
• Evaluate performance against large closed models and demonstrate real-world deployment potential.
• Promote transparency by open-sourcing model checkpoints, training data processing, and code.

Proposed method

• Use a sequence-to-sequence BiomedGPT with a BERT-style encoder and GPT-style decoder.
• Pretrain on 14 freely-available biomedical datasets covering imaging, text, and multimodal data (352,567 images; ~183M text sentences; 46,408 object-label pairs; ~271,803 image-text pairs).
• Employ five pretraining tasks: image-only masked image modeling, text-only masked language modeling, image captioning, and visual question answering (VQA).
• Fine-tune with instruction-based prompts for 25 downstream datasets across five medical AI tasks (classification, language understanding, summarization, captioning, VQA).
• Explore three model scales (BiomedGPT-S, -M, -B) to study scale effects; initialize from OFA to inject general and medical knowledge.

Figure 1: The overview of BiomedGPT: workflow, performance and pretraining datasets. (a) Graphical illustration of how BiomedGPT handle multi-modal inputs and perform diverse downstream tasks. The expected form of output for each task is determined by feeding the specific instruction to the model. (

Experimental results

Research questions

• RQ1 Can a single unified model effectively handle multiple biomedical modalities and tasks across vision and language?
• RQ2 Does diverse, multi-task pretraining improve generalization and downstream performance in biomedicine?
• RQ3 How does model size influence performance on vision, language understanding, and multimodal biomedical tasks?
• RQ4 What is the zero-shot VQA capability of a unified biomedical model compared to large multimodal systems?
• RQ5 How does BiomedGPT perform in real-world deployment and radiologist evaluations?

Key findings

• BiomedGPT achieves state-of-the-art results in 15 out of 25 experiments across 26 datasets.
• The 182M-parameter BiomedGPT-B model outperforms Med-PaLM M (12B) on VQA-RAD and SLAKE by large margins and nearly matches PathVQA.
• On image captioning, BiomedGPT outperforms SOTA on PEIR GROSS with substantial gains in CIDEr; CIDEr improvements are highlighted across datasets.
• In medical image classification across 11 MedMNIST-derived datasets, BiomedGPT outperforms several baselines, with notable gains on high-resolution datasets as model size increases.
• BiomedGPT demonstrates strong zero-shot VQA capability, generating free-form answers without additional task-specific components, achieving competitive performance with GPT-4V in a subset of evaluations.

Figure 2: BiomedGPT performs fine-tuning for vision-language downstream tasks. (a) Graphical illustration of inference workflow of BiomedGPT for VQA task. Our model can discrete both visual and linguistic inputs from questions into tokens and generated the corresponding answers. (b) VQA performance