[Paper Review] Advancing Multimodal Medical Capabilities of Gemini

Lin Yang, Shawn Xu+45 more|arXiv (Cornell University)|May 6, 2024

Biomedical and Engineering Education28 citations

TL;DR

This work introduces Med-Gemini, a family of medically tuned multimodal models built from Gemini, fine-tuned on 2D/3D radiology, histopathology, ophthalmology, dermatology, and genomics data, achieving state-of-the-art results on several medical tasks and showcasing 2D/3D report generation and polygenic risk prediction.

ABSTRACT

Many clinical tasks require an understanding of specialized data, such as medical images and genomics, which is not typically found in general-purpose large multimodal models. Building upon Gemini's multimodal models, we develop several models within the new Med-Gemini family that inherit core capabilities of Gemini and are optimized for medical use via fine-tuning with 2D and 3D radiology, histopathology, ophthalmology, dermatology and genomic data. Med-Gemini-2D sets a new standard for AI-based chest X-ray (CXR) report generation based on expert evaluation, exceeding previous best results across two separate datasets by an absolute margin of 1% and 12%, where 57% and 96% of AI reports on normal cases, and 43% and 65% on abnormal cases, are evaluated as "equivalent or better" than the original radiologists' reports. We demonstrate the first ever large multimodal model-based report generation for 3D computed tomography (CT) volumes using Med-Gemini-3D, with 53% of AI reports considered clinically acceptable, although additional research is needed to meet expert radiologist reporting quality. Beyond report generation, Med-Gemini-2D surpasses the previous best performance in CXR visual question answering (VQA) and performs well in CXR classification and radiology VQA, exceeding SoTA or baselines on 17 of 20 tasks. In histopathology, ophthalmology, and dermatology image classification, Med-Gemini-2D surpasses baselines across 18 out of 20 tasks and approaches task-specific model performance. Beyond imaging, Med-Gemini-Polygenic outperforms the standard linear polygenic risk score-based approach for disease risk prediction and generalizes to genetically correlated diseases for which it has never been trained. Although further development and evaluation are necessary in the safety-critical medical domain, our results highlight the potential of Med-Gemini across a wide range of medical tasks.

Motivation & Objective

• Extend Gemini with medically focused fine-tuning to handle diverse clinical modalities (2D/3D imaging, genomics, and text) while preserving general multimodal reasoning.
• Benchmark Med-Gemini across radiology, pathology, ophthalmology, dermatology, and genomics to assess clinical utility and generalization.
• Evaluate 2D/3D radiology report generation, VQA, image classification, and genomic risk prediction against open benchmarks and expert judgments.
• Demonstrate potential for Med-Gemini to approach or surpass task-specific models in several clinically relevant tasks.

Proposed method

• Fine-tune Med-Gemini variants (2D, 3D, Polygenic) from Gemini 1.5 Pro using a dataset of ~7 million samples across 3.7 million medical images/cases.
• Use three custom vision encoders for 2D, 3D, and genomics data to improve multimodal interpretation.
• Frame fine-tuning as captioning or VQA tasks, with expert evaluation for critical radiology report generation tasks.
• Preprocess radiology data into standardized 2D slices and 3D volumes; convert genomic features (PRSs) into image-like representations for Polygenic tasks.
• Evaluate on 22 datasets across 5 tasks and 6 modalities, including open and held-out out-of-distribution datasets.

Experimental results

Research questions

• RQ1 Can Med-Gemini achieve expert-quality radiology report generation for 2D chest X-rays and 3D CT volumes?
• RQ2 How well does Med-Gemini perform on radiology VQA, image classification, and pathology/ophthalmology/dermatology tasks compared to baselines or SoTA?
• RQ3 Does Med-Gemini-Polygenic provide superior disease risk prediction using genomic embeddings versus standard PRS approaches and generalize to correlated diseases?
• RQ4 What is the generalization capability of Med-Gemini across diverse medical data modalities and held-out datasets?

Key findings

• Med-Gemini-2D sets a new standard for AI-based chest X-ray report generation, exceeding previous best results by an absolute margin of 1% and 12% on two datasets; AI reports on normal cases were judged equivalent or better than radiologists in 57% and 96% of cases, and 43% and 65% for abnormal cases.
• Med-Gemini-3D reports on 3D CT volumes achieved 53% clinical acceptability, marking the first large multimodal model-based 3D CT report generation results (requiring further improvements).
• Med-Gemini-2D surpasses prior best in chest X-ray VQA and performs strongly in CXR classification and radiology VQA, exceeding SoTA or baselines on 17 of 20 tasks.
• In histopathology, ophthalmology, and dermatology image classification, Med-Gemini-2D outperformed baselines on 18 of 20 tasks and approaches task-specific model performance.
• Med-Gemini-Polygenic outperforms standard linear PRS approaches for disease risk prediction and generalizes to genetically correlated diseases not seen during training.