[Paper Review] GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI

Pengcheng Chen, Ye Jin+16 more|arXiv (Cornell University)|Aug 6, 2024

Artificial Intelligence in Healthcare8 citations

TL;DR

GMAI-MMBench is a comprehensive multimodal medical AI benchmark with 285 datasets across 39 modalities to evaluate LVLMs, reporting that even top models like GPT-4o achieve ~52% accuracy and highlighting key insufficiencies.

ABSTRACT

Large Vision-Language Models (LVLMs) are capable of handling diverse data types such as imaging, text, and physiological signals, and can be applied in various fields. In the medical field, LVLMs have a high potential to offer substantial assistance for diagnosis and treatment. Before that, it is crucial to develop benchmarks to evaluate LVLMs' effectiveness in various medical applications. Current benchmarks are often built upon specific academic literature, mainly focusing on a single domain, and lacking varying perceptual granularities. Thus, they face specific challenges, including limited clinical relevance, incomplete evaluations, and insufficient guidance for interactive LVLMs. To address these limitations, we developed the GMAI-MMBench, the most comprehensive general medical AI benchmark with well-categorized data structure and multi-perceptual granularity to date. It is constructed from 284 datasets across 38 medical image modalities, 18 clinical-related tasks, 18 departments, and 4 perceptual granularities in a Visual Question Answering (VQA) format. Additionally, we implemented a lexical tree structure that allows users to customize evaluation tasks, accommodating various assessment needs and substantially supporting medical AI research and applications. We evaluated 50 LVLMs, and the results show that even the advanced GPT-4o only achieves an accuracy of 53.96%, indicating significant room for improvement. Moreover, we identified five key insufficiencies in current cutting-edge LVLMs that need to be addressed to advance the development of better medical applications. We believe that GMAI-MMBench will stimulate the community to build the next generation of LVLMs toward GMAI.

Motivation & Objective

• Establish a comprehensive, clinically relevant multimodal benchmark for general medical AI (GMAI) applicable across modalities, tasks, and departments.
• Provide a well-structured lexical tree to enable highly customizable evaluation tailored to specific clinical needs.
• Assess a wide range of LVLMs (medical-specific, open-source, and proprietary) to identify strengths, weaknesses, and areas for improvement in medical AI.
• Offer insights into perceptual granularity requirements (image, region-level) for interactive LVLMs in real-world clinical scenarios.

Proposed method

• Assemble 285 high-quality datasets from public sources and hospitals spanning 39 modalities and 18 clinical VQA tasks across 18 departments.
• Standardize images and labels using SA-Med2D-20M protocol and MeSH terms to ensure consistency and reduce ambiguity.
• Construct a lexical tree with 18 clinical VQA tasks, 18 departments, and 4 perceptual granularities to enable customized evaluation.
• Generate 26K QA pairs with modality, task cue, and granularity annotations; perform manual validation and selection for quality and balance.
• Evaluate 44 LVLMs (open-source and medical-specific) and 6 proprietary models in a zero-shot setting using the VLMEvalKit and Multi-Modality-Arena frameworks.

Experimental results

Research questions

• RQ1 What is the performance of current LVLMs across a broad, clinically realistic set of medical modalities and tasks?
• RQ2 How do LVLMs perform when evaluated with varying perceptual granularities (image, box, mask, contour) and interactive cues?
• RQ3 Which factors most limit LVLMs in medical diagnosis and reasoning, and how do medical-domain models compare to general-purpose models?
• RQ4 Can a well-categorized lexical-tree benchmark support customized evaluation for diverse clinical departments and demands?

Key findings

• GPT-4o achieves 52.24% accuracy on GMAI-MMBench, indicating substantial room for improvement in clinical tasks.
• Open-source LVLMs like MedDr and DeepSeek-VL-7B reach around 41% accuracy, showing competitive performance relative to some proprietary models.
• Most medical-specific LVLMs struggle to reach mid-range performance (~30%), with MedDr achieving the best among medical-specific models in some settings.
• Box-level perception consistently yields the lowest accuracy compared to image-level or other granularities, highlighting challenges for region-based reasoning.
• Major bottlenecks include perceptual errors, limited medical domain knowledge, irrelevant responses, and safety/rejection of answers.