[Paper Review] PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathology

Yuxuan Sun, Hao Wu+12 more|arXiv (Cornell University)|Jan 29, 2024

Biomedical Text Mining and OntologiesBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

PathMMU is a large-scale, expert-validated multimodal benchmark comprising 33,428 pathology Q&A pairs and 24,067 images, constructed using GPT-4V and validated by seven pathologists. It reveals that even top LMMs like GPT-4V achieve only 49.8% zero-shot accuracy—well below human experts’ 71.8%—but fine-tuned open-source models surpass GPT-4V, highlighting PathMMU’s role in advancing pathology-specific LMM development.

ABSTRACT

The emergence of large multimodal models has unlocked remarkable potential in AI, particularly in pathology. However, the lack of specialized, high-quality benchmark impeded their development and precise evaluation. To address this, we introduce PathMMU, the largest and highest-quality expert-validated pathology benchmark for Large Multimodal Models (LMMs). It comprises 33,428 multimodal multi-choice questions and 24,067 images from various sources, each accompanied by an explanation for the correct answer. The construction of PathMMU harnesses GPT-4V's advanced capabilities, utilizing over 30,000 image-caption pairs to enrich captions and generate corresponding Q&As in a cascading process. Significantly, to maximize PathMMU's authority, we invite seven pathologists to scrutinize each question under strict standards in PathMMU's validation and test sets, while simultaneously setting an expert-level performance benchmark for PathMMU. We conduct extensive evaluations, including zero-shot assessments of 14 open-sourced and 4 closed-sourced LMMs and their robustness to image corruption. We also fine-tune representative LMMs to assess their adaptability to PathMMU. The empirical findings indicate that advanced LMMs struggle with the challenging PathMMU benchmark, with the top-performing LMM, GPT-4V, achieving only a 49.8% zero-shot performance, significantly lower than the 71.8% demonstrated by human pathologists. After fine-tuning, significantly smaller open-sourced LMMs can outperform GPT-4V but still fall short of the expertise shown by pathologists. We hope that the PathMMU will offer valuable insights and foster the development of more specialized, next-generation LMMs for pathology.

Motivation & Objective

• To address the lack of high-quality, expert-validated benchmarks for evaluating large multimodal models (LMMs) in pathology.
• To create a comprehensive, diverse, and explainable multimodal dataset that reflects real-world diagnostic challenges.
• To establish a human expert performance benchmark to quantify the gap between LMMs and clinical expertise.
• To evaluate the robustness and adaptability of both open-source and closed-source LMMs on a pathology-specific, expert-curated benchmark.
• To guide future development of specialized LMMs by identifying key limitations such as visual neglect and shortcut learning.

Proposed method

• Leveraged GPT-4V to generate over 30,000 image-caption pairs and cascade them into multimodal Q&A pairs using a hierarchical prompting strategy.
• Collected data from diverse sources including PubMed, pathology atlases, social media, educational videos, and classification datasets to ensure broad coverage of organ systems and disease types.
• Employed a rigorous two-stage curation process: initial AI generation followed by expert validation by seven board-certified pathologists on test and validation sets.
• Established a human expert benchmark by having pathologists answer the same questions, enabling direct comparison with LMM performance.
• Conducted zero-shot evaluations and fine-tuning experiments on 14 open-source and 4 closed-source LMMs using two fine-tuning strategies: direct answer generation and answer-with-explanation generation.
• Evaluated model robustness to image corruption and analyzed performance across multiple data subsets (e.g., PubMed, SocialPath, Atlas) to assess domain generalization.

Experimental results

Research questions

• RQ1 How do state-of-the-art LMMs perform on a comprehensive, expert-validated pathology benchmark compared to human pathologists?
• RQ2 Can fine-tuning on PathMMU significantly improve LMM performance, and do models fine-tuned to generate explanations outperform those that generate answers directly?
• RQ3 To what extent do LMMs rely on textual cues versus visual features when analyzing pathology images, and how does this affect their accuracy?
• RQ4 What are the key failure modes of current LMMs in pathology, such as visual neglect or shortcut learning, and how can they be mitigated?
• RQ5 How robust are LMMs to image corruptions, and does performance vary across different data sources within PathMMU?

Key findings

• GPT-4V, the top-performing LMM, achieved only 49.8% zero-shot accuracy on PathMMU, significantly below the 71.8% accuracy of human pathologists.
• After fine-tuning, smaller open-source LMMs such as InstructBLIP-FLAN-T5-XL and InstructBLIP-FLAN-T5-XXL surpassed GPT-4V, achieving up to 61.4% accuracy with answer-and-explanation fine-tuning.
• Fine-tuning for direct answer generation yielded better performance than fine-tuning for answer-with-explanation generation, with the latter showing slight performance drops of 0.3% and 2.3% respectively.
• The most significant performance gains came from fine-tuning, with InstructBLIP-FLAN-T5-XL improving by 21.5 percentage points when fine-tuned to generate answers directly.
• Models exhibited poor robustness to image corruption and often neglected salient visual features, indicating a critical reliance on textual cues over visual understanding.
• Performance varied across data subsets, with models performing worst on the PubMed and Educational Content subsets, suggesting domain-specific generalization challenges.