[Paper Review] MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine

Yunfei Xie, Ce Zhou+9 more|arXiv (Cornell University)|Aug 6, 2024

Biomedical Text Mining and Ontologies7 citations

TL;DR

Introduces MedTrinity-25M, a large-scale, multimodal medical dataset with over 25 million image-ROI-description triplets and multigranular annotations across 10 modalities and 65+ diseases, created without paired text via an automated pipeline using expert grounding, RAG, and MLLMs.

ABSTRACT

This paper introduces MedTrinity-25M, a comprehensive, large-scale multimodal dataset for medicine, covering over 25 million images across 10 modalities with multigranular annotations for more than 65 diseases. These multigranular annotations encompass both global information, such as modality and organ detection, and local information like ROI analysis, lesion texture, and region-wise correlations. Unlike the existing multimodal datasets, which are limited by the availability of image-text pairs, we have developed the first automated pipeline that scales up multimodal data by generating multigranular visual and textual annotations in the form of image-ROI-description triplets without the need for any paired text descriptions. Specifically, data from over 30 different sources have been collected, preprocessed, and grounded using domain-specific expert models to identify ROIs related to abnormal regions. We then build a comprehensive knowledge base and prompt multimodal large language models to perform retrieval-augmented generation with the identified ROIs as guidance, resulting in multigranular textual descriptions. Compared to existing datasets, MedTrinity-25M provides the most enriched annotations, supporting a comprehensive range of multimodal tasks such as captioning and report generation, as well as vision-centric tasks like classification and segmentation. We propose LLaVA-Tri by pretraining LLaVA on MedTrinity-25M, achieving state-of-the-art performance on VQA-RAD, SLAKE, and PathVQA, surpassing representative SOTA multimodal large language models. Furthermore, MedTrinity-25M can also be utilized to support large-scale pre-training of multimodal medical AI models, contributing to the development of future foundation models in the medical domain. We will make our dataset available.

Motivation & Objective

• Motivate the need for multigranular medical visual descriptions that link local ROIs to global disease context.
• Provide a scalable, automated pipeline to generate rich image-ROI-description annotations from unpaired medical images.
• Enable broad multimodal tasks (captioning, report generation, classification, segmentation) and large-scale pretraining for medical AI models.

Proposed method

• Create image-ROI-description triplets from 25M+ samples across 10 modalities and 65+ diseases sourced from 90+ online resources.
• Use expert grounding models to locate ROIs and convert masks to bounding boxes when needed.
• Construct a medical knowledge base from PubMed, StatPearls, and textbooks and index with Faiss for retrieval-augmented generation.
• Prompt a medical LLM stack (GPT-4V subset → LLaVA-Med Captioner, enhanced with LLAMA3 and multi-scale features) to generate multigranular textual descriptions guided by coarse captions, ROIs, and retrieved knowledge.
• Fine-tune LLaVA-Med++ on MedTrinity-25M to produce the full 25M image-ROI-description triplets.

Experimental results

Research questions

• RQ1 Can unpaired medical images be converted into high-quality, multigranular image-ROI-description triplets using automated grounding, retrieval-augmented generation, and MLLMs?
• RQ2 Do multigranular annotations improve performance on downstream multimodal medical tasks such as VQA and report generation compared to existing datasets?
• RQ3 Does pretraining on MedTrinity-25M yield superior results for medical VQA benchmarks relative to models trained without it?

Key findings

• MedTrinity-25M comprises over 25 million image-ROI-description triplets from 90+ sources, spanning 10 modalities and 65+ diseases.
• The dataset provides multigranular textual descriptions including modality, organ, ROI location, inter-regional relationships, and ROI-level bounding boxes or masks.
• GPT-4V alignment assessments on SLAKE and MIMIC-CXR indicate high alignment with human annotations (8.2/10 and 8.9/10 overall in the five-criteria rubric).
• LLaVA-Med++ pretrained on MedTrinity-25M achieves state-of-the-art performance on VQA-RAD and PathVQA and ranks third on SLAKE among evaluated baselines when pretrained on the dataset.
• Pretraining on MedTrinity-25M yielded improvements of about 10.75% on VQA-RAD, 6.1% on SLAKE, and 13.25% on PathVQA for downstream VQA benchmarks compared to not using the dataset.