[Paper Review] M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models

Fan Bai, Yuxin Du+3 more|arXiv (Cornell University)|Mar 31, 2024

Medical Imaging and Analysis15 citations

TL;DR

The paper builds M3D-Data, a large-scale 3D medical multi-modal dataset, introduces M3D-LaMed, a versatile 3D MLLM, and proposes M3D-Bench for eight tasks, achieving strong results across 3D image-text retrieval, report generation, VQA, positioning, and segmentation.

ABSTRACT

Medical image analysis is essential to clinical diagnosis and treatment, which is increasingly supported by multi-modal large language models (MLLMs). However, previous research has primarily focused on 2D medical images, leaving 3D images under-explored, despite their richer spatial information. This paper aims to advance 3D medical image analysis with MLLMs. To this end, we present a large-scale 3D multi-modal medical dataset, M3D-Data, comprising 120K image-text pairs and 662K instruction-response pairs specifically tailored for various 3D medical tasks, such as image-text retrieval, report generation, visual question answering, positioning, and segmentation. Additionally, we propose M3D-LaMed, a versatile multi-modal large language model for 3D medical image analysis. Furthermore, we introduce a new 3D multi-modal medical benchmark, M3D-Bench, which facilitates automatic evaluation across eight tasks. Through comprehensive evaluation, our method proves to be a robust model for 3D medical image analysis, outperforming existing solutions. All code, data, and models are publicly available at: https://github.com/BAAI-DCAI/M3D.

Motivation & Objective

• Motivate advancing 3D medical image analysis with multi-modal large language models (MLLMs).
• Create a large-scale 3D medical dataset and corresponding benchmark to enable robust multi-modal tasks.
• Develop a versatile 3D MLLM capable of retrieval, report generation, VQA, positioning, and segmentation.
• Enable automatic evaluation with an LLM-based benchmark

Proposed method

• Pre-train a 3D vision encoder from scratch using a CLIP-like cross-modal loss on M3D-Cap.
• Introduce a 3D spatial pooling perceiver to reduce tokens and align embeddings with the LLM.
• Integrate a pre-trained LLaMA-2-7B LLM via end-to-end fine-tuning with a 3D perceiver.
• Incorporate a promptable segmentation module (SegVol) to enable 3D vision-language segmentation.
• Utilize LoRA for parameter-efficient fine-tuning of the LLM to retain prior knowledge.
• Evaluate on eight tasks through M3D-Bench, including image-text retrieval, report generation, VQA, positioning, and segmentation.

Figure 1 : The generation pipelines for M3D-Data. (a) In the VQA data generation pipeline, we employ LLM to generate five types of questions from medical reports using a prompt-based method. Subsequently, we eliminate dirty data through self-filtering and check the test set by LLM and experts, achie

Experimental results

Research questions

• RQ1 Can 3D medical images be effectively analyzed by multi-modal LLMs trained with a 3D vision encoder and a 3D perceiver?
• RQ2 How does a large-scale 3D medical multi-modal dataset (M3D-Data) support diverse tasks (retrieval, RG, VQA, positioning, segmentation)?
• RQ3 What is the performance of M3D-LaMed across eight tasks compared to existing baselines?
• RQ4 Does a promptable segmentation module enable referring expression segmentation in 3D medical images?
• RQ5 Can LLMS be evaluated automatically with an LLM-based benchmark (M3D-Bench) for 3D tasks?

Key findings

• M3D-Data contains 120K 3D image-text pairs and 662K instruction-response pairs, supporting 8 tasks.
• M3D-LaMed outperforms prior 3D MLLMs on a wide range of tasks including retrieval, VQA, positioning, and segmentation.
• 3D image-text retrieval shows substantial gains over a 2D baseline (PMC-CLIP) across all test settings, with large improvements in IR and TR metrics.
• Report generation achieved higher BLEU, ROUGE, METEOR, and BERT-Score, as well as stronger LLN-based evaluations, compared to RadFM.
• VQA (both closed- and open-ended) and positioning tasks show strong gains, with ablations highlighting the importance of vision pre-training, spatial pooling, MLP design, and unlocked vision encoder.
• Segmentation tasks (semantic and referring expression) surpass prior methods and enable referring expression segmentation in 3D.

Figure 2 : The data statistics of M3D-VQA on five question types. What, which, and where are 3 typical questions. Samples of 5 topics are displayed in word clouds.