[Paper Review] MC-CoT: A Modular Collaborative CoT Framework for Zero-shot Medical-VQA with LLM and MLLM Integration

Wei Lai, Wenkai Wang+6 more|arXiv (Cornell University)|Oct 6, 2024

Radiomics and Machine Learning in Medical ImagingMedicine3 citations

TL;DR

MC-CoT proposes a modular, collaborative Chain-of-Thought framework that enhances zero-shot medical visual question answering (Med-VQA) by integrating large language models (LLMs) and multimodal large language models (MLLMs). It uses specialized modules—Anatomy, Pathology, and Radiology—to guide MLLMs with domain-specific reasoning and image captioning, achieving superior accuracy and recall over standalone MLLMs and existing CoT methods on PATH-VQA, VQA-RAD, and SLAKE.

ABSTRACT

In recent advancements, multimodal large language models (MLLMs) have been fine-tuned on specific medical image datasets to address medical visual question answering (Med-VQA) tasks. However, this common approach of task-specific fine-tuning is costly and necessitates separate models for each downstream task, limiting the exploration of zero-shot capabilities. In this paper, we introduce MC-CoT, a modular cross-modal collaboration Chain-of-Thought (CoT) framework designed to enhance the zero-shot performance of MLLMs in Med-VQA by leveraging large language models (LLMs). MC-CoT improves reasoning and information extraction by integrating medical knowledge and task-specific guidance, where LLM provides various complex medical reasoning chains and MLLM provides various observations of medical images based on instructions of the LLM. Our experiments on datasets such as SLAKE, VQA-RAD, and PATH-VQA show that MC-CoT surpasses standalone MLLMs and various multimodality CoT frameworks in recall rate and accuracy. These findings highlight the importance of incorporating background information and detailed guidance in addressing complex zero-shot Med-VQA tasks.

Motivation & Objective

• To address the high cost and limited scalability of task-specific fine-tuning for Med-VQA by enabling zero-shot performance in MLLMs.
• To enhance multimodal reasoning in Med-VQA by integrating medical knowledge and structured guidance from LLMs.
• To design a modular, reusable framework that dynamically activates specialized reasoning modules based on question requirements.
• To evaluate the framework’s effectiveness across diverse MLLM and LLM combinations and multiple Med-VQA benchmarks.
• To investigate the impact of key components such as image captioning, LLM-guided reasoning, and answer summarization on performance.

Proposed method

• MC-CoT employs three pre-designed image feature extraction modules—Anatomy, Pathology, and Radiology—each tailored to specific aspects of medical image analysis.
• The LLM first analyzes the input question, providing contextual medical knowledge and strategic instructions to guide the MLLM’s reasoning process.
• The MLLM generates image observations based on LLM-provided instructions, extracting modality-specific features from medical images.
• The LLM synthesizes outputs from activated modules to produce a cohesive, comprehensive answer, ensuring reasoning traceability.
• Image captioning is used as a preprocessing step to improve guidance relevance and narrow the problem scope before MLLM inference.
• The framework supports dynamic module activation based on question content, enabling flexible, task-adaptive reasoning.

Experimental results

Research questions

• RQ1 Can a modular, collaborative CoT framework significantly improve the zero-shot performance of MLLMs in Med-VQA without task-specific fine-tuning?
• RQ2 How does integrating domain-specific LLM-guided reasoning with MLLM image understanding affect answer accuracy and recall?
• RQ3 Which specialized module (Anatomy, Pathology, Radiology) contributes most to performance on different types of Med-VQA questions?
• RQ4 To what extent does image captioning improve the quality of LLM-guided reasoning in the MC-CoT framework?
• RQ5 How does MC-CoT compare to existing CoT frameworks like MMCoT and DDCoT in terms of reasoning rigor and clinical relevance?

Key findings

• MC-CoT outperforms standalone MLLMs and multiple CoT frameworks (e.g., MMCoT, DDCoT) in both answer accuracy and recall rate across PATH-VQA, VQA-RAD, and SLAKE datasets.
• The radiology module contributes the most to answer accuracy, indicating that modality-specific imaging features are critical for diagnostic reasoning.
• The anatomy module achieves the highest average recall, highlighting the importance of identifying relevant anatomical structures in question understanding.
• Image captioning significantly improves guidance relevance, enabling the LLM to generate more precise and context-aware instructions for the MLLM.
• MC-CoT’s reasoning process is more rigorous than baseline CoT methods, as it explicitly verifies anatomical structures and avoids over-reliance on commonsense reasoning.
• The framework maintains strong performance across various LLM and MLLM combinations, demonstrating broad generalizability and robustness.