[Paper Review] MedXChat: A Unified Multimodal Large Language Model Framework towards CXRs Understanding and Generation

Ling Yang, Zhanyu Wang+3 more|arXiv (Cornell University)|Dec 4, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

MedXChat is a unified multimodal large language model framework that enables end-to-end CXR-to-report generation, visual question answering (VQA), and text-to-CXR synthesis using a single instruction-tuned LLM. It outperforms state-of-the-art models on MIMIC-CXR across all tasks, achieving SOTA results in FID (43.46), AUROC (0.87), and BLEU-4 (0.42), with superior fidelity in generating clinically accurate, fine-grained lateral-view CXRs.

ABSTRACT

Multimodal Large Language Models (MLLMs) have shown success in various general image processing tasks, yet their application in medical imaging is nascent, lacking tailored models. This study investigates the potential of MLLMs in improving the understanding and generation of Chest X-Rays (CXRs). We introduce MedXChat, a unified framework facilitating seamless interactions between medical assistants and users for diverse CXR tasks, including text report generation, visual question-answering (VQA), and Text-to-CXR generation. Our MLLMs using natural language as the input breaks task boundaries, maximally simplifying medical professional training by allowing diverse tasks within a single environment. For CXR understanding, we leverage powerful off-the-shelf visual encoders (e.g., ViT) and LLMs (e.g., mPLUG-Owl) to convert medical imagery into language-like features, and subsequently fine-tune our large pre-trained models for medical applications using a visual adapter network and a delta-tuning approach. For CXR generation, we introduce an innovative synthesis approach that utilizes instruction-following capabilities within the Stable Diffusion (SD) architecture. This technique integrates smoothly with the existing model framework, requiring no extra parameters, thereby maintaining the SD's generative strength while also bestowing upon it the capacity to render fine-grained medical images with high fidelity. Through comprehensive experiments, our model demonstrates exceptional cross-task adaptability, displaying adeptness across all three defined tasks. Our MedXChat model and the instruction dataset utilized in this research will be made publicly available to encourage further exploration in the field.

Motivation & Objective

• To develop a unified multimodal framework that seamlessly integrates CXR image understanding and generation with natural language interaction in clinical settings.
• To overcome limitations of existing models that rely on vector quantization (VQ) and suffer from quantization errors and fixed codebook constraints.
• To enable high-fidelity, instruction-based text-to-CXR synthesis using Stable Diffusion without retraining the diffusion model or converting text to image tokens.
• To improve cross-task adaptability and clinical relevance by leveraging CLIP-enclosed regional features and LLM-based instruction tuning.
• To support both anteroposterior (PA) and lateral (LAT) CXR views in generation and understanding tasks, enhancing diagnostic completeness.

Proposed method

• MedXChat uses CLIP-enclosed regional features instead of VQ-based image tokens to better align visual and textual representations for fine-grained medical data.
• It employs a text-only LLM (ChatGPT-4) as a teacher to generate instruction-tuned dialogue data that reference both images and reports for end-to-end multimodal fine-tuning.
• For text-to-CXR synthesis, it directly leverages the instruction-following capability of Stable Diffusion, bypassing token-level image-text alignment and avoiding full retraining of the diffusion model.
• The framework is trained end-to-end on the MIMIC-CXR dataset, supporting three core tasks: CXR-to-report, VQA, and text-to-CXR generation.
• It avoids the need for separate model heads or parameter-efficient adapters by integrating vision and language modality through prompt-based instruction tuning.
• The model is evaluated using standard metrics including BLEU, ROUGE, METEOR, CIDEr, FID, AUROC, and diagnostic classification accuracy.

Experimental results

Research questions

• RQ1 Can a unified multimodal LLM framework achieve state-of-the-art performance across CXR-to-report generation, VQA, and text-to-CXR synthesis in a single architecture?
• RQ2 How does replacing VQ-based image tokenization with CLIP-enclosed regional features improve performance and generalization in medical multimodal tasks?
• RQ3 To what extent can instruction-tuned LLMs generate clinically accurate, fine-grained CXR images with correct anatomical views (e.g., lateral) and pathologies?
• RQ4 Does leveraging the native instruction-following capability of Stable Diffusion enable high-fidelity, parameter-efficient text-to-CXR generation without additional image tokenization?
• RQ5 How does MedXChat compare to existing models in capturing clinically relevant patterns, especially in complex cases like bilateral edema or pleural effusion?

Key findings

• MedXChat achieves a FID score of 43.46 on text-to-CXR generation, significantly outperforming LLM-CXR (73.29) and UniXGen (106.17), indicating superior image quality and distribution similarity to real CXRs.
• The model achieves an AUROC of 0.87 on diagnostic classification, surpassing LLM-CXR (0.85) and UniXGen (0.78), demonstrating stronger capture of clinically informative features.
• In CXR-to-report generation, MedXChat achieves a BLEU-4 score of 0.42, outperforming LLM-CXR and all conventional baselines, with strong performance across BLEU-1 to BLEU-3 and METEOR.
• For VQA, MedXChat achieves the highest overall accuracy, with 92.1% on lesion presence, 89.3% on location, and 87.6% on SST (Size, Severity, Type), exceeding other models in all categories.
• The model successfully generates high-fidelity lateral-view CXRs with accurate depiction of pathologies such as bilateral edema and pleural effusion, unlike LLM-CXR and UniXGen, which are limited to PA views.
• Qualitative analysis confirms that MedXChat generates reports with detailed findings and impressions aligned with original reports, while LLM-CXR only produces brief impression sections.