[Paper Review] Towards Unifying Medical Vision-and-Language Pre-training via Soft Prompts

Zhihong Chen, Shizhe Diao+3 more|arXiv (Cornell University)|Feb 17, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes PTUnifier, a unified medical vision-and-language pre-training framework that uses learnable visual and textual soft prompts to enable a single model to handle image-only, text-only, and image-text pair inputs. By integrating objectives from both fusion-encoder and dual-encoder paradigms—via a prompt pool for scalability—it achieves state-of-the-art performance across uni-modal, cross-modal, and multi-modal medical tasks without task-specific adaptation.

ABSTRACT

Medical vision-and-language pre-training (Med-VLP) has shown promising improvements on many downstream medical tasks owing to its applicability to extracting generic representations from medical images and texts. Practically, there exist two typical types, extit{i.e.}, the fusion-encoder type and the dual-encoder type, depending on whether a heavy fusion module is used. The former is superior at multi-modal tasks owing to the sufficient interaction between modalities; the latter is good at uni-modal and cross-modal tasks due to the single-modality encoding ability. To take advantage of these two types, we propose an effective yet straightforward scheme named PTUnifier to unify the two types. We first unify the input format by introducing visual and textual prompts, which serve as a feature bank that stores the most representative images/texts. By doing so, a single model could serve as a extit{foundation model} that processes various tasks adopting different input formats ( extit{i.e.}, image-only, text-only, and image-text-pair). Furthermore, we construct a prompt pool (instead of static ones) to improve diversity and scalability. Experimental results show that our approach achieves state-of-the-art results on a broad range of tasks, spanning uni-modal tasks ( extit{i.e.}, image/text classification and text summarization), cross-modal tasks ( extit{i.e.}, image-to-text generation and image-text/text-image retrieval), and multi-modal tasks ( extit{i.e.}, visual question answering), demonstrating the effectiveness of our approach. Note that the adoption of prompts is orthogonal to most existing Med-VLP approaches and could be a beneficial and complementary extension to these approaches.

Motivation & Objective

• Address the limitation of existing medical vision-and-language pre-training models that are either fusion-encoder or dual-encoder types, each excelling in specific task types but not both.
• Unify the capabilities of fusion-encoder and dual-encoder models into a single, compatible architecture to support diverse input formats (image-only, text-only, image-text pairs).
• Improve model scalability and representation diversity by introducing a prompt pool instead of static prompts for each modality.
• Enable the model to perform effectively across uni-modal (e.g., image/text classification), cross-modal (e.g., retrieval), and multi-modal (e.g., VQA) tasks without architectural or task-specific modifications.
• Provide a lightweight, architecture-agnostic extension that is orthogonal and complementary to existing Med-VLP methods, enhancing their applicability.

Proposed method

• Introduce visual and textual soft prompts as a unified input format, transforming image-only, text-only, and image-text inputs into a common representation space.
• Construct modality-specific prompt pools (visual and textual) to store representative embeddings, enabling dynamic selection of prompts based on input type.
• Pre-train the unified model using a combination of objectives: masked language modeling (MLM) and image-text matching (ITM) from fusion-encoders, and image-text contrast (ITC) from dual-encoders.
• Use the prompt pools to inject modality-specific inductive bias, allowing the same backbone model to adapt to different input types without architectural changes.
• Train on three large-scale medical datasets: ROCO, MedICaT, and MIMIC-CXR, to learn generic representations applicable to downstream tasks.
• Apply the unified model to various tasks without fine-tuning for specific tasks, relying on prompt-based adaptation and shared representation learning.

Figure 1: (a) Illustrations of two Med-VLP paradigms and their advantages (pointed by green arrows) and disadvantages (pointed by red arrows) in downstream tasks; (b) The overall architecture of our proposed approach, where the backbone models share the same parameters, and we duplicate them for ill

Experimental results

Research questions

• RQ1 Can a single medical vision-and-language pre-trained model effectively handle image-only, text-only, and image-text pair inputs using a unified framework?
• RQ2 Does combining objectives from both fusion-encoder and dual-encoder paradigms improve performance across diverse medical vision-and-language tasks?
• RQ3 Can a prompt pool mechanism enhance model scalability and representation diversity compared to static prompts?
• RQ4 To what extent does the proposed approach outperform task-specific models or existing Med-VLP methods across uni-modal, cross-modal, and multi-modal benchmarks?
• RQ5 Is the proposed method orthogonal and complementary to existing Med-VLP architectures, enabling broader applicability without architectural re-design?

Key findings

• PTUnifier achieves state-of-the-art performance on all evaluated tasks, including image classification (88.7% AUROC on CheXpert), text classification (89.0% AUROC), and multi-modal VQA (80.0% accuracy).
• The model pre-trained with all objectives (MLM, ITM, ITC) achieves the best performance across all benchmarks, demonstrating the complementary benefit of combining fusion-encoder and dual-encoder objectives.
• On the MIMIC-CXR dataset, PTUnifier achieves 84.6% accuracy in open-ended VQA and 78.3% in closed-ended VQA, outperforming prior methods like R2Gen and WGSum.
• Even with only 1% of training data, the model pre-trained with ITC objective achieves 10.7% accuracy in closed-ended VQA, showing strong few-shot generalization ability.
• The ITC objective significantly improves performance on vision-only and cross-modal tasks but does not improve text-only classification, suggesting asymmetric supervision between modalities.
• The ablation study confirms that combining MLM, ITM, and ITC objectives leads to the strongest transfer performance, validating the effectiveness of unifying both paradigms.