[Paper Review] MedPrompt: Cross-Modal Prompting for Multi-Task Medical Image Translation

Xuhang Chen, Chi‐Man Pun+1 more|arXiv (Cornell University)|Oct 4, 2023

Multimodal Machine Learning ApplicationsComputer Science3 citations

TL;DR

MedPrompt proposes a cross-modal prompting framework for multi-task medical image translation using a self-adaptive prompt block with prompt extraction and fusion mechanisms, enhanced by a Transformer encoder for global feature learning. It achieves state-of-the-art performance across five datasets and four modality pairs, demonstrating superior visual quality and generalization with single-stage training.

ABSTRACT

Cross-modal medical image translation is an essential task for synthesizing missing modality data for clinical diagnosis. However, current learning-based techniques have limitations in capturing cross-modal and global features, restricting their suitability to specific pairs of modalities. This lack of versatility undermines their practical usefulness, particularly considering that the missing modality may vary for different cases. In this study, we present MedPrompt, a multi-task framework that efficiently translates different modalities. Specifically, we propose the Self-adaptive Prompt Block, which dynamically guides the translation network towards distinct modalities. Within this framework, we introduce the Prompt Extraction Block and the Prompt Fusion Block to efficiently encode the cross-modal prompt. To enhance the extraction of global features across diverse modalities, we incorporate the Transformer model. Extensive experimental results involving five datasets and four pairs of modalities demonstrate that our proposed model achieves state-of-the-art visual quality and exhibits excellent generalization capability.

Motivation & Objective

• To address the limitations of existing medical image translation models in capturing cross-modal and global features across diverse modalities.
• To develop a versatile, multi-task framework that generalizes across different modality pairs without requiring retraining for each pair.
• To improve translation fidelity by integrating modality-specific prompts with global context modeling via Transformers.
• To reduce training complexity by enabling single-stage training across multiple modality translation tasks.

Proposed method

• The Self-adaptive Prompt Block dynamically guides the translation network using modality-specific prompts, enabling adaptation to different input-output modality pairs.
• The Prompt Extraction Block (PEB) generates modality-specific prompt weights by encoding unique characteristics of each input modality.
• The Prompt Fusion Block (PFB) adaptively fuses extracted prompts based on their relevance to the target modality, enhancing cross-modal feature alignment.
• A Transformer-based encoder is integrated to capture long-range spatial dependencies and global context, improving feature representation across diverse modalities.
• The framework uses a single-stage training process that jointly optimizes multiple modality translation tasks, improving training efficiency and generalization.
• The model is trained end-to-end with cycle-consistency loss and perceptual losses to ensure structural and texture fidelity in generated images.

Experimental results

Research questions

• RQ1 Can a unified framework effectively translate multiple pairs of medical imaging modalities without requiring separate models for each pair?
• RQ2 How can modality-specific cross-modal features be efficiently encoded and fused to improve translation accuracy?
• RQ3 To what extent does incorporating a Transformer-based architecture enhance global feature learning in multi-task medical image translation?
• RQ4 Does the proposed prompting mechanism improve generalization across diverse imaging modalities compared to conventional CNN-based or GAN-based methods?
• RQ5 Can a single-stage training process achieve state-of-the-art performance across multiple modality translation tasks?

Key findings

• MedPrompt achieves state-of-the-art performance across five datasets and four modality pairs, outperforming existing methods in both quantitative metrics and visual quality.
• The removal of the Prompt Extraction Block (PEB) or Prompt Fusion Block (PFB) leads to a drop of approximately 3 dB in PSNR, 0.7 dB in SSIM, and 2 dB in MAE, demonstrating their critical role in multi-task learning.
• Removing the Transformer block results in a 2 dB decrease in PSNR, 0.03 dB in SSIM, and 0.5 dB in MAE, confirming its contribution to global feature learning.
• Visual comparisons show that MedPrompt produces results closest to the ground truth in terms of shape, detail, and texture fidelity across all datasets and modality pairs.
• The method requires only a single training process for all modality translations, significantly improving training convenience compared to multi-stage or pair-specific approaches.
• The model exhibits strong generalization capability, successfully translating across diverse modalities such as MRI to CT, T1 to T2, and CT to CBCT without fine-tuning.