[Paper Review] Adaptive Latent Diffusion Model for 3D Medical Image to Image Translation: Multi-modal Magnetic Resonance Imaging Study

Jonghun Kim, Hyunjin Park|arXiv (Cornell University)|Nov 1, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes an Adaptive Latent Diffusion Model (ALDM) for 3D medical image-to-image translation that uses switchable spatially adaptive normalization (MS-SPADE) blocks to enable high-fidelity, one-to-many translation across multi-modal MRI without patch cropping. The model achieves state-of-the-art performance on BraTS and IXI datasets, surpassing existing methods in both quantitative metrics and generalization across diverse source-target modality pairs.

ABSTRACT

Multi-modal images play a crucial role in comprehensive evaluations in medical image analysis providing complementary information for identifying clinically important biomarkers. However, in clinical practice, acquiring multiple modalities can be challenging due to reasons such as scan cost, limited scan time, and safety considerations. In this paper, we propose a model based on the latent diffusion model (LDM) that leverages switchable blocks for image-to-image translation in 3D medical images without patch cropping. The 3D LDM combined with conditioning using the target modality allows generating high-quality target modality in 3D overcoming the shortcoming of the missing out-of-slice information in 2D generation methods. The switchable block, noted as multiple switchable spatially adaptive normalization (MS-SPADE), dynamically transforms source latents to the desired style of the target latents to help with the diffusion process. The MS-SPADE block allows us to have one single model to tackle many translation tasks of one source modality to various targets removing the need for many translation models for different scenarios. Our model exhibited successful image synthesis across different source-target modality scenarios and surpassed other models in quantitative evaluations tested on multi-modal brain magnetic resonance imaging datasets of four different modalities and an independent IXI dataset. Our model demonstrated successful image synthesis across various modalities even allowing for one-to-many modality translations. Furthermore, it outperformed other one-to-one translation models in quantitative evaluations.

Motivation & Objective

• To address the challenge of missing multi-modal MRI scans in clinical settings due to cost, time, or safety constraints.
• To overcome limitations of 2D GAN-based methods that fail to preserve 3D structural consistency across slices.
• To develop a single, flexible model capable of one-to-many image translation across multiple target modalities without retraining.
• To eliminate the need for patch cropping during training and inference, preserving global 3D context in medical image synthesis.
• To improve performance and generalization in 3D medical image-to-image translation using latent diffusion with adaptive style control.

Proposed method

• The model employs a 3D latent diffusion framework that operates directly on full 3D medical volumes, avoiding patch-based training and inference.
• It uses a conditional denoising mechanism where the target modality is used as a conditioning signal to guide the generation process.
• The core innovation is the MS-SPADE (Multiple Switchable Spatially Adaptive Normalization) block, which dynamically adapts the style of source latents to match the target modality's distribution.
• MS-SPADE enables one model to handle multiple translation tasks (e.g., T1 → T2, T1 → FLAIR) by switching normalization statistics based on the target modality.
• The model is trained using a VQ-VAE-based latent bottleneck with a learned codebook, enabling efficient 3D latent space representation.
• A combination of L2 and VQ-regularization losses ensures high-fidelity reconstruction and disentangled, semantically meaningful latent features.

Experimental results

Research questions

• RQ1 Can a single 3D latent diffusion model generalize across multiple source-to-target modality translation tasks without retraining?
• RQ2 Does the use of MS-SPADE blocks improve performance and enable one-to-many translation in 3D medical image synthesis?
• RQ3 How does the proposed model compare to existing 2D and 3D GAN-based methods in terms of image quality and structural consistency?
• RQ4 Can the model generate high-fidelity 3D images without patch cropping, preserving global anatomical context?
• RQ5 What is the contribution of each component (diffusion, palette, MS-SPADE, compression) to the final translation performance?

Key findings

• The proposed ALDM achieved the highest PSNR (25.818 ± 0.857), lowest NMSE (0.079 ± 0.016), and highest SSIM (0.904 ± 0.012) on the T1 → T2 translation task among all compared methods.
• For the T1 → T1ce translation, the model achieved a PSNR of 25.818 ± 0.857, outperforming all ablated variants and baseline models.
• The model demonstrated superior performance when using T1ce as the source modality for generating T1, T1ce, T2, and FLAIR targets, indicating its robustness and generalization capability.
• Ablation studies confirmed that the full model with all components (diffusion, palette, MS-SPADE, VQ-regularization) achieved the best performance, with each component contributing incrementally.
• The model successfully performed one-to-many translation, generating multiple target modalities from a single source input using a single unified architecture.
• The model outperformed existing one-to-one translation models on both BraTS and IXI datasets, demonstrating state-of-the-art results in quantitative metrics and visual quality.