[Paper Review] DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion

Zixiang Zhao, Haowen Bai+8 more|arXiv (Cornell University)|Mar 13, 2023

Advanced Image Fusion Techniques10 citations

TL;DR

DDFM introduces a DDPM-based posterior sampling framework for multi-modality image fusion that uses unconditional diffusion sampling combined with likelihood rectification via an EM procedure, enabling high-quality IVF and MIF fusion without fine-tuning the generative model.

ABSTRACT

Multi-modality image fusion aims to combine different modalities to produce fused images that retain the complementary features of each modality, such as functional highlights and texture details. To leverage strong generative priors and address challenges such as unstable training and lack of interpretability for GAN-based generative methods, we propose a novel fusion algorithm based on the denoising diffusion probabilistic model (DDPM). The fusion task is formulated as a conditional generation problem under the DDPM sampling framework, which is further divided into an unconditional generation subproblem and a maximum likelihood subproblem. The latter is modeled in a hierarchical Bayesian manner with latent variables and inferred by the expectation-maximization (EM) algorithm. By integrating the inference solution into the diffusion sampling iteration, our method can generate high-quality fused images with natural image generative priors and cross-modality information from source images. Note that all we required is an unconditional pre-trained generative model, and no fine-tuning is needed. Our extensive experiments indicate that our approach yields promising fusion results in infrared-visible image fusion and medical image fusion. The code is available at \url{https://github.com/Zhaozixiang1228/MMIF-DDFM}.

Motivation & Objective

• Motivate and address challenges in multi-modality image fusion (MMIF) with GAN-based methods, such as unstable training and lack of interpretability.
• Propose a DDPM-based posterior sampling model for MMIF that leverages natural image priors.
• Develop a likelihood rectification mechanism using an EM algorithm to guide diffusion sampling toward source-image consistency.
• Ensure fusion quality without fine-tuning pre-trained generative models, and demonstrate applicability to IVF and MIF tasks.
• Provide extensive experimental validation across IVF and MIF datasets to show competitive performance.

Proposed method

• Formulate MMIF as a conditional generation problem under a denoising diffusion probabilistic model (DDPM) framework.
• Decompose sampling into an unconditional generation subproblem and a maximum-likelihood (likelihood rectification) subproblem.
• Model the likelihood rectification via a hierarchical Bayesian formulation with latent variables and solve it with an EM algorithm.
• Integrate the EM-derived inference solution into the diffusion sampling loop so that fusion respects both natural image priors and cross-modality information.
• Require only an unconditional pre-trained diffusion model; no fine-tuning is needed.

Experimental results

Research questions

• RQ1 Can DDPM-based posterior sampling yield high-quality fused images that preserve cross-modality information without adversarial training?
• RQ2 How can likelihood rectification be formulated and inferred in a diffusion framework to enforce agreement with source modalities?
• RQ3 Does integrating EM-based inference into diffusion sampling improve IVF and MIF fusion quality compared to unconditional diffusion or GAN-based methods?
• RQ4 How well does DDFM perform across IVF and MIF benchmarks without fine-tuning on target data?

Key findings

• DDFM consistently delivers favorable fusion results, preserving structure and texture from source images while maintaining visual fidelity.
• The approach demonstrates strong performance in IVF and medical image fusion tasks across multiple datasets.
• Likelihood rectification via EM can be embedded inside the diffusion sampling loop, enabling conditional generation without discriminator training.
• The method achieves stable and controllable fusion generation using pre-trained unconditional diffusion models without requiring fine-tuning.
• Ablation studies show the contribution of the EM-based likelihood rectification and the hierarchical Bayesian formulation to fusion quality.