[Paper Review] Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models

Jun Yue, Leyuan Fang+3 more|arXiv (Cornell University)|Jan 19, 2023

Advanced Image Fusion Techniques15 citations

TL;DR

Dif-Fusion leverages diffusion models to learn a multi-channel distribution from infrared and visible images, directly producing a color-fidelity fused image with multi-channel losses.

ABSTRACT

Color plays an important role in human visual perception, reflecting the spectrum of objects. However, the existing infrared and visible image fusion methods rarely explore how to handle multi-spectral/channel data directly and achieve high color fidelity. This paper addresses the above issue by proposing a novel method with diffusion models, termed as Dif-Fusion, to generate the distribution of the multi-channel input data, which increases the ability of multi-source information aggregation and the fidelity of colors. In specific, instead of converting multi-channel images into single-channel data in existing fusion methods, we create the multi-channel data distribution with a denoising network in a latent space with forward and reverse diffusion process. Then, we use the the denoising network to extract the multi-channel diffusion features with both visible and infrared information. Finally, we feed the multi-channel diffusion features to the multi-channel fusion module to directly generate the three-channel fused image. To retain the texture and intensity information, we propose multi-channel gradient loss and intensity loss. Along with the current evaluation metrics for measuring texture and intensity fidelity, we introduce a new evaluation metric to quantify color fidelity. Extensive experiments indicate that our method is more effective than other state-of-the-art image fusion methods, especially in color fidelity.

Motivation & Objective

• Motivate improved color fidelity in infrared and visible image fusion beyond single-channel post-processing.
• Propose a diffusion-based framework that treats multi-channel input as a latent distribution to better fuse information from infrared and visible sources.
• Directly generate three-channel fused images without color space transformations while preserving texture and color.
• Introduce new evaluation metrics to quantify color fidelity in fusion results.

Proposed method

• Concatenate infrared (1 channel) with visible (3 channels) to form a 4-channel input and model its joint distribution with a diffusion process.
• Use forward diffusion to progressively add Gaussian noise and a reverse diffusion network to denoise and learn the multi-channel latent structure.
• Extract multi-channel diffusion features from the denoising network across multiple diffusion steps to capture infrared and visible information.
• Fuse diffusion features via a multi-channel fusion module that outputs a 3-channel fused image.
• Introduce multi-channel gradient loss (LMCG) and multi-channel intensity loss (LMCI) to guide texture and intensity preservation in the three-channel output.

Experimental results

Research questions

• RQ1 How can diffusion models be used to construct the distribution of multi-channel infrared and visible data for image fusion?
• RQ2 Can diffusion-based features enable direct generation of high-color-fidelity three-channel fused images without color-space conversion?
• RQ3 What losses are effective for preserving texture, gradient, and intensity in multi-channel fused outputs?
• RQ4 How does the proposed approach perform across standard infrared-visible fusion datasets compared to state-of-the-art methods?

Key findings

• The method yields fused images with improved color fidelity and better preservation of texture and intensity than several state-of-the-art methods on public datasets.
• A diffusion-based framework can directly produce 3-channel fused images without color space transformation.
• New multi-channel gradient and multi-channel intensity losses guide the fusion toward color-accurate, detail-rich outputs.
• Qualitative and quantitative analyses indicate advantages in color preservation and perceptual quality.