[Paper Review] CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion

Zixiang Zhao, Haowen Bai+6 more|arXiv (Cornell University)|Nov 26, 2022

Advanced Image Fusion Techniques23 citations

TL;DR

CDDFuse introduces a dual-branch Transformer-CNN encoder with correlation-driven loss to decompose base (shared) and detail (specific) features for multi-modality image fusion and achieves state-of-the-art IVF and MIF results.

ABSTRACT

Multi-modality (MM) image fusion aims to render fused images that maintain the merits of different modalities, e.g., functional highlight and detailed textures. To tackle the challenge in modeling cross-modality features and decomposing desirable modality-specific and modality-shared features, we propose a novel Correlation-Driven feature Decomposition Fusion (CDDFuse) network. Firstly, CDDFuse uses Restormer blocks to extract cross-modality shallow features. We then introduce a dual-branch Transformer-CNN feature extractor with Lite Transformer (LT) blocks leveraging long-range attention to handle low-frequency global features and Invertible Neural Networks (INN) blocks focusing on extracting high-frequency local information. A correlation-driven loss is further proposed to make the low-frequency features correlated while the high-frequency features uncorrelated based on the embedded information. Then, the LT-based global fusion and INN-based local fusion layers output the fused image. Extensive experiments demonstrate that our CDDFuse achieves promising results in multiple fusion tasks, including infrared-visible image fusion and medical image fusion. We also show that CDDFuse can boost the performance in downstream infrared-visible semantic segmentation and object detection in a unified benchmark. The code is available at https://github.com/Zhaozixiang1228/MMIF-CDDFuse.

Motivation & Objective

• Motivate improved cross-modality feature modeling for MMIF by separating modality-shared and modality-specific information.
• Develop a dual-branch encoder that combines global (Transformer) and local (CNN/INN) feature extraction.
• Propose a correlation-driven loss to enforce base features to be correlated and detail features to be decorrelated across modalities.
• Enable effective fusion through LT-based global fusion and INN-based local fusion layers.
• Demonstrate state-of-the-art performance on IVF and MIF benchmarks and show downstream benefits for segmentation and detection.

Proposed method

• Use Restormer-based shallow feature extraction to capture cross-modality cues.
• Employ a dual-branch Long-Short Range Encoder: a Base Transformer Encoder (LT blocks) for low-frequency base features and a Detail CNN Encoder (INN blocks) for high-frequency details.
• Introduce Invertible Neural Networks (INN) in the detail path to preserve high-frequency information.
• Fuse base features with LT-based fusion and details with INN-based fusion.
• Train in two stages: Stage I reconstructs input images to guide feature decomposition; Stage II performs fusion using the decomposed features.
• Adopt a correlation-driven decomposition loss that encourages high correlation of base features and decorrelation of detail features: L_decomp = (CC(D_I,V))^2 / (CC(B_I,B_V) + ε).
• Stage II loss includes image reconstruction/gradient terms plus the same decomposition constraint to guide fused output quality.

Experimental results

Research questions

• RQ1 How can MMIF benefit from explicitly decomposing features into modality-shared (base) and modality-specific (detail) components?
• RQ2 Can a hybrid Transformer-CNN architecture effectively capture global and local cross-modality information for IVF and MIF?
• RQ3 Does a correlation-driven loss improve the separability of base and detail features and enhance fusion quality?
• RQ4 Do LT-based global fusion and INN-based local fusion improve performance while maintaining efficiency?
• RQ5 What is the impact of CDDFuse on downstream tasks like infrared-visible semantic segmentation and object detection?

Key findings

• CDDFuse achieves state-of-the-art or competitive results across IVF benchmarks (MSRS, RoadScene, TNO) using eight fusion metrics.
• On MSRS, CDDFuse attains EN=6.70, SD=43.38, SF=11.56, MI=3.47, SCD=1.62, VIF=1.05, Qbaf=0.69, SSIM=1.00.
• On RoadScene, CDDFuse attains EN=7.44, SD=54.67, SF=16.36, MI=2.30, SCD=1.81, VIF=0.69, Qbaf=0.52, SSIM=0.98.
• On TNO, CDDFuse attains EN=7.44, SD=54.67, SF=16.36, MI=2.30, SCD=1.81, VIF=0.69, Qbaf=0.52, SSIM=0.98.
• Ablation studies validate the effectiveness of the two-stage training, LT+INN combination, and the correlation-based decomposition loss.
• Downstream MM object detection and semantic segmentation show improved performance when using fused outputs, indicating practical benefits beyond fusion quality.