[Paper Review] Deep Convolutional Sparse Coding Networks for Image Fusion

Shuang Xu, Zixiang Zhao+4 more|arXiv (Cornell University)|May 18, 2020

Advanced Image Fusion Techniques43 references7 citations

TL;DR

This paper proposes three deep convolutional sparse coding (CSC) networks for infrared-visible, multi-exposure, and multi-modal image fusion by unrolling the iterative shrinkage-thresholding algorithm (ISTA) into learnable dictionary convolutional units (DCUs). The method learns all hyperparameters (e.g., penalty, thresholds, filters) from data, achieving state-of-the-art performance in both quantitative metrics and visual quality across diverse datasets.

ABSTRACT

Image fusion is a significant problem in many fields including digital photography, computational imaging and remote sensing, to name but a few. Recently, deep learning has emerged as an important tool for image fusion. This paper presents three deep convolutional sparse coding (CSC) networks for three kinds of image fusion tasks (i.e., infrared and visible image fusion, multi-exposure image fusion, and multi-modal image fusion). The CSC model and the iterative shrinkage and thresholding algorithm are generalized into dictionary convolution units. As a result, all hyper-parameters are learned from data. Our extensive experiments and comprehensive comparisons reveal the superiority of the proposed networks with regard to quantitative evaluation and visual inspection.

Motivation & Objective

• To address the limitation of traditional image fusion methods that rely on handcrafted hyperparameters and lack generalization.
• To improve fusion performance by learning all hyperparameters (e.g., regularization, thresholds, filters) directly from data.
• To develop a unified, interpretable, and reproducible deep learning framework for multiple image fusion tasks.
• To overcome the poor generalization of existing methods evaluated only on cherry-picked images.
• To integrate the iterative shrinkage-thresholding algorithm (ISTA) into a differentiable neural network unit for end-to-end training.

Proposed method

• Generalizes the convolutional sparse coding (CSC) model and ISTA algorithm into a differentiable layer called the dictionary convolutional unit (DCU).
• Replaces fixed hyperparameters in ISTA with learnable components: convolutional filters, penalty parameters, and thresholding functions.
• Uses two convolutional layers to represent the forward and transposed convolution operations in ISTA, with batch normalization and learnable activation functions.
• Employs soft shrinkage thresholding (SST) or alternative activation functions (e.g., ReLU, PReLU) depending on the regularizer used.
• Stacks multiple DCUs to form deep networks tailored for three fusion tasks: infrared-visible, multi-exposure, and multi-modal image fusion.
• Trains the networks end-to-end using standard optimization (Adam) with loss functions tailored to each task, including perceptual and reconstruction losses.

Experimental results

Research questions

• RQ1 Can the iterative shrinkage-thresholding algorithm (ISTA) for convolutional sparse coding be effectively unrolled into a learnable neural network layer?
• RQ2 Does learning all hyperparameters (e.g., penalty, thresholds, filters) from data improve fusion performance across diverse image fusion tasks?
• RQ3 How does the proposed deep CSC network compare to state-of-the-art methods in terms of quantitative metrics and visual quality?
• RQ4 Can the proposed method generalize well across large, diverse datasets beyond cherry-picked test images?
• RQ5 Is the proposed framework reproducible and robust across different image fusion scenarios?

Key findings

• The proposed deep CSC networks achieve the best performance in all three image fusion tasks: infrared-visible, multi-exposure, and multi-modal fusion.
• On the multi-exposure fusion benchmark, the method achieves a MEON of 8.1776, Brisque of 18.2694, Niqe of 2.3980, and Piqe of 27.8342—best among all compared methods.
• For multi-modal fusion on the Cave dataset, the method achieves a PSNR of 46.2319 and SSIM of 0.9900, significantly outperforming the second-best method (GLPHS) with 39.4215 and 0.9706, respectively.
• Visual inspection confirms that the proposed method produces fusion images with balanced contrast, reduced halo effects, and better preservation of fine details and textures.
• The method demonstrates high reproducibility, as shown in supplementary experiments, with consistent performance across multiple runs and datasets.
• Error maps show that the proposed method minimizes reconstruction errors, especially at edges and texture-rich regions, compared to SOTA methods like CNMF and PFCN.