[Paper Review] Multispectral and Hyperspectral Image Fusion by MS/HS Fusion Net

Qi Xie, Minghao Zhou+4 more|arXiv (Cornell University)|Jan 10, 2019

Advanced Image Fusion Techniques51 references19 citations

TL;DR

This paper proposes MS/HS Fusion Net, a deep learning framework that integrates physical observation models and low-rank spectral priors into a learnable optimization-based network for fusing high-resolution multispectral (HrMS) and low-resolution hyperspectral (LrHS) images into high-resolution hyperspectral (HrHS) images. By unfolding an iterative proximal algorithm into a deep residual network with learned operators, the method achieves state-of-the-art performance in both simulated and real-world data, significantly outperforming existing methods in PSNR, SAM, and SSIM metrics.

ABSTRACT

Hyperspectral imaging can help better understand the characteristics of different materials, compared with traditional image systems. However, only high-resolution multispectral (HrMS) and low-resolution hyperspectral (LrHS) images can generally be captured at video rate in practice. In this paper, we propose a model-based deep learning approach for merging an HrMS and LrHS images to generate a high-resolution hyperspectral (HrHS) image. In specific, we construct a novel MS/HS fusion model which takes the observation models of low-resolution images and the low-rankness knowledge along the spectral mode of HrHS image into consideration. Then we design an iterative algorithm to solve the model by exploiting the proximal gradient method. And then, by unfolding the designed algorithm, we construct a deep network, called MS/HS Fusion Net, with learning the proximal operators and model parameters by convolutional neural networks. Experimental results on simulated and real data substantiate the superiority of our method both visually and quantitatively as compared with state-of-the-art methods along this line of research.

Motivation & Objective

• To address the ill-posed nature of hyperspectral image fusion by incorporating physical observation models and intrinsic image priors.
• To overcome the limitations of conventional methods that rely on subjective, fixed priors such as sparsity or total variation.
• To develop a deep learning model that is interpretable and specifically designed for MS/HS fusion, rather than adapting generic networks.
• To enable end-to-end learning of spatial and spectral response operators (R and C) from data, improving fusion accuracy.
• To achieve superior performance on both simulated and real-world datasets, including challenging real data without ground truth.

Proposed method

• The method formulates MS/HS fusion as an optimization problem incorporating the observation models: Y = XR + Ny and Z = CX + Nz, where X is the target HrHS image.
• It enforces low-rankness along the spectral dimension of X to exploit intrinsic spectral correlation, reducing spectral distortion.
• An iterative proximal gradient algorithm is designed to solve the optimization problem, integrating both observation models and low-rank prior.
• The algorithm is unfolded into a deep residual network (MS/HS Fusion Net), where proximal operators and model parameters are learned via convolutional layers.
• The network is trained end-to-end on paired HrMS and LrHS patches with corresponding HrHS targets, enabling joint learning of spatial-spectral reconstruction and response operators.
• The architecture is specifically designed to preserve physical consistency and interpretability by embedding the underlying image formation process.

Experimental results

Research questions

• RQ1 Can a deep learning model for MS/HS fusion achieve better performance by embedding physical observation models rather than relying on generic network architectures?
• RQ2 How does incorporating spectral low-rankness improve fusion accuracy and reduce spectral distortion?
• RQ3 Can an optimization-based iterative algorithm be effectively unfolded into a deep network that generalizes well to real-world data?
• RQ4 To what extent can the spatial and spectral response operators (R and C) be end-to-end learned from data?
• RQ5 Does the proposed method outperform state-of-the-art methods in both simulated and real-world MS/HS fusion scenarios?

Key findings

• On the Chikusei dataset, the proposed MHF-net achieved a PSNR of 32.26 dB, significantly outperforming the second-best method (ResNet at 29.35 dB).
• The method achieved a SAM of 3.02° on Chikusei, substantially lower than ResNet (3.69°), indicating better spectral fidelity.
• In visual comparisons, MHF-net produced results closest to ground truth, with minimal spectral distortion and better preservation of fine textures and structural details.
• On real-world Roman Colosseum data, MHF-net produced visually superior results with more accurate color and brightness matching to the LrHS image compared to ResNet.
• The method achieved the highest SSIM (0.890) and FSIM (0.946) on Chikusei, indicating superior structural and perceptual quality.
• The network demonstrated strong generalization on real data without ground truth, with visual quality consistently better than competing methods.