[Paper Review] FUSFORMER: A TRANSFORMER-BASED FUSION APPROACH FOR HYPERSPECTRAL IMAGE SUPER-RESOLUTION

Jinfan Hu|arXiv (Cornell University)|Sep 5, 2021

Advanced Image Fusion Techniques35 references6 citations

TL;DR

This paper proposes Fusformer, a novel transformer-based fusion network for hyperspectral image super-resolution that leverages self-attention mechanisms to capture global spatial-spectral dependencies between low-resolution hyperspectral images (LR-HSI) and high-resolution multispectral images (HR-MSI). By learning residuals instead of reconstructing the full HR-HSI, the method achieves state-of-the-art performance with only 0.1 million parameters, significantly reducing computational cost while improving accuracy.

ABSTRACT

Hyperspectral image has become increasingly crucial due to its abundant spectral information. However, It has poor spatial resolution with the limitation of the current imaging mechanism. Nowadays, many convolutional neural networks have been proposed for the hyperspectral image super-resolution problem. However, convolutional neural network (CNN) based methods only consider the local information instead of the global one with the limited kernel size of receptive field in the convolution operation. In this paper, we design a network based on the transformer for fusing the low-resolution hyperspectral images and high-resolution multispectral images to obtain the high-resolution hyperspectral images. Thanks to the representing ability of the transformer, our approach is able to explore the intrinsic relationships of features globally. Furthermore, considering the LR-HSIs hold the main spectral structure, the network focuses on the spatial detail estimation releasing from the burden of reconstructing the whole data. It reduces the mapping space of the proposed network, which enhances the final performance. Various experiments and quality indexes show our approach's superiority compared with other state-of-the-art methods.

Motivation & Objective

• To address the limitation of CNN-based methods in capturing global spatial-spectral relationships in hyperspectral image super-resolution (HSI-SR).
• To leverage the self-attention mechanism of transformers to model long-range dependencies across spectral and spatial dimensions more effectively than convolutional operations.
• To reduce the network's mapping complexity by focusing on residual learning, thereby improving training stability and performance.
• To develop a lightweight, efficient architecture suitable for real-world deployment with minimal parameters and high generalization ability.

Proposed method

• The proposed Fusformer employs a transformer encoder with multi-head self-attention to model global dependencies between features in LR-HSI and HR-MSI.
• The network is trained to predict the residual between the upsampled LR-HSI and the ground-truth HR-HSI, rather than reconstructing the full HR-HSI directly.
• The self-attention mechanism enables the model to capture long-range, non-local relationships across spectral bands and spatial locations more effectively than local convolutions.
• The architecture is designed to be simple and parameter-efficient, with only 0.1 million parameters, enhancing practicality and generalization.
• Feature fusion is performed early in the network, allowing the transformer to jointly attend to complementary information from both input modalities.
• The model is trained end-to-end using a perceptual loss and standard metrics such as PSNR, SAM, ERGAS, and SSIM.

Experimental results

Research questions

• RQ1 Can a transformer-based architecture outperform existing CNN-based methods in hyperspectral image super-resolution by modeling global spatial-spectral dependencies?
• RQ2 How does residual learning affect the performance and stability of HSI super-resolution networks?
• RQ3 To what extent does a lightweight transformer architecture with minimal parameters generalize across different datasets and image types?
• RQ4 Does the use of self-attention improve reconstruction quality compared to local receptive field-based convolutions in HSI-SR?
• RQ5 Can a simple, parameter-efficient transformer design achieve state-of-the-art performance without sacrificing accuracy?

Key findings

• Fusformer achieves a PSNR of 48.56 on the CAVE dataset, outperforming all state-of-the-art methods, including those with significantly more parameters.
• On the Harvard dataset, Fusformer achieves a PSNR of 44.42, ranking first and demonstrating strong generalization capability.
• The method achieves the lowest SAM (2.52) and ERGAS (1.30) values on CAVE, indicating superior spectral fidelity and reconstruction accuracy.
• The ablation study confirms that residual learning is crucial, as removing it reduces PSNR by over 5.8 dB, highlighting its role in stability and performance.
• Despite having only 0.1 million parameters, Fusformer outperforms models with up to 3.63 million parameters, proving its efficiency and effectiveness.
• Visual results and residual maps show that Fusformer produces the most accurate and detailed reconstructions with the darkest residual maps, indicating minimal reconstruction error.