[Paper Review] Hyperspectral Image Super-Resolution via Dual-domain Network Based on Hybrid Convolution

Tingting Liu, Yuan Liu+4 more|arXiv (Cornell University)|Apr 10, 2023

Advanced Image Fusion Techniques4 citations

TL;DR

This paper proposes SRDNet, a dual-domain hyperspectral image super-resolution method that leverages hybrid 2D/3D convolutional units with progressive upsampling and a dynamic frequency loss to enhance spatial-spectral feature learning. By integrating self-attention in the spatial domain and frequency-domain optimization, SRDNet achieves state-of-the-art performance on benchmark datasets, significantly improving texture detail and perceptual quality.

ABSTRACT

Since the number of incident energies is limited, it is difficult to directly acquire hyperspectral images (HSI) with high spatial resolution. Considering the high dimensionality and correlation of HSI, super-resolution (SR) of HSI remains a challenge in the absence of auxiliary high-resolution images. Furthermore, it is very important to extract the spatial features effectively and make full use of the spectral information. This paper proposes a novel HSI super-resolution algorithm, termed dual-domain network based on hybrid convolution (SRDNet). Specifically, a dual-domain network is designed to fully exploit the spatial-spectral and frequency information among the hyper-spectral data. To capture inter-spectral self-similarity, a self-attention learning mechanism (HSL) is devised in the spatial domain. Meanwhile the pyramid structure is applied to increase the acceptance field of attention, which further reinforces the feature representation ability of the network. Moreover, to further improve the perceptual quality of HSI, a frequency loss(HFL) is introduced to optimize the model in the frequency domain. The dynamic weighting mechanism drives the network to gradually refine the generated frequency and excessive smoothing caused by spatial loss. Finally, In order to better fully obtain the mapping relationship between high-resolution space and low-resolution space, a hybrid module of 2D and 3D units with progressive upsampling strategy is utilized in our method. Experiments on a widely used benchmark dataset illustrate that the proposed SRDNet method enhances the texture information of HSI and is superior to state-of-the-art methods.

Motivation & Objective

• To address the challenge of low spatial resolution in hyperspectral images (HSI) due to limited incident energy.
• To improve feature representation by effectively capturing spatial-spectral correlations and inter-spectral self-similarity in HSI.
• To enhance perceptual quality of reconstructed HSI by incorporating frequency-domain optimization.
• To establish a robust mapping between low-resolution and high-resolution HSI using a hybrid 2D/3D convolutional architecture with progressive upsampling.
• To overcome limitations of existing methods that lack auxiliary high-resolution data and fail to fully exploit spectral and spatial information.

Proposed method

• A dual-domain network is designed to jointly exploit spatial-spectral and frequency-domain information in hyperspectral data.
• A self-attention mechanism (HSL) is applied in the spatial domain to capture inter-spectral self-similarity, enhanced by a pyramid structure to increase receptive field.
• A frequency loss (HFL) is introduced to optimize the model in the frequency domain, improving perceptual quality and reducing excessive smoothing.
• A dynamic weighting mechanism balances spatial and frequency losses, enabling progressive refinement of high-frequency components.
• A hybrid module combining 2D and 3D convolutional units is used with a progressive upsampling strategy to better learn the LR-to-HR mapping.
• The network is trained end-to-end with a multi-loss objective combining L1 loss, perceptual loss, and frequency loss for improved reconstruction fidelity.

Experimental results

Research questions

• RQ1 How can spatial-spectral features be effectively captured in hyperspectral image super-resolution without auxiliary high-resolution data?
• RQ2 To what extent does incorporating frequency-domain optimization improve the perceptual quality of super-resolved HSI?
• RQ3 Can a hybrid 2D/3D convolutional architecture with progressive upsampling better model the LR-to-HR mapping in HSI than standard 2D or 3D networks?
• RQ4 How does self-attention with a pyramid structure enhance feature representation in the spatial domain for HSI SR?
• RQ5 What is the impact of dynamic loss weighting between spatial and frequency domains on reconstruction accuracy and texture preservation?

Key findings

• SRDNet achieves superior performance on a widely used benchmark dataset, outperforming state-of-the-art methods in both quantitative metrics and visual quality.
• The proposed frequency loss (HFL) significantly improves perceptual quality by preserving high-frequency details and reducing over-smoothing.
• The integration of self-attention with a pyramid structure enhances the network's ability to model long-range spectral dependencies and improve feature representation.
• The dynamic weighting mechanism effectively balances spatial and frequency losses, leading to more accurate and detailed reconstructions.
• The hybrid 2D/3D convolutional module with progressive upsampling enables better learning of the low-to-high-resolution mapping, resulting in sharper textures and finer spatial details.
• Ablation studies confirm the effectiveness of each component, including HSL, HFL, and the hybrid architecture, in boosting overall performance.