[Paper Review] Deep Hyperspectral Unmixing using Transformer Network

Preetam Ghosh, Swalpa Kumar Roy+3 more|arXiv (Cornell University)|Mar 31, 2022

Remote-Sensing Image Classification4 citations

TL;DR

This paper proposes a novel deep hyperspectral unmixing model that combines a convolutional autoencoder with a transformer-based encoder to capture long-range spectral dependencies, significantly improving endmember estimation and abundance map accuracy. The method achieves state-of-the-art performance on three benchmark datasets, outperforming existing methods in both RMSE and spectral angle distance (SAD) metrics.

ABSTRACT

Currently, this paper is under review in IEEE. Transformers have intrigued the vision research community with their state-of-the-art performance in natural language processing. With their superior performance, transformers have found their way in the field of hyperspectral image classification and achieved promising results. In this article, we harness the power of transformers to conquer the task of hyperspectral unmixing and propose a novel deep unmixing model with transformers. We aim to utilize the ability of transformers to better capture the global feature dependencies in order to enhance the quality of the endmember spectra and the abundance maps. The proposed model is a combination of a convolutional autoencoder and a transformer. The hyperspectral data is encoded by the convolutional encoder. The transformer captures long-range dependencies between the representations derived from the encoder. The data are reconstructed using a convolutional decoder. We applied the proposed unmixing model to three widely used unmixing datasets, i.e., Samson, Apex, and Washington DC mall and compared it with the state-of-the-art in terms of root mean squared error and spectral angle distance. The source code for the proposed model will be made publicly available at \url{https://github.com/preetam22n/DeepTrans-HSU}.

Motivation & Objective

• To address the limitations of CNN-based models in capturing long-range spectral dependencies for hyperspectral unmixing.
• To improve the accuracy of endmember extraction and abundance map estimation in highly mixed hyperspectral data.
• To leverage the self-attention mechanism of transformers to model global feature relationships beyond local receptive fields.
• To develop a unified deep learning framework for blind hyperspectral unmixing that jointly optimizes endmembers and abundances.
• To validate the model's robustness and generalization across diverse real-world hyperspectral datasets with varying complexity.

Proposed method

• The model employs a convolutional encoder to extract local spatial-spectral features from hyperspectral data.
• A transformer-based encoder with Multihead Self-Patch Attention is applied to model long-range dependencies across spectral bands.
• The decoder is a transposed convolutional network that reconstructs the hyperspectral data from the latent representation.
• The loss function combines data fidelity (reconstruction error) and geometrical regularization terms to enforce physical constraints such as non-negativity and sum-to-one on abundances.
• The model is trained end-to-end using a combination of mean squared error and spectral angle distance as optimization objectives.
• Hyperparameters such as γ (regularization strength), learning rate, and weight decay are tuned via grid search to optimize performance.

Experimental results

Research questions

• RQ1 Can a transformer-based architecture improve hyperspectral unmixing by capturing long-range spectral dependencies beyond what CNNs can achieve?
• RQ2 How does the proposed model compare to state-of-the-art methods in terms of endmember estimation accuracy and abundance map quality across diverse datasets?
• RQ3 What is the impact of key hyperparameters such as γ, learning rate, and weight decay on the model’s unmixing performance?
• RQ4 Does the integration of VCA-initialized endmembers with a transformer refinement mechanism lead to better convergence and lower spectral angle distance?
• RQ5 Can the model generalize well across datasets with varying levels of spectral mixing and complexity?

Key findings

• The proposed model achieved the lowest RMSE and SAD values across all three benchmark datasets—Samson, Apex, and Washington DC Mall—outperforming existing state-of-the-art methods.
• The model demonstrated superior visual consistency in abundance maps, with the most accurate representation of challenging endmembers such as 'Water' and 'Road'.
• The use of Multihead Self-Patch Attention in the transformer encoder enabled better capture of long-range spectral dependencies, leading to more accurate endmember spectra.
• The sensitivity analysis showed that optimal performance was achieved with γ in the range 1×10⁻⁴ to 1×10⁻², with lower values preferred for datasets with more endmembers.
• The best learning rate was found to be between 0.006 and 0.009, while optimal weight decay was approximately 3×10⁻⁵, with higher values degrading performance rapidly.
• The model maintained consistent performance across all endmembers, avoiding the inconsistencies seen in methods like NMF, FCLSU, and Collab, which underperformed on specific materials.