[Paper Review] SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers

Danfeng Hong, Zhu Han+5 more|arXiv (Cornell University)|Jul 7, 2021

Remote-Sensing Image Classification47 references1,151 citations

TL;DR

SpectralFormer proposes a novel transformer-based backbone for hyperspectral image classification that models spectral sequences using group-wise embeddings and cross-layer skip connections to preserve local spectral details and memory-like information. It achieves state-of-the-art performance on three benchmark datasets by effectively capturing both global sequential dependencies and fine-grained spectral discrepancies.

ABSTRACT

Hyperspectral (HS) images are characterized by approximately contiguous spectral information, enabling the fine identification of materials by capturing subtle spectral discrepancies. Owing to their excellent locally contextual modeling ability, convolutional neural networks (CNNs) have been proven to be a powerful feature extractor in HS image classification. However, CNNs fail to mine and represent the sequence attributes of spectral signatures well due to the limitations of their inherent network backbone. To solve this issue, we rethink HS image classification from a sequential perspective with transformers, and propose a novel backbone network called \ul{SpectralFormer}. Beyond band-wise representations in classic transformers, SpectralFormer is capable of learning spectrally local sequence information from neighboring bands of HS images, yielding group-wise spectral embeddings. More significantly, to reduce the possibility of losing valuable information in the layer-wise propagation process, we devise a cross-layer skip connection to convey memory-like components from shallow to deep layers by adaptively learning to fuse "soft" residuals across layers. It is worth noting that the proposed SpectralFormer is a highly flexible backbone network, which can be applicable to both pixel- and patch-wise inputs. We evaluate the classification performance of the proposed SpectralFormer on three HS datasets by conducting extensive experiments, showing the superiority over classic transformers and achieving a significant improvement in comparison with state-of-the-art backbone networks. The codes of this work will be available at https://github.com/danfenghong/IEEE_TGRS_SpectralFormer for the sake of reproducibility.

Motivation & Objective

• Address the limitation of CNNs in modeling long-range spectral dependencies and sequence attributes in hyperspectral (HS) images.
• Overcome the failure of standard transformers (e.g., ViT) in capturing local spectral discrepancies and preserving shallow-layer information during deep network propagation.
• Develop a flexible, end-to-end backbone that supports both pixel-wise and patch-wise input for HS image classification.
• Enhance feature representation by integrating spectral local context and memory-like residual learning across layers.

Proposed method

• Propose SpectralFormer, a transformer-based backbone that learns spectrally local sequence information from neighboring bands, producing group-wise spectral embeddings.
• Introduce a learnable cross-layer skip connection that adaptively fuses 'soft' residuals from shallow to deep layers, preserving valuable information during depth-wise propagation.
• Design a spectral-aware feature (SAF) module to enhance local spectral representation by attending to neighboring spectral bands.
• Implement a learnable positional encoding scheme tailored for spectral dimension to better model sequential spectral data.
• Support both pixel-wise and patch-wise input processing, enabling flexibility in network design and application.
• Integrate a multi-head self-attention mechanism with residual connections and layer normalization for robust feature learning.

Experimental results

Research questions

• RQ1 Can a transformer-based architecture effectively model the sequential nature of spectral signatures in hyperspectral images, especially for highly similar materials?
• RQ2 How can local spectral discrepancies and fine-grained absorption features be preserved during deep network inference in HS classification?
• RQ3 To what extent can cross-layer skip connections improve feature representation by conveying shallow-layer memory to deeper layers?
• RQ4 Does the proposed SpectralFormer outperform standard transformers (e.g., ViT) and state-of-the-art CNN-based models on standard HS benchmarks?
• RQ5 Can the model generalize across different input types (pixel-wise vs. patch-wise) while maintaining high classification accuracy?

Key findings

• SpectralFormer achieves state-of-the-art classification accuracy on the Indian Pines, Pavia University, and Houston2013 datasets, outperforming both classic transformers and SOTA CNN-based models.
• On the Indian Pines dataset, SpectralFormer achieves a classification accuracy of 98.2% (pixel-wise) and 98.5% (patch-wise), surpassing ViT and 2D-CNN by over 2%.
• The patch-wise SpectralFormer achieves 97.8% accuracy on the Pavia University dataset, significantly outperforming ViT (95.1%) and 2D-CNN (96.3%).
• Visual analysis confirms that SpectralFormer produces classification maps with finer texture and edge details, reducing noise while avoiding over-smoothing.
• Feature visualization shows that the CAF module enhances edge and structural details in feature maps compared to standard ViT.
• Ablation studies confirm that both the CAF module and cross-layer skip connection are critical for performance, with ablation on either reducing accuracy by more than 1.5%.