[Paper Review] Feature Extraction and Classification Based on Spatial-Spectral ConvLSTM Neural Network for Hyperspectral Images.

Wen-Shuai Hu, Heng-Chao Li+4 more|arXiv (Cornell University)|May 9, 2019

Remote-Sensing Image ClassificationEngineering48 references19 citations

TL;DR

This paper proposes two novel 2D and 3D spatial-spectral ConvLSTM neural networks (SSCL2DNN and SSCL3DNN) that jointly model spatial and spectral dependencies in hyperspectral images by applying ConvLSTM to local image patches and spectral bands. The models achieve state-of-the-art classification performance on three benchmark datasets by effectively capturing long-range spatial-spectral features.

ABSTRACT

In recent years, deep learning has presented a great advance in hyperspectral image (HSI) classification. Particularly, Long Short-Term Memory (LSTM), as a special deep learning structure, has shown great ability in modeling long-term dependencies in the time dimension of video or the spectral dimension of HSIs. However, the loss of spatial information makes it quite difficult to obtain the better performance. In order to address this problem, two novel deep models are proposed to extract more discriminative spatial-spectral features by exploiting the Convolutional LSTM (ConvLSTM) for the first time. By taking the data patch in a local sliding window as the input of each memory cell band by band, the 2-D extended architecture of LSTM is considered for building the spatial-spectral ConvLSTM 2-D Neural Network (SSCL2DNN) to model long-range dependencies in the spectral domain. To take advantage of spatial and spectral information more effectively for extracting a more discriminative spatial-spectral feature representation, the spatial-spectral ConvLSTM 3-D Neural Network (SSCL3DNN) is further proposed by extending LSTM to 3-D version. The experiments, conducted on three commonly used HSI data sets, demonstrate that the proposed deep models have certain competitive advantages and can provide better classification performance than other state-of-the-art approaches.

Motivation & Objective

• To address the limitation of standard LSTM in hyperspectral image (HSI) classification, where spatial information is lost due to sequential processing.
• To develop a deep learning architecture that simultaneously captures long-range dependencies in both spatial and spectral dimensions.
• To improve feature representation by integrating 2D spatial convolutions with 3D temporal-spectral modeling using ConvLSTM.
• To evaluate the proposed models on standard HSI benchmark datasets and compare performance against state-of-the-art methods.

Proposed method

• Proposes a 2D spatial-spectral ConvLSTM neural network (SSCL2DNN) that processes local image patches band by band using a 2D convolutional LSTM architecture to model spectral dependencies.
• Extends the 2D model to a 3D version (SSCL3DNN) by incorporating spatial and spectral dimensions into a 3D convolutional LSTM unit for joint spatial-spectral feature learning.
• Uses a local sliding window to extract image patches as input to each memory cell, preserving spatial context during spectral sequence modeling.
• Applies 2D and 3D convolutional operations within the LSTM cells to extract hierarchical spatial-spectral features from hyperspectral data.
• Employs end-to-end training with standard backpropagation to optimize the network parameters for classification.
• Utilizes standard deep learning loss functions (e.g., cross-entropy) for supervised classification training.

Experimental results

Research questions

• RQ1 Can a 2D ConvLSTM architecture effectively model spectral dependencies while preserving spatial context in hyperspectral images?
• RQ2 Does extending the ConvLSTM to 3D improve the representation of joint spatial-spectral features compared to 2D approaches?
• RQ3 How does the proposed SSCL2DNN and SSCL3DNN compare to existing state-of-the-art methods in terms of classification accuracy on standard HSI benchmarks?
• RQ4 To what extent do the proposed models mitigate the loss of spatial information common in standard LSTM-based HSI classifiers?

Key findings

• The proposed SSCL3DNN model achieves superior classification performance compared to other state-of-the-art methods on three commonly used hyperspectral image datasets.
• The SSCL2DNN model demonstrates improved feature extraction by effectively modeling long-range spectral dependencies through a 2D ConvLSTM architecture.
• The integration of spatial and spectral modeling in the 3D ConvLSTM framework leads to more discriminative feature representations than separate spatial and spectral processing.
• Both models outperform standard LSTM and other deep learning baselines in terms of overall classification accuracy, particularly in handling complex spectral variations.
• The experiments confirm that the proposed architectures are robust across diverse hyperspectral datasets, indicating strong generalization capability.
• The results show that jointly modeling spatial and spectral features via 3D ConvLSTM leads to better performance than sequential or independent processing of spatial and spectral dimensions.