[论文解读] S$^2$Mamba: A Spatial-spectral State Space Model for Hyperspectral Image Classification

Guanchun Wang, Xiangrong Zhang+3 more|arXiv (Cornell University)|Apr 28, 2024

Remote-Sensing Image Classification被引用 15

一句话总结

S2Mamba 通过使用 Patch Cross Scanning 与 Bi-directional Spectral Scanning，以及一个 Spatial-spectral Mixture Gate，引入了用于高光谱图像分类的时空-光谱状态空间模型，在实现线性复杂度的同时，相比若干基线获得了更高的准确性。

ABSTRACT

Land cover analysis using hyperspectral images (HSI) remains an open problem due to their low spatial resolution and complex spectral information. Recent studies are primarily dedicated to designing Transformer-based architectures for spatial-spectral long-range dependencies modeling, which is computationally expensive with quadratic complexity. Selective structured state space model (Mamba), which is efficient for modeling long-range dependencies with linear complexity, has recently shown promising progress. However, its potential in hyperspectral image processing that requires handling numerous spectral bands has not yet been explored. In this paper, we innovatively propose S$^2$Mamba, a spatial-spectral state space model for hyperspectral image classification, to excavate spatial-spectral contextual features, resulting in more efficient and accurate land cover analysis. In S$^2$Mamba, two selective structured state space models through different dimensions are designed for feature extraction, one for spatial, and the other for spectral, along with a spatial-spectral mixture gate for optimal fusion. More specifically, S$^2$Mamba first captures spatial contextual relations by interacting each pixel with its adjacent through a Patch Cross Scanning module and then explores semantic information from continuous spectral bands through a Bi-directional Spectral Scanning module. Considering the distinct expertise of the two attributes in homogenous and complicated texture scenes, we realize the Spatial-spectral Mixture Gate by a group of learnable matrices, allowing for the adaptive incorporation of representations learned across different dimensions. Extensive experiments conducted on HSI classification benchmarks demonstrate the superiority and prospect of S$^2$Mamba. The code will be made available at: https://github.com/PURE-melo/S2Mamba.

研究动机与目标

• 通过高效的长程依赖建模，推动对高光谱图像的地表覆盖分类的改进。
• 开发一种时空-光谱框架，联合捕获空间上下文和连续光谱信息。
• 提出一个可学习的融合门控，按像素自适应融合空间和光谱表示。
• 展示相对于基于 Transformer 的方法的线性复杂度和有竞争力的参数效率。

提出的方法

• Patch Cross Scanning (PCS) 将选择性状态空间扫描应用于补丁级空间路径，以捕捉相邻像素之间的上下文关系。
• Bi-directional Spectral Scanning (BSS) 通过双向交互分析连续光谱带，以提取语义光谱特征。
• 一个 Spatial-spectral Mixture Gate (SMG) 学习逐像素的融合权重，以自适应地组合空间和光谱表示。
• 该框架使用选择性的结构化状态空间模型（Mamba）来实现具有线性复杂度的长程依赖建模。
• 通过零阶保持和一阶泰勒近似对状态空间方程进行离散化，从而实现高效的序列处理。
• 对每个像素的融合时空-光谱特征进行分类。

实验结果

研究问题

• RQ1 如何扩展选择性的结构化状态空间模型，以在高光谱数据中同时捕获空间和光谱依赖？
• RQ2 PCS 和 BSS 能否提供互补信息，融合后提升高光谱土地覆盖分类？
• RQ3 可学习的时空-光谱融合门控是否比简单特征拼接或简单加权提高性能？
• RQ4 相对于基于 Transformer 的高光谱图像分类方法，S2Mamba 在复杂度和参数方面的效率优势是什么？

主要发现

• S2Mamba 在 Indian Pines、Pavia University 和 Houston 2013 数据集上，整体准确率、平均准确率和 Kappa 分数均优于包括基于 Transformer 的方法在内的多种基线。
• PCS 通过使用基于 Mamba 的机制从多个方向扫描补丁来捕捉空间上下文关系。
• BSS 通过在双向方向扫描光谱带来利用光谱连续性，以提取光谱语义。
• SMG 按像素动态融合空间和光谱特征，并使用门控阈值来修剪冗余信息。
• S2Mamba 展示了线性计算复杂度和较小的参数规模，在实验中优于最先进的基于 Transformer 的模型。
• 消融研究表明，结合 PCS、BSS 和 SMG 能在所有数据集上获得最佳 OA/AA/Kappa。