[论文解读] SpectralMamba: Efficient Mamba for Hyperspectral Image Classification

Jing Yao, Danfeng Hong+2 more|arXiv (Cornell University)|Apr 12, 2024

Remote-Sensing Image Classification被引用 35

一句话总结

SpectralMamba 引入一个基于状态空间模型的骨干网络，采用分段序列扫描与门控时空融合，在高光谱图像分类中实现高精度且计算成本低。

ABSTRACT

Recurrent neural networks and Transformers have recently dominated most applications in hyperspectral (HS) imaging, owing to their capability to capture long-range dependencies from spectrum sequences. However, despite the success of these sequential architectures, the non-ignorable inefficiency caused by either difficulty in parallelization or computationally prohibitive attention still hinders their practicality, especially for large-scale observation in remote sensing scenarios. To address this issue, we herein propose SpectralMamba -- a novel state space model incorporated efficient deep learning framework for HS image classification. SpectralMamba features the simplified but adequate modeling of HS data dynamics at two levels. First, in spatial-spectral space, a dynamical mask is learned by efficient convolutions to simultaneously encode spatial regularity and spectral peculiarity, thus attenuating the spectral variability and confusion in discriminative representation learning. Second, the merged spectrum can then be efficiently operated in the hidden state space with all parameters learned input-dependent, yielding selectively focused responses without reliance on redundant attention or imparallelizable recurrence. To explore the room for further computational downsizing, a piece-wise scanning mechanism is employed in-between, transferring approximately continuous spectrum into sequences with squeezed length while maintaining short- and long-term contextual profiles among hundreds of bands. Through extensive experiments on four benchmark HS datasets acquired by satellite-, aircraft-, and UAV-borne imagers, SpectralMamba surprisingly creates promising win-wins from both performance and efficiency perspectives.

研究动机与目标

• 解决高维度、光谱可变性和光谱混淆在高光谱（HS）数据中的挑战。
• 开发一个针对 HS 分类的高效基于状态空间模型的骨干。
• 利用时空门控和分段光谱扫描来减少参数和计算量。
• 展示在来自卫星、飞机、无人机平台的多样化 HS 数据集上的强性能。

提出的方法

• 提出 SpectralMamba，一个基于 Mamba–S6 的骨干，能够在时空和隐藏状态空间中建模 HS 数据动力学。
• 引入 Piece-wise Sequential Scanning (PSS) 将光谱分段，在序列学习前形成 C×R 表示。
• 添加 Gated Spatial-Spectral Merging (GSSM)，通过深度卷积的轻量级空间门控动态合并光谱。
• 利用带输入相关参数的状态空间（S6）块，实现对光谱的选择性、高效序列建模。
• 提供像素级和块级处理路径，以适应不同输入颗粒度。
• 通过比较 MACs 和参数数量以及标准 HS 分类指标，报告训练效率和性能。

实验结果

研究问题

• RQ1 如何将状态空间模型定制化，以高效处理高维度的高光谱数据？
• RQ2 将分段光谱扫描结合门控时空融合，是否能在降低计算成本的同时提升判别能力？
• RQ3 SpectralMamba 在多种 HS 数据集和获取平台上是否实现了有利的性能–效率权衡？
• RQ4 关键组件（PSS 和 GSSM）对整体性能的贡献是什么？

主要发现

方法	OA (%)	AA (%)	Kappa	MACs (M)	Params (K)
MLP (Pixelwise)	69.94	74.26	0.6749	19.64	305.94
CNN (Pixelwise)	67.58	71.77	0.6501	26.83	418.77
CasRNN (Patchwise)	80.57	83.46	0.7894	640.09	1192.96
SpectralFormer (Pixelwise)	82.94	85.42	0.8149	681.19	72.56
SpectralMamba (Pixelwise)	89.52	90.50	0.8864	36.21	36.55

• SpectralMamba 在基准 HS 数据集上与 CNN、CasRNN、SpectralFormer 等相比，达到或超过 OA/AA/Kappa 分数。
• 消融研究显示 PSS 相比基线在 OA 提升约 4%，同时将参数减少约 60%、计算量减少约 40%。
• SpectralMamba 在卫星、飞机和无人机平台的数据集上保持相对较低的 MACs 和参数量，同时提供强劲性能。
• 该方法表现出有利的效率提升，在准确性与计算资源之间取得平衡。
• 提出的组件—PSS 和 GSSM—对性能提升和对光谱变异与混淆的鲁棒性有显著贡献。