[论文解读] SITSMamba for Crop Classification based on Satellite Image Time Series

Xiaolei Qin, Xin Su+1 more|arXiv (Cornell University)|Sep 15, 2024

Remote Sensing and Land Use被引用 5

一句话总结

SITSMamba 将 CNN 基于的空间编码与 Mamba 基于的时间编码器在一个多任务框架中结合，用于从卫星影像时间序列分类作物，在 PASTIS32 和 MTLCC 数据集上实现了最先进的准确率。

ABSTRACT

Satellite image time series (SITS) data provides continuous observations over time, allowing for the tracking of vegetation changes and growth patterns throughout the seasons and years. Numerous deep learning (DL) approaches using SITS for crop classification have emerged recently, with the latest approaches adopting Transformer for SITS classification. However, the quadratic complexity of self-attention in Transformer poses challenges for classifying long time series. While the cutting-edge Mamba architecture has demonstrated strength in various domains, including remote sensing image interpretation, its capacity to learn temporal representations in SITS data remains unexplored. Moreover, the existing SITS classification methods often depend solely on crop labels as supervision signals, which fails to fully exploit the temporal information. In this paper, we proposed a Satellite Image Time Series Mamba (SITSMamba) method for crop classification based on remote sensing time series data. The proposed SITSMamba contains a spatial encoder based on Convolutional Neural Networks (CNN) and a Mamba-based temporal encoder. To exploit richer temporal information from SITS, we design two branches of decoder used for different tasks. The first branch is a crop Classification Branch (CBranch), which includes a ConvBlock to decode the feature to a crop map. The second branch is a SITS Reconstruction Branch that uses a Linear layer to transform the encoded feature to predict the original input values. Furthermore, we design a Positional Weight (PW) applied to the RBranch to help the model learn rich latent knowledge from SITS. We also design two weighting factors to control the balance of the two branches during training. The code of SITSMamba is available at: https://github.com/XiaoleiQinn/SITSMamba.

研究动机与目标

• 通过利用长程时间建模来推动对卫星影像时间序列（SITS）中的精确作物分类。
• 引入 SITSMamba，一种 CNN+Mamba 架构，用于 SITS 的时空特征学习。
• 提出一个多任务框架，包含作物分类分支（CBranch）和 SITS 重建分支（RBranch），以丰富监督信号。
• 引入一个位置权重（Positional Weight）以加强对后期时间位置的学习并平衡双任务损失。

提出的方法

• 使用基于 CNN 的 ConvBlock 作为空间编码器，以提取每帧的空间特征。
• 应用 Mamba 块作为时间编码器，以高效计算捕获长程时间依赖。
• 实现两个解码分支：CBranch 用于作物图预测，RBranch 用于重建原始的 SITS。
• 引入一个位置权重（PW）以在重建损失中强调后期时间步，并为两个分支设置两个平衡权重。
• 用组合损失进行训练：L = L_cls + w0 * w1 * L_tp，其中 L_tp 是时间预测损失，L_cls 是作物分类损失。
• 推理时仅使用 CBranch 进行作物分类，而 RBranch 保持未使用。

实验结果

研究问题

• RQ1 基于 Mamba 的时间编码器是否能提升 SITS 作物分类的长程时间建模？
• RQ2 在分类的同时重建 SITS 的多任务框架是否比单任务方法在鲁棒性和准确性方面有提升？
• RQ3 时间位置权重如何影响从不同时间步对 SITS 基作物分类的学习？

主要发现

• 在 PASTIS32 数据集上，SITSMamba 达到 OA 0.7416、mIoU 0.5005、mF1 0.6449，优于若干 SOTA 方法。
• 在 MTLCC 数据集上，SITSMamba 达到 OA 0.9104、mIoU 0.7639、mF1 0.8496，是所比较方法中表现最佳的。
• 与基线相比，SITSMamba 在许多作物类别上获得更高的 F1 分数，包括若干较困难的类别。
• 消融研究表明 RBranch、PW 和损失平衡参数 w1 在提高性能方面的有效性。