[论文解读] MambaTS: Improved Selective State Space Models for Long-term Time Series Forecasting

Xiuding Cai, Yaoyao Zhu+2 more|arXiv (Cornell University)|May 26, 2024

Stock Market Forecasting Methods被引用 11

一句话总结

MambaTS 通过具备变量感知的扫描和置换策略，扩展选择性状态空间模型，以实现具有线性复杂度的长时段多变量时间序列预测的最新水平。

ABSTRACT

In recent years, Transformers have become the de-facto architecture for long-term sequence forecasting (LTSF), but faces challenges such as quadratic complexity and permutation invariant bias. A recent model, Mamba, based on selective state space models (SSMs), has emerged as a competitive alternative to Transformer, offering comparable performance with higher throughput and linear complexity related to sequence length. In this study, we analyze the limitations of current Mamba in LTSF and propose four targeted improvements, leading to MambaTS. We first introduce variable scan along time to arrange the historical information of all the variables together. We suggest that causal convolution in Mamba is not necessary for LTSF and propose the Temporal Mamba Block (TMB). We further incorporate a dropout mechanism for selective parameters of TMB to mitigate model overfitting. Moreover, we tackle the issue of variable scan order sensitivity by introducing variable permutation training. We further propose variable-aware scan along time to dynamically discover variable relationships during training and decode the optimal variable scan order by solving the shortest path visiting all nodes problem during inference. Extensive experiments conducted on eight public datasets demonstrate that MambaTS achieves new state-of-the-art performance.

研究动机与目标

• 以超越 Transformers 的高效、可扩展模型来推动改进的长期多变量时间序列预测。
• 研究 Mamba 在 LTSF 中的局限性，并开发有针对性的改进以提升全局与变量依赖关系。
• 提出一系列创新（VST、TMB、对选择参数的 dropout、VPT、VAST）以实现最先进的结果。
• 展示在八个公开的 LTSF 数据集上的可扩展性和竞争性表现。

提出的方法

• 采用 Variable Scan along Time (VST) 通过在同一时间步交替混合来自不同变量的 token 来形成全局回顾序列。
• 通过在 SSM 之前移除局部卷积，用 Temporal Mamba Block (TMB) 替代 Mamba 中的因果卷积。
• 在 TMB 的选择参数上引入 dropout 以缓解过拟合。
• 应用 Variable Permutation Training (VPT) 以降低对未知变量顺序的敏感性并提升局部交互能力。
• 开发 Variable-Aware Scan along Time (VAST) 以通过估计变量关系并在推断阶段使用模拟退火求解一个非对称旅行商问题（ATSP）来推断最优变量扫描顺序。
• 保持按通道解码头并使用受 RevIN 启发的实例归一化以在训练中保持稳定。

实验结果

研究问题

• RQ1 Can Mamba-based architectures be adapted to efficiently model long-term dependencies in multivariate time series without quadratic complexity?
• RQ2 Do variable-order sensitivity and local-context interactions limit Mamba in LTSF, and can permutation and variable-aware strategies mitigate these issues?
• RQ3 Can a data-driven, inference-time optimal variable scan order improve LTSF performance over fixed or randomly shuffled orders?
• RQ4 What is the impact of removing causal convolution in temporal blocks for LTSF, and does dropout help counteract overfitting in selective Mamba blocks?
• RQ5 How do the proposed components interact to yield state-of-the-art results across diverse LTSF datasets?

主要发现

• MambaTS achieves new state-of-the-art results on most long-term multivariate forecasting tasks across eight public datasets.
• VST enables a global retrospective representation by aggregating histories from all variables.
• Temporal Mamba Block (TMB)—without causal convolution—improves temporal modeling efficiency and, with dropout, reduces overfitting.
• Variable Permutation Training (VPT) and Variable-Aware Scan along Time (VAST) substantially mitigate sensitivity to variable order and improve local interaction capacity.
• VAST uses a learned cost matrix to guide a simulated-annealing ATSP solver, yielding effective scan orders during inference.
• Overall, MambaTS attains linear (in sequence length) computational complexity and strong empirical performance compared to Transformer-based and other baselines.