[论文解读] SST: Multi-Scale Hybrid Mamba-Transformer Experts for Time Series Forecasting

Xiongxiao Xu, Chen, Canyu+5 more|arXiv (Cornell University)|Apr 23, 2024

Neural Networks and Applications被引用 15

一句话总结

提出 Mambaformer，一种混合的 Mamba-Transformer 架构，用于捕捉时间序列预测中的长程和短程依赖，并在真实数据集上展示其优于 Mamba 和 Transformer 的性能。

ABSTRACT

Time series forecasting has made significant advances, including with Transformer-based models. The attention mechanism in Transformer effectively captures temporal dependencies by attending to all past inputs simultaneously. However, its quadratic complexity with respect to sequence length limits the scalability for long-range modeling. Recent state space models (SSMs) such as Mamba offer a promising alternative by achieving linear complexity without attention. Yet, Mamba compresses historical information into a fixed-size latent state, potentially causing information loss and limiting representational effectiveness. This raises a key research question: Can we design a hybrid Mamba-Transformer architecture that is both effective and efficient for time series forecasting? To address it, we adapt a hybrid Mamba-Transformer architecture Mambaformer, originally proposed for language modeling, to the time series domain. Preliminary experiments reveal that naively stacking Mamba and Transformer layers in Mambaformer is suboptimal for time series forecasting, due to an information interference problem. To mitigate this issue, we introduce a new time series decomposition strategy that separates time series into long-range patterns and short-range variations. Then we show that Mamba excels at capturing long-term structures, while Transformer is more effective at modeling short-term dynamics. Building on this insight, we propose State Space Transformer (SST), a multi-scale hybrid model with expert modules: a Mamba expert for long-range patterns and a Transformer expert for short-term variations. SST also employs a multi-scale patching mechanism to adaptively adjust time series resolution: low resolution for long-term patterns and high resolution for short-term variations. Experiments show that SST obtains SOTA performance with linear scalability. The code is at https://github.com/XiongxiaoXu/SST.

研究动机与目标

• 在时间序列预测中同时考虑长程和短程依赖的动机。
• 引入用于时间序列数据的混合 Mamba-Transformer 架构（Mambaformer）。
• 证明 Mambaformer 在基准数据集上优于 Mamba 和 Transformer。

提出的方法

• 使用 token 与时间嵌入对时间序列数据进行嵌入。
• 使用 Mamba 块对嵌入进行预处理，以在不使用显式位置编码的情况下注入位置信息。
• 在解码器仅 Mambaformer 层中交错 Mamba 层与自注意力层，以融合长短程建模。
• 通过最终线性层进行预测，将嵌入映射回原始特征空间。

实验结果

研究问题

• RQ1 一个混合的 Mamba-Transformer 架构是否能够在时间序列预测中改进长短程预测，相较于单独使用 Mamba 或 Transformer？
• RQ2 使用 Mamba 块进行预处理是否降低或消除了在时间序列预测中对显式位置编码的需求？

主要发现

• Mambaformer 在真实多变量时间序列数据集上提供优于 Mamba 和 Transformer 的预测性能。
• Mambaformer 在 Mambaformer 家族中达到最好结果，表明长短程建模的有效整合。
• 交错 Mamba 与注意力层的顺序（混合变体）显示出可比的性能，表明架构设计具有灵活性。