[Paper Review] Bi-Mamba+: Bidirectional Mamba for Time Series Forecasting

Aobo Liang, Xingguo Jiang+3 more|arXiv (Cornell University)|Apr 24, 2024

Stock Market Forecasting Methods11 citations

TL;DR

Bi-Mamba4TS proposes a bidirectional Mamba encoder with a series-relation-aware (SRA) decider to automatically choose between channel-independent and channel-mixing tokenization, using patching to enrich local information for long-term multivariate time series forecasting.

ABSTRACT

Long-term time series forecasting (LTSF) provides longer insights into future trends and patterns. Over the past few years, deep learning models especially Transformers have achieved advanced performance in LTSF tasks. However, LTSF faces inherent challenges such as long-term dependencies capturing and sparse semantic characteristics. Recently, a new state space model (SSM) named Mamba is proposed. With the selective capability on input data and the hardware-aware parallel computing algorithm, Mamba has shown great potential in balancing predicting performance and computational efficiency compared to Transformers. To enhance Mamba's ability to preserve historical information in a longer range, we design a novel Mamba+ block by adding a forget gate inside Mamba to selectively combine the new features with the historical features in a complementary manner. Furthermore, we apply Mamba+ both forward and backward and propose Bi-Mamba+, aiming to promote the model's ability to capture interactions among time series elements. Additionally, multivariate time series data in different scenarios may exhibit varying emphasis on intra- or inter-series dependencies. Therefore, we propose a series-relation-aware decider that controls the utilization of channel-independent or channel-mixing tokenization strategy for specific datasets. Extensive experiments on 8 real-world datasets show that our model achieves more accurate predictions compared with state-of-the-art methods.

Motivation & Objective

• Motivate long-term multivariate time series forecasting (LTSF) and address Transformer efficiency/accuracy trade-offs.
• Leverage State Space Models (SSM) with Mamba for linear-time complexity and long-range dependency modeling.
• Automate tokenization strategy selection via a Series-Relation-Aware (SRA) decider based on Pearson correlations.
• Enrich time series representations by patching to capture finer-grained evolutionary patterns.
• Demonstrate superior forecasting performance across seven real-world datasets compared to state-of-the-art baselines.

Proposed method

• Divide input time series into patches to obtain patch-wise tokens.
• Use a Series-Relation-Aware (SRA) decider to automatically choose between channel-independent and channel-mixing tokenization based on Pearson correlations.
• Apply a bidirectional Mamba encoder with forward and backward branches to model intra- and inter-series dependencies.
• Employ RevIN to normalize non-stationary inputs and denormalize outputs.
• Train with MSE loss and use ADAM optimization with early stopping across multiple encoder layers.

Experimental results

Research questions

• RQ1 Can a data-driven decider effectively select between channel-independent and channel-mixing tokenization to suit different MTS datasets?
• RQ2 Does patch-based tokenization combined with bidirectional Mamba improve long-term forecasting accuracy over Transformer-based and other SSM-based models?
• RQ3 What is the impact of bidirectional design on modeling intra- vs inter-series dependencies in MTS data?
• RQ4 How robust is Bi-Mamba4TS to hyper-parameter variations across diverse real-world datasets?

Key findings

• Bi-Mamba4TS achieves superior forecasting accuracy across seven real-world datasets for multiple horizons compared with strong baselines.
• The model reduces MSE/MAE versus iTransformer by average margins of 4.92% and 2.16%, respectively.
• Using channel-mixing on Weather, Traffic, Electricity and channel-independent on four ETT datasets shows dataset-dependent gains.
• Ablation studies indicate the SRA decider and bidirectional Mamba contribute meaningfully to performance gains.
• Removing the SRA decider or the backward Mamba branch degrades performance, evidencing the importance of strategy selection and bidirectionality.