[Paper Review] EchoMamba4Rec: Harmonizing Bidirectional State Space Models with Spectral Filtering for Advanced Sequential Recommendation

Yuda Wang, Xuxin He+1 more|arXiv (Cornell University)|Jun 4, 2024

Machine Learning in Healthcare4 citations

TL;DR

EchoMamba4Rec proposes a novel sequential recommendation model that harmonizes bidirectional Mamba state space networks with spectral filtering via FFT to enhance long-range dependency modeling and inference efficiency. It achieves state-of-the-art performance across multiple benchmarks while maintaining linear scalability and reduced memory usage.

ABSTRACT

Predicting user preferences and sequential dependencies based on historical behavior is the core goal of sequential recommendation. Although attention-based models have shown effectiveness in this field, they often struggle with inference inefficiency due to the quadratic computational complexity inherent in attention mechanisms, especially with long-range behavior sequences. Drawing inspiration from the recent advancements of state space models (SSMs) in control theory, which provide a robust framework for modeling and controlling dynamic systems, we introduce EchoMamba4Rec. Control theory emphasizes the use of SSMs for managing long-range dependencies and maintaining inferential efficiency through structured state matrices. EchoMamba4Rec leverages these control relationships in sequential recommendation and integrates bi-directional processing with frequency-domain filtering to capture complex patterns and dependencies in user interaction data more effectively. Our model benefits from the ability of state space models (SSMs) to learn and perform parallel computations, significantly enhancing computational efficiency and scalability. It features a bi-directional Mamba module that incorporates both forward and reverse Mamba components, leveraging information from both past and future interactions. Additionally, a filter layer operates in the frequency domain using learnable Fast Fourier Transform (FFT) and learnable filters, followed by an inverse FFT to refine item embeddings and reduce noise. We also integrate Gate Linear Units (GLU) to dynamically control information flow, enhancing the model's expressiveness and training stability. Experimental results demonstrate that EchoMamba significantly outperforms existing models, providing more accurate and personalized recommendations.

Motivation & Objective

• Address the quadratic computational complexity and memory bottlenecks of attention-based models like Transformers in sequential recommendation.
• Leverage state space models (SSMs) from control theory to enable efficient, linear-time modeling of long sequential dependencies.
• Integrate bidirectional processing and frequency-domain filtering to capture richer temporal patterns from both past and future user interactions.
• Improve model expressiveness and training stability through learnable Gated Linear Units (GLU) and structured state matrices.
• Achieve superior recommendation accuracy and inference efficiency compared to SOTA models including Mamba, Transformers, and RNNs.

Proposed method

• Employ a bidirectional Mamba module that processes user interaction sequences in both forward and reverse directions using selective SSMs.
• Apply Fast Fourier Transform (FFT) to project item embeddings into the frequency domain for spectral filtering with learnable filters.
• Use inverse FFT to reconstruct filtered embeddings, reducing noise and enhancing representation quality.
• Integrate Gated Linear Units (GLU) to dynamically gate information flow, improving model expressiveness and training stability.
• Combine the filtered embeddings with residual connections and layer normalization for stable training and improved feature learning.
• Utilize structured state matrices in Mamba blocks to ensure linear computational complexity and hardware-efficient inference.

Experimental results

Research questions

• RQ1 Can bidirectional processing in Mamba-based models significantly improve sequential recommendation performance by leveraging future context?
• RQ2 To what extent does spectral filtering in the frequency domain enhance representation learning and noise suppression in user behavior sequences?
• RQ3 How does the integration of GLU units affect the expressiveness and training stability of SSM-based recommendation models?
• RQ4 Does EchoMamba4Rec outperform existing SOTA models in both accuracy and inference efficiency on long sequential recommendation tasks?
• RQ5 Can the proposed model maintain low memory consumption while scaling to long user interaction sequences?

Key findings

• EchoMamba4Rec achieves the highest HR@10 score of 0.3215 on the MovieLens-1M dataset, outperforming the second-best model (Bi-Mambaformer4Rec) by 0.0051.
• On Amazon-Beauty, EchoMamba4Rec achieves an NDCG@10 of 0.1529, surpassing Mamba4Rec (0.1509) and BERT4Rec (0.0979), indicating superior ranking quality.
• In the Amazon-Video-Games dataset, EchoMamba4Rec achieves an MRR@10 of 0.0445, the highest among all models, showing improved first-relevant-item prediction.
• EchoMamba4Rec reduces training time per epoch to 70.187 seconds on RTX 4090, significantly faster than SASRec (163.543 s) and BERT4Rec (215.800 s), while using only 1.61 GB GPU memory.
• The model maintains low memory usage (1.61 GB) and achieves inference time of 0.0252 seconds, outperforming Mamba4Rec (0.0305 s) and being over 5x faster than Transformers.
• The ablation study confirms that both bidirectional processing and spectral filtering contribute significantly to performance gains, with the full model outperforming ablated variants across all metrics.