[Paper Review] An Attentive Inductive Bias for Sequential Recommendation beyond the Self-Attention

Yehjin Shin, Jeongwhan Choi+2 more|arXiv (Cornell University)|Dec 16, 2023

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This paper proposes BSARec, a novel sequential recommendation model that addresses the oversmoothing problem in Transformer-based SR by integrating Fourier transform-based attentive inductive bias and a learnable frequency rescaler to balance low- and high-frequency patterns. Experiments on 6 benchmark datasets show BSARec outperforms 7 state-of-the-art baselines in recommendation performance.

ABSTRACT

Sequential recommendation (SR) models based on Transformers have achieved remarkable successes. The self-attention mechanism of Transformers for computer vision and natural language processing suffers from the oversmoothing problem, i.e., hidden representations becoming similar to tokens. In the SR domain, we, for the first time, show that the same problem occurs. We present pioneering investigations that reveal the low-pass filtering nature of self-attention in the SR, which causes oversmoothing. To this end, we propose a novel method called $ extbf{B}$eyond $ extbf{S}$elf-$ extbf{A}$ttention for Sequential $ extbf{Rec}$ommendation (BSARec), which leverages the Fourier transform to i) inject an inductive bias by considering fine-grained sequential patterns and ii) integrate low and high-frequency information to mitigate oversmoothing. Our discovery shows significant advancements in the SR domain and is expected to bridge the gap for existing Transformer-based SR models. We test our proposed approach through extensive experiments on 6 benchmark datasets. The experimental results demonstrate that our model outperforms 7 baseline methods in terms of recommendation performance. Our code is available at https://github.com/yehjin-shin/BSARec.

Motivation & Objective

• To investigate the root cause of performance degradation in Transformer-based sequential recommendation models.
• To identify the low-pass filtering nature of self-attention as a key factor leading to oversmoothing in sequential recommendation.
• To enhance model generalization by introducing an attentive inductive bias through frequency-domain modeling.
• To mitigate oversmoothing by explicitly integrating high-frequency behavioral patterns such as short-term trends.
• To develop a unified framework that balances long-term user interests (low-frequency) and short-term preferences (high-frequency) in sequential recommendations.

Proposed method

• Leverages the Fourier transform to extract frequency-domain representations from user interaction sequences, injecting an inductive bias based on periodicity and sequential patterns.
• Introduces a learnable frequency rescaler that applies high-pass filtering to emphasize transient, short-term user behaviors while preserving long-term interests.
• Combines the Fourier-transformed representations with vanilla self-attention to maintain global dependency modeling while reducing oversmoothing.
• Designs a frequency-aware loss function that encourages the model to preserve high-frequency components critical for capturing dynamic user interests.
• Employs a dual-path architecture where the frequency domain captures inductive bias and the attention mechanism models long-range dependencies.
• Uses a spectral decomposition of user interaction sequences to separate and process low-frequency (stable preferences) and high-frequency (trending interests) components.

Experimental results

Research questions

• RQ1 Does the self-attention mechanism in Transformer-based sequential recommendation models exhibit a low-pass filtering behavior that leads to oversmoothing?
• RQ2 Can injecting an attentive inductive bias via the Fourier transform improve model generalization and recommendation accuracy?
• RQ3 To what extent can high-pass filtering of frequency components mitigate the oversmoothing problem in sequential recommendation?
• RQ4 How does the integration of low- and high-frequency information affect the model's ability to capture both long-term and short-term user preferences?
• RQ5 Is the proposed BSARec framework more effective than existing self-attention-based models in diverse sequential recommendation benchmarks?

Key findings

• The self-attention mechanism in Transformer-based SR models exhibits a low-pass filtering nature, causing hidden representations to oversmooth and lose fine-grained sequential patterns.
• The proposed BSARec model achieves state-of-the-art performance across 6 benchmark datasets, outperforming 7 existing baseline methods in recommendation accuracy.
• The frequency rescaler effectively captures high-frequency behavioral patterns such as short-term trends, improving the model's responsiveness to changing user interests.
• The integration of Fourier-based inductive bias enhances model generalization by preserving both long-term preferences and transient user behaviors.
• Ablation studies confirm that both the attentive inductive bias and high-pass filtering components are essential for performance gains, with each contributing significantly to the final results.
• The model demonstrates robustness across diverse datasets, including Amazon-Beauty, ML-1M, and others, confirming its generalizability beyond specific domains.