[Paper Review] Frequency Enhanced Hybrid Attention Network for Sequential Recommendation

Xinyu Du, Huanhuan Yuan+5 more|arXiv (Cornell University)|Apr 18, 2023

Recommender Systems and Techniques4 citations

TL;DR

This paper proposes FEARec, a frequency-enhanced hybrid attention network that improves sequential recommendation by modeling both time and frequency domains. It introduces a frequency ramp structure to preserve high-frequency information and a frequency-domain self-attention mechanism via auto-correlation to capture periodic patterns, achieving state-of-the-art performance on four benchmark datasets through contrastive learning and frequency regularization.

ABSTRACT

The self-attention mechanism, which equips with a strong capability of modeling long-range dependencies, is one of the extensively used techniques in the sequential recommendation field. However, many recent studies represent that current self-attention based models are low-pass filters and are inadequate to capture high-frequency information. Furthermore, since the items in the user behaviors are intertwined with each other, these models are incomplete to distinguish the inherent periodicity obscured in the time domain. In this work, we shift the perspective to the frequency domain, and propose a novel Frequency Enhanced Hybrid Attention Network for Sequential Recommendation, namely FEARec. In this model, we firstly improve the original time domain self-attention in the frequency domain with a ramp structure to make both low-frequency and high-frequency information could be explicitly learned in our approach. Moreover, we additionally design a similar attention mechanism via auto-correlation in the frequency domain to capture the periodic characteristics and fuse the time and frequency level attention in a union model. Finally, both contrastive learning and frequency regularization are utilized to ensure that multiple views are aligned in both the time domain and frequency domain. Extensive experiments conducted on four widely used benchmark datasets demonstrate that the proposed model performs significantly better than the state-of-the-art approaches.

Motivation & Objective

• To address the limitation of self-attention mechanisms in capturing high-frequency and periodic user behavior patterns in sequential recommendation.
• To improve modeling of long-range dependencies by shifting from the time domain to the frequency domain using Fourier transforms.
• To explicitly learn both low- and high-frequency information through a novel frequency ramp structure in the attention mechanism.
• To capture inherent periodicity in user behaviors using auto-correlation in the frequency domain.
• To align multiple views (time and frequency) via contrastive learning and frequency regularization for improved representation learning.

Proposed method

• Proposes a frequency ramp structure to modify the original self-attention mechanism, enabling explicit learning of both low- and high-frequency components in the frequency domain.
• Introduces a frequency-domain attention mechanism based on auto-correlation of lagged sequences to model periodic behavioral patterns.
• Applies Discrete Fourier Transform (DFT) to convert time-domain sequences into frequency-domain representations for enhanced spectral analysis.
• Employs Fast Fourier Transform (FFT) and Inverse FFT (IFFT) to efficiently compute DFT and reconstruct time-domain signals with reduced computational complexity.
• Utilizes the Wiener-Khinchin theorem to compute auto-correlation in the frequency domain via FFT, enabling efficient detection of time-delayed similarities.
• Integrates contrastive learning and frequency regularization to align representations across time and frequency domains, improving generalization and robustness.

Experimental results

Research questions

• RQ1 Can self-attention mechanisms be enhanced to preserve high-frequency information lost in traditional models?
• RQ2 How can periodic patterns in user behavior sequences be effectively modeled in the frequency domain?
• RQ3 Can combining time-domain and frequency-domain attention improve sequential recommendation performance?
• RQ4 To what extent does frequency regularization and contrastive learning improve alignment between time and frequency representations?
• RQ5 Does the proposed FEARec model outperform existing state-of-the-art methods on standard sequential recommendation benchmarks?

Key findings

• FEARec achieves state-of-the-art performance on four widely used benchmark datasets, outperforming existing SOTA models in sequential recommendation tasks.
• The model demonstrates significant gains in recommendation accuracy by explicitly capturing high-frequency user interaction patterns previously ignored by standard self-attention mechanisms.
• The integration of frequency-domain auto-correlation attention enables effective modeling of periodic user behavior trends, such as daily or weekly interaction cycles.
• Contrastive learning and frequency regularization successfully align representations across time and frequency domains, enhancing model robustness and generalization.
• The frequency ramp structure effectively balances low- and high-frequency information, preventing the model from acting as a pure low-pass filter.
• Empirical results show that FEARec consistently improves performance across multiple metrics, including Recall and NDCG, on all evaluated datasets.