[Paper Review] Next Item Recommendation with Self-Attention

Shuai Zhang, Yi Tay+2 more|arXiv (Cornell University)|Aug 20, 2018

Recommender Systems and TechniquesComputer Science44 references51 citations

TL;DR

AttRec proposes a self-attention based sequential recommender that models short-term user intents via self-attention and long-term preferences via metric learning, achieving state-of-the-art results across diverse datasets.

ABSTRACT

In this paper, we propose a novel sequence-aware recommendation model. Our model utilizes self-attention mechanism to infer the item-item relationship from user's historical interactions. With self-attention, it is able to estimate the relative weights of each item in user interaction trajectories to learn better representations for user's transient interests. The model is finally trained in a metric learning framework, taking both short-term and long-term intentions into consideration. Experiments on a wide range of datasets on different domains demonstrate that our approach outperforms the state-of-the-art by a wide margin.

Motivation & Objective

• Motivate the need for modeling item-item interactions within a user’s history for better sequential recommendations.
• Introduce a self-attention based module to capture short-term user intents from recent interactions.
• Couple the self-attentive representations with a collaborative metric learning component to model long-term user preferences.
• Evaluate AttRec comprehensively on dense and sparse datasets and analyze hyper-parameters and attention behavior.

Proposed method

• Use a self-attention module where query, key, and value are derived from the last L interacted items to produce an attentive short-term representation of user intent.
• Incorporate time information through positional (time) embeddings added to query and key prior to projection.
• Represent long-term user preference and item embeddings via latent factors U (users) and V (items) and model interaction with Euclidean distance.
• Combine short-term (m_t via self-attention) and long-term signals in a unified score y_{t+1} = ω ||U_u - V_{H^u_{t+1}}||^2 + (1-ω) ||m^u_t - X^u_{t+1}||^2.
• Train with a pairwise margin-based hinge loss over positive and negative items and regularize with L2, using adaptive gradient optimization.
• Set L (short-term window), T (next items), and other hyper-parameters via grid search; evaluate with HR@50 and MRR across multiple datasets.

Experimental results

Research questions

• RQ1 Does AttRec achieve state-of-the-art performance on diverse sequential recommendation datasets?
• RQ2 How does the self-attention component affect modeling of short-term intents and overall ranking performance?
• RQ3 What is the impact of key hyper-parameters (e.g., L, aggregation method, ω, d) on performance?
• RQ4 Can AttRec perform well on both dense and sparse datasets and handle short sequences effectively?

Key findings

• AttRec consistently outperforms all baselines on 12 benchmark datasets across HR@50 and MRR metrics.
• In ablations, including self-attention improves performance over a non-attentive variant, though the model remains competitive without it on some datasets.
• Self-attention attention maps are interpretable, showing varying weights across past actions rather than simply favoring the most recent item.
• Aggregation method for combining attention outputs affects performance, with mean aggregation typically used as a baseline.
• The model integrates short-term intents (via self-attention) and long-term preferences (via latent factors) to achieve improved recommendations.