[Paper Review] BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer

Fei Sun, Jun Liu+5 more|arXiv (Cornell University)|Apr 14, 2019

Recommender Systems and Techniques56 references275 citations

TL;DR

BERT4Rec introduces a bidirectional Transformer-based model for sequential recommendation and uses a Cloze (masked language modeling) objective to train without leakage, achieving state-of-the-art results across four datasets.

ABSTRACT

Modeling users' dynamic and evolving preferences from their historical behaviors is challenging and crucial for recommendation systems. Previous methods employ sequential neural networks (e.g., Recurrent Neural Network) to encode users' historical interactions from left to right into hidden representations for making recommendations. Although these methods achieve satisfactory results, they often assume a rigidly ordered sequence which is not always practical. We argue that such left-to-right unidirectional architectures restrict the power of the historical sequence representations. For this purpose, we introduce a Bidirectional Encoder Representations from Transformers for sequential Recommendation (BERT4Rec). However, jointly conditioning on both left and right context in deep bidirectional model would make the training become trivial since each item can indirectly "see the target item". To address this problem, we train the bidirectional model using the Cloze task, predicting the masked items in the sequence by jointly conditioning on their left and right context. Comparing with predicting the next item at each position in a sequence, the Cloze task can produce more samples to train a more powerful bidirectional model. Extensive experiments on four benchmark datasets show that our model outperforms various state-of-the-art sequential models consistently.

Motivation & Objective

• Motivate modeling dynamic, order-uncertain user preferences from historical interactions.
• Propose a bidirectional Transformer-based architecture for sequential recommendation.
• Address training challenges from bidirectionality with a Cloze (masked language model) objective.
• Demonstrate superior performance over state-of-the-art baselines on multiple datasets.
• Provide ablation studies to isolate the contributions of bidirectionality and the Cloze objective.

Proposed method

• Adopt Bidirectional Encoder Representations from Transformers (BERT) to model user behavior sequences.
• Use a Transformer with multi-head self-attention to capture dependencies across the entire sequence.
• Train with a Cloze objective by masking a subset of items and predicting them from left and right context to avoid information leakage.
• Append a final [mask] token at test time to predict the next item consistent with the sequential task.
• Share item embeddings between input and output layers to reduce parameters and prevent overfitting.
• Conduct ablation studies to separate effects of bidirectionality and Cloze objective.

Experimental results

Research questions

• RQ1 Does bidirectional self-attention improve sequential recommendation over unidirectional models?
• RQ2 What is the effect of the Cloze objective on training efficiency and recommendation performance?
• RQ3 How much do bidirectionality and the Cloze objective each contribute to gains over strong baselines?
• RQ4 How does BERT4Rec perform compared with existing state-of-the-art sequential models across diverse datasets?

Key findings

• BERT4Rec consistently outperforms state-of-the-art baselines across four benchmark datasets.
• Ablation shows bidirectional modeling and the Cloze objective both contribute to performance gains, with bidirectionality providing substantial benefit.
• The model achieves notable improvements in HR@k, NDCG@k, and MRR across multiple datasets.
• The Cloze objective yields more training samples per sequence, enabling a more powerful bidirectional representation.
• Visual analysis of attention patterns indicates meaningful cross-position dependencies captured by the bidirectional Transformer.