[Paper Review] Behavior Sequence Transformer for E-commerce Recommendation in Alibaba

Qiwei Chen, Huan Zhao+3 more|arXiv (Cornell University)|May 15, 2019

Recommender Systems and Techniques18 references34 citations

TL;DR

The paper introduces a behavior sequence transformer (BST) that uses a Transformer to model users’ click sequences for Taobao recommendations, showing offline AUC and online CTR gains over baselines and deploying in production.

ABSTRACT

Deep learning based methods have been widely used in industrial recommendation systems (RSs). Previous works adopt an Embedding&MLP paradigm: raw features are embedded into low-dimensional vectors, which are then fed on to MLP for final recommendations. However, most of these works just concatenate different features, ignoring the sequential nature of users' behaviors. In this paper, we propose to use the powerful Transformer model to capture the sequential signals underlying users' behavior sequences for recommendation in Alibaba. Experimental results demonstrate the superiority of the proposed model, which is then deployed online at Taobao and obtain significant improvements in online Click-Through-Rate (CTR) comparing to two baselines.

Motivation & Objective

• Motivate the need to capture sequential signals in user behavior for better CTR predictions in e-commerce.
• Propose a Transformer-based architecture (BST) to model behavior sequences on top of an embedding&MLP framework.
• Show offline and online gains of BST against strong baselines and discuss production deployment in Taobao.

Proposed method

• Embed diverse features into low-dimensional vectors using an embedding layer.
• Apply a Transformer (self-attention) to learn deeper representations of items in the user behavior sequence.
• Concatenate sequence representations with Other Features and pass through a three-layer MLP for CTR prediction.
• Incorporate item position using positional features with a custom position encoding.
• Train with cross-entropy loss and use dropout and LayerNorm for regularization.

Experimental results

Research questions

• RQ1 Does incorporating Transformer-based sequential modeling improve CTR prediction over WDL and DIN baselines?
• RQ2 How does BST perform offline (AUC) and online (CTR) compared to baselines and sequentially-enhanced WDL?
• RQ3 What is the impact of the number of Transformer blocks on performance and production efficiency?
• RQ4 Is BST feasible for large-scale production in terms of latency (RT) and deployment?

Key findings

• BST improves offline AUC to 0.7894, outperforming WDL and DIN baselines.
• BST yields the largest online CTR gain of +7.57% over the control group at b=1.
• Sequential information helps WDL when augmented with simple averaging (WDL(+Seq)); BST with Transformer captures richer sequential signals.
• BST with a single Transformer block (b=1) achieves best offline AUC and maintains competitive latency for production.
• BST was deployed in Taobao’s rank stage, providing service to hundreds of millions of users and maintaining acceptable average response times.