[Paper Review] Multi-Interest Network with Dynamic Routing for Recommendation at Tmall

Chao Li, Zhiyuan Liu+8 more|arXiv (Cornell University)|Apr 17, 2019

Recommender Systems and Techniques28 references51 citations

TL;DR

MIND represents user interests with multiple vectors learned via dynamic routing, improving matching-stage recommendations and deployed at Tmall homepage for Mobile App. It outperforms baselines on public and industrial data.

ABSTRACT

Industrial recommender systems usually consist of the matching stage and the ranking stage, in order to handle the billion-scale of users and items. The matching stage retrieves candidate items relevant to user interests, while the ranking stage sorts candidate items by user interests. Thus, the most critical ability is to model and represent user interests for either stage. Most of the existing deep learning-based models represent one user as a single vector which is insufficient to capture the varying nature of user's interests. In this paper, we approach this problem from a different view, to represent one user with multiple vectors encoding the different aspects of the user's interests. We propose the Multi-Interest Network with Dynamic routing (MIND) for dealing with user's diverse interests in the matching stage. Specifically, we design a multi-interest extractor layer based on capsule routing mechanism, which is applicable for clustering historical behaviors and extracting diverse interests. Furthermore, we develop a technique named label-aware attention to help learn a user representation with multiple vectors. Through extensive experiments on several public benchmarks and one large-scale industrial dataset from Tmall, we demonstrate that MIND can achieve superior performance than state-of-the-art methods for recommendation. Currently, MIND has been deployed for handling major online traffic at the homepage on Mobile Tmall App.

Motivation & Objective

• Motivate modeling diverse user interests to improve the matching stage in industrial recommender systems.
• Propose a multi-interest extractor layer based on dynamic routing to cluster user behaviors into multiple interest vectors.
• Introduce a label-aware attention mechanism to align target items with multiple user interest vectors.
• Demonstrate effectiveness on public benchmarks and a large-scale Tmall industrial dataset and discuss deployment for billion-scale traffic.

Proposed method

• Represent a user with K interest vectors V_u from historical behaviors via a multi-interest extractor layer that uses dynamic routing to form interest capsules.
• Use a shared bilinear mapping S and Behavior-to-Interest (B2I) dynamic routing to compute routing logits and form interest capsules.
• Activate a label-aware attention layer to weight and combine interest capsules when evaluating a target item.
• Train with a sampled softmax objective to handle billions of items and optimize using Adam.
• At serving time, use multiple user vectors in approximate nearest neighbor retrieval to select top-N candidate items for matching.

Experimental results

Research questions

• RQ1 Can multiple interest representations better capture diverse user preferences than a single vector in industrial-scale matching?
• RQ2 Does dynamic routing improve the clustering of historical behaviors into meaningful interest capsules for recommendation?
• RQ3 How effective is a label-aware attention mechanism in selecting and weighting user interests for a given target item?
• RQ4 What is the impact of the number of interests (K) on offline and online performance in billion-scale settings?

Key findings

• MIND with multiple interests (K>1) consistently outperforms baselines (WALS, YouTube DNN, MaxMF variants) on Amazon Books and TmallData in HitRate@N, with larger gains when user diversity is higher.
• Dynamic routing–based multi-interest extraction yields better performance than simple pooling (e.g., average pooling) and than single-vector approaches like YouTube DNN.
• Online A/B tests show MIND with 5–7 interests achieves the best CTR, outperforming item-based CF and YouTube DNN; increasing beyond 7 yields diminishing returns.
• Label-aware attention enables the target item to select and weight the most relevant interests, contributing to improved matching accuracy.