[Paper Review] Multi-Interest-Aware User Modeling for Large-Scale Sequential Recommendations

Jianxun Lian, Iyad Batal+5 more|arXiv (Cornell University)|Feb 18, 2021

Recommender Systems and Techniques35 references4 citations

TL;DR

This paper proposes Sequential User Matrix (SUM), a multi-channel memory network that models users' diverse, evolving interests in sequential recommendations by learning interest-level and instance-level attention, local proximity debuff, and highway connections. SUM outperforms state-of-the-art models on two real-world datasets and achieves 1.46% higher click yield and 1.32% higher revenue in online A/B tests on Bing Native Ads.

ABSTRACT

Precise user modeling is critical for online personalized recommendation services. Generally, users' interests are diverse and are not limited to a single aspect, which is particularly evident when their behaviors are observed for a longer time. For example, a user may demonstrate interests in cats/dogs, dancing and food \& delights when browsing short videos on Tik Tok; the same user may show interests in real estate and women's wear in her web browsing behaviors. Traditional models tend to encode a user's behaviors into a single embedding vector, which do not have enough capacity to effectively capture her diverse interests. This paper proposes a Sequential User Matrix (SUM) to accurately and efficiently capture users' diverse interests. SUM models user behavior with a multi-channel network, with each channel representing a different aspect of the user's interests. User states in different channels are updated by an \emph{erase-and-add} paradigm with interest- and instance-level attention. We further propose a local proximity debuff component and a highway connection component to make the model more robust and accurate. SUM can be maintained and updated incrementally, making it feasible to be deployed for large-scale online serving. We conduct extensive experiments on two datasets. Results demonstrate that SUM consistently outperforms state-of-the-art baselines.

Motivation & Objective

• To address the limitation of single-vector user representations in capturing diverse, dynamic user interests from long behavior sequences.
• To design a scalable, incrementally updatable model suitable for real-time online serving with sub-10ms latency.
• To improve sequential recommendation performance by modeling multiple interests simultaneously through a multi-channel architecture.
• To enhance representation learning with instance-level attention and robustness via local proximity debuff and highway connections.

Proposed method

• SUM uses a multi-channel memory network where each channel represents a distinct user interest, enabling parallel modeling of diverse preferences.
• It applies interest-level and instance-level attention to dynamically route user behavior events to relevant channels based on relevance and context.
• The erase-and-add update mechanism in each channel uses attention to selectively overwrite or preserve memory states, improving precision in interest tracking.
• A local proximity debuff component suppresses noise from temporally close but semantically irrelevant events, enhancing robustness.
• Highway connections between channels allow information flow and improve gradient propagation, supporting deeper and more accurate modeling.
• The model supports incremental updates, enabling efficient online inference with low latency, suitable for large-scale deployment.

Experimental results

Research questions

• RQ1 Can a multi-channel memory network effectively model multiple, diverse user interests in long sequential behavior data?
• RQ2 How does instance-level attention improve the accuracy of interest representation compared to interest-level attention alone?
• RQ3 To what extent does the local proximity debuff mechanism reduce noise from temporally proximate but irrelevant events?
• RQ4 How do highway connections between channels affect model performance and training stability?
• RQ5 Can SUM achieve superior performance and scalability in real-world industrial online serving environments?

Key findings

• SUM consistently outperforms state-of-the-art baselines on two real-world datasets, demonstrating superior performance in sequential recommendation tasks.
• On the Amazon and Taobao datasets, SUM achieves significant gains in AUC and NDCG, with improvements over existing models in both metrics.
• The model's online A/B test on Bing Native Ads showed a 1.46% increase in click yield and a 1.32% increase in revenue compared to the GRU-based production model.
• The ablation study confirms that all components—instance-level attention, debuff, and highway connections—contribute positively to model performance.
• The incremental update mechanism enables SUM to maintain low inference latency, making it suitable for large-scale online deployment.