[Paper Review] Improving Multi-Interest Network with Stable Learning

Zhaocheng Liu, Yingtao Luo+5 more|arXiv (Cornell University)|Jul 14, 2022

Recommender Systems and Techniques4 citations

TL;DR

This paper proposes DESMIL, a deep stable multi-interest learning framework that improves sequential recommendation by reducing spurious correlations from subtle dependencies among user interests. By using Hilbert-Schmidt Independence Criterion (HSIC) to measure inter-interest dependence and learning sample weights to suppress irrelevant dependencies, DESMIL achieves superior generalization on out-of-distribution data and outperforms state-of-the-art models across public, industrial, and synthetic datasets.

ABSTRACT

Modeling users' dynamic preferences from historical behaviors lies at the core of modern recommender systems. Due to the diverse nature of user interests, recent advances propose the multi-interest networks to encode historical behaviors into multiple interest vectors. In real scenarios, the corresponding items of captured interests are usually retrieved together to get exposure and collected into training data, which produces dependencies among interests. Unfortunately, multi-interest networks may incorrectly concentrate on subtle dependencies among captured interests. Misled by these dependencies, the spurious correlations between irrelevant interests and targets are captured, resulting in the instability of prediction results when training and test distributions do not match. In this paper, we introduce the widely used Hilbert-Schmidt Independence Criterion (HSIC) to measure the degree of independence among captured interests and empirically show that the continuous increase of HSIC may harm model performance. Based on this, we propose a novel multi-interest network, named DEep Stable Multi-Interest Learning (DESMIL), which tries to eliminate the influence of subtle dependencies among captured interests via learning weights for training samples and make model concentrate more on underlying true causation. We conduct extensive experiments on public recommendation datasets, a large-scale industrial dataset and the synthetic datasets which simulate the out-of-distribution data. Experimental results demonstrate that our proposed DESMIL outperforms state-of-the-art models by a significant margin. Besides, we also conduct comprehensive model analysis to reveal the reason why DESMIL works to a certain extent.

Motivation & Objective

• To address the instability of multi-interest networks in out-of-distribution (OOD) settings due to spurious correlations from dependencies among captured interests.
• To investigate how increasing inter-interest dependence, measured via HSIC, correlates with declining model performance during training.
• To develop a method that encourages models to focus on true causal patterns rather than spurious dependencies in user behavior sequences.
• To improve generalization in sequential recommendation by learning sample weights that reduce the influence of subtle inter-interest dependencies.

Proposed method

• Introduces HSIC as a statistical measure to quantify the degree of independence among multiple user interest vectors captured by a multi-interest network.
• Proposes a sample-weighting mechanism in the loss function to dynamically down-weight training samples that contribute to increasing inter-interest dependence.
• Designs a regularizer based on HSIC to encourage the model to minimize spurious correlations between unrelated interests during training.
• Employs a soft, differentiable weighting strategy that allows the model to focus on causally relevant interactions while suppressing noise from correlated but irrelevant interests.
• Uses early stopping and adaptive learning rates to balance convergence and generalization, with a fixed λ=1.0 for stable training.
• Tunes the number of interest vectors (c) per user, finding optimal values of c=2 for Book, CDs and Vinyl, and Movies and TV, and c=6 for the industrial dataset.

Experimental results

Research questions

• RQ1 How does the degree of inter-interest dependence, measured by HSIC, evolve during training, and does it correlate with performance degradation?
• RQ2 Can controlling inter-interest dependence via sample weighting improve model generalization under distribution shift?
• RQ3 Does reducing spurious correlations between unrelated interests lead to better performance on out-of-distribution test data?
• RQ4 How robust is the proposed method to hyperparameter choices such as learning rate and number of interest vectors?

Key findings

• The continuous rise of HSIC during training correlates with performance saturation or decline, indicating that increasing inter-interest dependence harms model generalization.
• DESMIL achieves significant performance gains over state-of-the-art models on public benchmarks, including Book, CDs and Vinyl, and Movies and TV datasets.
• On a large-scale industrial dataset, DESMIL demonstrates superior robustness and generalization, especially under distribution shift.
• The model achieves better performance on synthetic OOD data, confirming its ability to learn causal patterns rather than spurious correlations.
• The histogram of sample weights shows that most samples are assigned weights near 1.0, while a small subset receives near-zero weights, indicating effective marginalization of noisy or irrelevant interactions.
• The method shows stable performance across different hyperparameter settings, particularly in the range of λ=1.0 and optimal c values, reducing the burden of hyperparameter tuning.