[Paper Review] MetaSelector: Meta-Learning for Recommendation with User-Level Adaptive Model Selection

Mi Luo, Fei Chen+5 more|arXiv (Cornell University)|Jan 22, 2020

Recommender Systems and Techniques37 references4 citations

TL;DR

MetaSelector proposes a meta-learning framework for user-level adaptive model selection in recommender systems, where a meta-learner selects the optimal recommendation model per user based on their historical data. It achieves state-of-the-art performance with significant improvements in AUC and LogLoss over single-model baselines and sample-level selectors, especially on heterogeneous user populations.

ABSTRACT

Recommender systems often face heterogeneous datasets containing highly personalized historical data of users, where no single model could give the best recommendation for every user. We observe this ubiquitous phenomenon on both public and private datasets and address the model selection problem in pursuit of optimizing the quality of recommendation for each user. We propose a meta-learning framework to facilitate user-level adaptive model selection in recommender systems. In this framework, a collection of recommenders is trained with data from all users, on top of which a model selector is trained via meta-learning to select the best single model for each user with the user-specific historical data. We conduct extensive experiments on two public datasets and a real-world production dataset, demonstrating that our proposed framework achieves improvements over single model baselines and sample-level model selector in terms of AUC and LogLoss. In particular, the improvements may lead to huge profit gain when deployed in online recommender systems.

Motivation & Objective

• To address the challenge of model heterogeneity in recommender systems, where no single model performs best for all users.
• To improve personalized recommendation quality by enabling user-level adaptive model selection rather than relying on global or sample-level selection.
• To formulate the model selection problem as a meta-learning task, leveraging fast adaptation to user-specific data patterns.
• To demonstrate the effectiveness of meta-learned model selection in real-world and public datasets, showing consistent gains in AUC and LogLoss.
• To validate that user-level adaptation via meta-learning leads to more stable and fair performance distributions across users.

Proposed method

• MetaSelector employs an optimization-based meta-learning approach, specifically adapting the MAML framework to model selection.
• Each user's data is treated as a task, with support and query sets used in episodic training to simulate user-specific adaptation.
• The model selector is a 200-200-200 MLP that outputs a probability distribution over base recommendation models for each user.
• During meta-training, the selector is updated via gradient descent using the loss from the adapted model on the query set, enabling fast adaptation to new users.
• The framework uses Meta-SGD to learn the inner learning rate adaptively, improving generalization across user tasks.
• During inference, the final prediction is a weighted average of the selected models' outputs, with weights determined by the meta-learner.

Experimental results

Research questions

• RQ1 RQ1: Can user-level model selection via meta-learning outperform single-model baselines and sample-level model selectors in terms of AUC and LogLoss?
• RQ2 RQ2: How does MetaSelector distribute prediction errors across users, and does it achieve a more consistent and fair performance profile?
• RQ3 RQ3: Is in-task adaptation crucial for effective user-specific model selection, and how does it compare to simplified variants?
• RQ4 RQ4: How do the performance gains of MetaSelector translate into real-world business impact, particularly in production environments?

Key findings

• MetaSelector achieves significant improvements in AUC and LogLoss over single-model baselines and sample-level model selectors on both public (Movielens, Amazon) and real-world production datasets.
• The framework reduces the variance of LogLoss across users, resulting in a more concentrated and fair performance distribution, as shown by kernel density estimation on Movielens-1m.
• User-level model selection outperforms sample-level selection in practice, despite sample-level perfect selector having a higher theoretical upper bound, indicating that user-level characteristics are more discriminative.
• The simplified version of MetaSelector, which omits in-task adaptation, shows a slight performance drop, confirming the importance of per-user adaptation in the meta-learning process.
• RelaImpr analysis shows that MetaSelector's improvements can lead to substantial profit gains when deployed in online recommender systems, highlighting its real-world business value.