[Paper Review] ReLoop: A Self-Correction Continual Learning Loop for Recommender Systems

Guohao Cai, Jieming Zhu+5 more|arXiv (Cornell University)|Apr 24, 2022

Recommender Systems and Techniques5 citations

TL;DR

ReLoop proposes a self-correction continual learning framework for recommender systems that reduces prediction errors from previous model versions during training. By introducing a customized loss that penalizes performance degradation on past predictions, ReLoop enables models to actively learn from their own mistakes, achieving consistent improvements in both offline benchmarks and online A/B tests across public and industrial datasets.

ABSTRACT

Deep learning-based recommendation has become a widely adopted technique in various online applications. Typically, a deployed model undergoes frequent re-training to capture users' dynamic behaviors from newly collected interaction logs. However, the current model training process only acquires users' feedbacks as labels, but fail to take into account the errors made in previous recommendations. Inspired by the intuition that humans usually reflect and learn from mistakes, in this paper, we attempt to build a self-correction learning loop (dubbed ReLoop) for recommender systems. In particular, a new customized loss is employed to encourage every new model version to reduce prediction errors over the previous model version during training. Our ReLoop learning framework enables a continual self-correction process in the long run and thus is expected to obtain better performance over existing training strategies. Both offline experiments and an online A/B test have been conducted to validate the effectiveness of ReLoop.

Motivation & Objective

• To address the limitation in existing recommender training loops that ignore past recommendation errors.
• To enable recommender models to actively learn from their own mistakes, inspired by human self-reflection and correction.
• To design a generic, model-agnostic framework that improves performance through continual self-correction in online training loops.
• To validate the effectiveness of self-correction in both offline evaluation and real-world online deployment.

Proposed method

• Introduces a self-correction module that compares current model predictions with those of the previous model version on the same data samples.
• Defines sample-wise prediction errors as the absolute difference between model scores and ground-truth labels (e.g., click/no-click).
• Proposes a customized loss function that penalizes the current model if its prediction error exceeds that of the previous model on any sample.
• Applies the loss during training to enforce that each new model version improves upon the previous one in terms of error reduction.
• Makes the framework model-agnostic, allowing integration with any existing recommendation model architecture.
• Uses both offline datasets and real industrial data for evaluation, including online A/B testing in a newsfeed system.

Experimental results

Research questions

• RQ1 Can a recommender system improve its performance by actively learning from its own past prediction errors?
• RQ2 How can a continual training loop be enhanced to incorporate feedback not only from user clicks but also from model errors in prior versions?
• RQ3 Can a generic, model-agnostic loss function be designed to enforce self-correction across successive model updates?
• RQ4 Does the self-correction mechanism lead to measurable improvements in both offline metrics and online business KPIs?

Key findings

• ReLoop achieves consistent performance gains over strong baselines on four public CTR prediction datasets, including improvements in AUC and logloss.
• The framework outperforms knowledge distillation and continual learning baselines by explicitly learning from past model errors rather than just mimicking past behavior.
• An online A/B test in a real industrial newsfeed system confirms that ReLoop improves key business metrics such as click-through rate and user engagement.
• The self-correction mechanism leads to a measurable reduction in prediction errors over time, demonstrating the effectiveness of the proposed loss function.