[Paper Review] Reinforcing User Retention in a Billion Scale Short Video Recommender System

Qingpeng Cai, Shuchang Liu+7 more|arXiv (Cornell University)|Feb 3, 2023

Recommender Systems and Techniques4 citations

TL;DR

This paper proposes RLUR, a reinforcement learning framework that optimizes user retention in a billion-scale short video recommender system by modeling retention as an infinite-horizon request-based Markov Decision Process. It addresses long-delayed, uncertain, and biased retention signals through a novel normalization technique and a dual-critic training objective, achieving consistent improvements in user retention and DAU in live deployment on Kuaishou.

ABSTRACT

Recently, short video platforms have achieved rapid user growth by recommending interesting content to users. The objective of the recommendation is to optimize user retention, thereby driving the growth of DAU (Daily Active Users). Retention is a long-term feedback after multiple interactions of users and the system, and it is hard to decompose retention reward to each item or a list of items. Thus traditional point-wise and list-wise models are not able to optimize retention. In this paper, we choose reinforcement learning methods to optimize the retention as they are designed to maximize the long-term performance. We formulate the problem as an infinite-horizon request-based Markov Decision Process, and our objective is to minimize the accumulated time interval of multiple sessions, which is equal to improving the app open frequency and user retention. However, current reinforcement learning algorithms can not be directly applied in this setting due to uncertainty, bias, and long delay time incurred by the properties of user retention. We propose a novel method, dubbed RLUR, to address the aforementioned challenges. Both offline and live experiments show that RLUR can significantly improve user retention. RLUR has been fully launched in Kuaishou app for a long time, and achieves consistent performance improvement on user retention and DAU.

Motivation & Objective

• To address the challenge of optimizing long-term user retention in short video recommendation, which cannot be effectively captured by traditional point-wise or list-wise models.
• To model user retention as an infinite-horizon request-based Markov Decision Process where the goal is to minimize cumulative returning time between sessions.
• To overcome the challenges of uncertainty, bias, and long delay in retention feedback that hinder direct application of standard reinforcement learning algorithms.
• To develop a scalable, stable, and effective RL-based policy that improves app open frequency and user retention in real-world deployment.
• To validate the method through offline evaluation and long-term live A/B experiments on a production-scale platform.

Proposed method

• Formulate the recommendation problem as an infinite-horizon request-based MDP, where the agent (recommender) selects ranking weights to optimize long-term user return.
• Introduce a novel normalization technique that predicts returning time and reduces variance in retention reward signals, improving training stability.
• Design a dual-critic training objective that combines a retention critic (for session-level return) and an immediate reward critic (for per-request feedback like watch time and likes).
• Use a continuous action space of 8-dimensional vectors to represent ranking weights for eight different feedback signals (e.g., watch time, likes, comments).
• Apply a Gaussian policy network (Actor) that outputs mean and variance for action sampling, enabling exploration during training.
• Train the policy using a weighted loss combining the retention critic and immediate reward critic, with hyperparameters tuned for stability and performance.

Figure 1. The infinite horizon request-based MDP and the framework of RLUR

Experimental results

Research questions

• RQ1 Can reinforcement learning be effectively applied to optimize long-term user retention in a real-world short video recommender system?
• RQ2 How can the long delay, uncertainty, and bias in user retention feedback be mitigated to enable stable RL training?
• RQ3 Does a dual-critic approach combining session-level retention and per-request feedback improve policy learning compared to single-critic or naive RL methods?
• RQ4 Can the proposed normalization technique reduce variance in retention reward signals and improve training convergence?
• RQ5 What is the long-term impact of the RLUR policy on key business metrics like DAU and user retention in live deployment?

Key findings

• RLUR achieved a 1.892 returning time (in hours) in offline evaluation, significantly outperforming baseline methods like CEM (2.036) and TD3 (2.009).
• In live experiments, RLUR increased app open frequency by 0.450% after 100 days, with a 0.2% gap in DAU compared to the CEM baseline.
• User retention at 1st day improved by 0.053% and at 7th day by 0.063% over the baseline, with statistical significance in the context of billion-scale platforms.
• The performance gains were consistent and progressive, with sharp improvements observed from day 80 to day 100, indicating stable and convergent training.
• RLUR demonstrated sustained improvement over time, with no degradation, confirming its robustness and scalability in production deployment.
• The ablation study confirmed that the proposed normalization and dual-critic components were essential, as RLUR (naive) with γ=0.9 performed worse than the full RLUR model.

Figure 2. Live performance gap of each day.