[Paper Review] KuaiSim: A Comprehensive Simulator for Recommender Systems

Kesen Zhao, Shuchang Liu+6 more|arXiv (Cornell University)|Sep 22, 2023

Recommender Systems and Techniques7 citations

TL;DR

KuaiSim is a comprehensive reinforcement learning simulator for recommender systems that models multi-behavior user feedback, session-level interactions, and cross-session retention. It supports three task levels—request-level, whole-session, and cross-session—using real-world log data to pre-train user response, leave, and retention models, enabling robust, reproducible evaluation with built-in benchmarks and cross-dataset adaptability.

ABSTRACT

Reinforcement Learning (RL)-based recommender systems (RSs) have garnered considerable attention due to their ability to learn optimal recommendation policies and maximize long-term user rewards. However, deploying RL models directly in online environments and generating authentic data through A/B tests can pose challenges and require substantial resources. Simulators offer an alternative approach by providing training and evaluation environments for RS models, reducing reliance on real-world data. Existing simulators have shown promising results but also have limitations such as simplified user feedback, lacking consistency with real-world data, the challenge of simulator evaluation, and difficulties in migration and expansion across RSs. To address these challenges, we propose KuaiSim, a comprehensive user environment that provides user feedback with multi-behavior and cross-session responses. The resulting simulator can support three levels of recommendation problems: the request level list-wise recommendation task, the whole-session level sequential recommendation task, and the cross-session level retention optimization task. For each task, KuaiSim also provides evaluation protocols and baseline recommendation algorithms that further serve as benchmarks for future research. We also restructure existing competitive simulators on the KuaiRand Dataset and compare them against KuaiSim to future assess their performance and behavioral differences. Furthermore, to showcase KuaiSim's flexibility in accommodating different datasets, we demonstrate its versatility and robustness when deploying it on the ML-1m dataset.

Motivation & Objective

• Address the limitations of existing simulators in modeling multi-behavior user feedback and long-term user retention.
• Provide a unified simulation framework that supports three distinct recommendation task levels: request-level, whole-session, and cross-session.
• Ensure consistency with real-world data by pre-training user models on actual log data and enabling flexible data migration to other datasets.
• Establish standardized evaluation protocols and baseline algorithms for each task level to serve as benchmarks for future research.
• Improve simulator evaluation by enabling comparative analysis with existing simulators on the KuaiRand dataset and demonstrating robustness on ML-1m.

Proposed method

• Design a multi-level simulator architecture with three distinct task levels: list-wise recommendation (request-level), sequential recommendation (whole-session), and retention optimization (cross-session).
• Implement three core models: a user immediate response model for multi-behavior feedback (e.g., click, like, forward), a user leave model to detect session end, and a user retention model to predict return behavior.
• Pre-train all user behavior models using real-world log data from the KuaiRand dataset to ensure fidelity to real user behavior distributions.
• Integrate user sampling during simulation to maintain consistency with real-world data distribution and reduce distribution shift.
• Enable cross-dataset adaptability by structuring the framework to support different data formats, demonstrated on the ML-1m dataset.
• Provide standardized evaluation protocols and baseline algorithms for each task level to ensure reproducibility and benchmarking capability.

Experimental results

Research questions

• RQ1 How well can a simulator capture multi-behavior user feedback (e.g., click, like, forward) and long-term retention signals in a realistic way?
• RQ2 To what extent does a simulator trained on real log data maintain consistency with real-world user behavior distributions?
• RQ3 Can a unified simulator framework effectively support diverse recommendation tasks across different levels of abstraction (request, session, cross-session)?
• RQ4 How does the simulator’s performance vary with key hyperparameters such as slate size and maximum time steps?
• RQ5 How does KuaiSim compare to existing simulators in terms of behavioral fidelity and model performance on benchmark tasks?

Key findings

• The optimal slate size for model performance in KuaiSim is 20, with performance degrading when the slate size is too small (insufficient item coverage) or too large (excessive noise and user fatigue).
• The maximum time step of 20 yields the highest return in the simulation, with return day values stabilizing between 2.2 and 2.3 across other settings, indicating robustness to hyperparameter variation.
• KuaiSim demonstrates consistent and stable training performance across different parameter settings, confirming the reliability and robustness of the simulator’s training agent.
• The simulator successfully reproduces key user behaviors—immediate feedback, session termination, and cross-session retention—using real log data, enhancing its realism and fidelity.
• The framework is adaptable to other datasets, as demonstrated by successful deployment on the ML-1m dataset, confirming its flexibility and extensibility.
• Comparative evaluation on the KuaiRand dataset shows that KuaiSim outperforms or matches existing simulators in capturing behavioral dynamics, especially in modeling long-term retention signals.