[Paper Review] Toward Simulating Environments in Reinforcement Learning Based Recommendations

Xiangyu Zhao, Long Xia+3 more|arXiv (Cornell University)|Jun 27, 2019

Advanced Bandit Algorithms Research46 references17 citations

TL;DR

This paper proposes RecSimu, a GAN-based user simulator that generates realistic user feedback sequences from historical e-commerce logs to pre-train and evaluate reinforcement learning (RL)-based recommender systems. By jointly training a generator to model complex item distribution and a discriminator to predict user feedback, the simulator improves behavior prediction performance significantly over baselines, enabling efficient offline RL algorithm development.

ABSTRACT

With the recent advances in Reinforcement Learning (RL), there have been tremendous interests in employing RL for recommender systems. However, directly training and evaluating a new RL-based recommendation algorithm needs to collect users' real-time feedback in the real system, which is time and efforts consuming and could negatively impact on users' experiences. Thus, it calls for a user simulator that can mimic real users' behaviors where we can pre-train and evaluate new recommendation algorithms. Simulating users' behaviors in a dynamic system faces immense challenges -- (i) the underlining item distribution is complex, and (ii) historical logs for each user are limited. In this paper, we develop a user simulator base on Generative Adversarial Network (GAN). To be specific, the generator captures the underlining distribution of users' historical logs and generates realistic logs that can be considered as augmentations of real logs; while the discriminator not only distinguishes real and fake logs but also predicts users' behaviors. The experimental results based on real-world e-commerce data demonstrate the effectiveness of the proposed simulator.

Motivation & Objective

• To address the high cost and risk of online A/B testing for RL-based recommender systems by enabling pre-training and evaluation in a simulated environment.
• To model complex, high-dimensional item distribution patterns in sparse user historical logs, which are challenging for traditional generative models.
• To generate realistic user feedback sequences (click, skip, purchase) that mimic real user behavior for robust RL algorithm evaluation.
• To overcome data scarcity per user by leveraging adversarial training to augment limited user logs with synthetic but realistic sequences.
• To validate the simulator’s effectiveness in improving downstream user behavior prediction performance using real-world e-commerce data.

Proposed method

• A Generative Adversarial Network (GAN) framework is employed, where the generator learns the underlying distribution of user interaction sequences from historical logs.
• The generator produces synthetic user feedback sequences that are indistinguishable from real logs, serving as data augmentation for training RL agents.
• The discriminator is trained to distinguish real logs from fake logs and simultaneously predict user feedback (click/skip/purchase) for a given item in a sequence.
• The discriminator leverages both supervised learning (on feedback labels) and unsupervised learning (on log realism), combining multiple objectives to improve behavior modeling.
• The generator and discriminator are trained in an adversarial manner, with the generator improving its realism and the discriminator improving its ability to detect fake logs and predict feedback.
• The framework is designed to handle sparse user data by learning from aggregated user behavior patterns across the entire user base, not per-individual user data.

Experimental results

Research questions

• RQ1 Can a GAN-based simulator generate user feedback sequences that are realistic enough to improve the training and evaluation of RL-based recommender systems?
• RQ2 How effective is the proposed simulator in improving user behavior prediction performance compared to existing baselines?
• RQ3 To what extent can the simulator mitigate data scarcity issues in user logs, especially for rare feedback types like purchases?
• RQ4 Can the discriminator’s feedback prediction capability be leveraged to improve the realism and utility of generated logs?
• RQ5 How does the simulator perform when used to pre-train RL agents before online deployment?

Key findings

• The proposed RecSimu simulator significantly improves user behavior prediction performance over strong baselines, demonstrating its effectiveness in generating realistic user feedback sequences.
• The GAN-based approach outperforms traditional data augmentation and generative models in capturing complex item distribution patterns in sparse user logs.
• The discriminator’s dual role—distinguishing real vs. fake logs and predicting feedback—leads to more robust and realistic behavior modeling.
• The simulator enables effective pre-training of RL agents, reducing the need for costly and risky online A/B testing in production systems.
• The framework shows strong generalization on real-world e-commerce data, validating its potential for practical deployment in recommendation systems.
• The ablation study confirms that both the generator’s distribution learning and the discriminator’s feedback prediction are critical components for the simulator’s success.