[Paper Review] Model-Based Reinforcement Learning with Adversarial Training for Online Recommendation

Xueying Bai, Jian Guan+1 more|arXiv (Cornell University)|Nov 10, 2019

Reinforcement Learning in Robotics40 references42 citations

TL;DR

Introduces IRecGAN, a model-based RL framework that uses adversarial training to learn user behavior and policy from offline data for online recommendation, improving sample efficiency and reducing bias.

ABSTRACT

Reinforcement learning is well suited for optimizing policies of recommender systems. Current solutions mostly focus on model-free approaches, which require frequent interactions with the real environment, and thus are expensive in model learning. Offline evaluation methods, such as importance sampling, can alleviate such limitations, but usually request a large amount of logged data and do not work well when the action space is large. In this work, we propose a model-based reinforcement learning solution which models user-agent interaction for offline policy learning via a generative adversarial network. To reduce bias in the learned model and policy, we use a discriminator to evaluate the quality of generated data and scale the generated rewards. Our theoretical analysis and empirical evaluations demonstrate the effectiveness of our solution in learning policies from the offline and generated data.

Motivation & Objective

• Motivate learning recommender policies from sparse, offline logged data to maximize long-term user satisfaction.
• Propose a model-based RL framework that explicitly models user behavior and interactions.
• Mitigate model bias and unstable policy updates via adversarial training and reward scaling.
• Demonstrate theoretical bias-control benefits and empirical efficiency over baselines.

Proposed method

• Model the environment as a user behavior model U and an agent A that interact to generate recommendation sequences.
• Use a recurrent neural network to represent user state s^u_t and agent state s^a_t, with softmax-based click and reward predictions (Eq. 1–2).
• Learn the user model U from offline data by maximum likelihood, jointly predicting clicks and rewards (Eq. 3).
• Apply policy gradient (REINFORCE) for the agent A, using both offline and generated data (g) to update the policy (Eq. 9–10).
• Introduce a discriminator D to distinguish real vs generated sequences and use its output to (i) debias the user model and (ii) scale generated rewards, enabling adversarial training (Eq. 5–8).
• Compute a sequence generation score q_D for partial sequences via Monte-Carlo rollouts to guide both U and A (Eq. 6).

Experimental results

Research questions

• RQ1 Can offline data be effectively leveraged to learn a high-quality policy for online recommendations through a model-based RL framework?
• RQ2 Does adversarial training reduce bias in the learned user model and improve policy learning in offline-to-online settings?
• RQ3 How does incorporating a discriminator to scale generated rewards affect value estimation and stability of policy updates?
• RQ4 What is the impact of combining offline and generated data on policy performance and sample efficiency?
• RQ5 How does the proposed approach perform relative to model-free baselines and other offline evaluation methods in simulated environments?

Key findings

• IRecGAN achieves better sample efficiency and competitive or superior performance compared to baselines in simulated online evaluation.
• Adversarial training helps the user model U capture real data patterns, reducing bias and stabilizing policy learning.
• The discriminator-based reward scaling reduces bias in value estimation for the generated data, improving policy gradients.
• Combining offline data with generated samples enables effective learning when online interactions are costly.
• Agent updates benefit from sequence-generation objectives that align generated sequences with real data distributions, matching or exceeding baselines under various data logging policies.