[Paper Review] Generative Adversarial User Model for Reinforcement Learning Based Recommendation System

Xinshi Chen, Shuang Li+4 more|arXiv (Cornell University)|Dec 27, 2018

Advanced Bandit Algorithms ResearchDecision Sciences28 references62 citations

TL;DR

The paper proposes a model-based RL framework for recommender systems using a generative adversarial user model to learn user behavior dynamics and rewards, and introduces cascading DQN for efficient combinatorial recommendations.

ABSTRACT

There are great interests as well as many challenges in applying reinforcement learning (RL) to recommendation systems. In this setting, an online user is the environment; neither the reward function nor the environment dynamics are clearly defined, making the application of RL challenging. In this paper, we propose a novel model-based reinforcement learning framework for recommendation systems, where we develop a generative adversarial network to imitate user behavior dynamics and learn her reward function. Using this user model as the simulation environment, we develop a novel Cascading DQN algorithm to obtain a combinatorial recommendation policy which can handle a large number of candidate items efficiently. In our experiments with real data, we show this generative adversarial user model can better explain user behavior than alternatives, and the RL policy based on this model can lead to a better long-term reward for the user and higher click rate for the system.

Motivation & Objective

• Motivate reinforcement learning for long-term user engagement in recommendations where the environment is unknown and rewards are implicit.
• Propose a unified, model-based RL framework that learns both a user behavior model and a reward function simultaneously.
• Develop a generative adversarial training procedure to jointly estimate user dynamics and rewards.
• Introduce a cascading DQN policy to efficiently select top-k items from large candidate pools.

Proposed method

• Formulate user behavior as a reward-maximizing, sequential decision process where the user selects an item to maximize her reward, with history-dependent rewards.
• Parameterize user state as an embedding of past clicked items; define reward function r(s,a) using learned neural network components.
• Model training via Generative Adversarial Training: simultaneous optimization of a behavior model φ and a reward r in a mini-max setup (GAN-like).
• Provide a closed-form solution under entropy regularization for stable initialization and training when learning φ and r.
• Develop Cascading Q-Networks (and associated loss) to efficiently optimize a combinatorial action space by breaking the max over subsets into a cascade of smaller Q-functions.
• Train and evaluate policies using a simulated environment built from the GAN user model to predict long-term rewards and click-through rates.

Experimental results

Research questions

• RQ1 Can the GAN-based user model better predict user behavior and rewards than baselines?
• RQ2 Does the model-based RL policy improve long-term user reward and system CTR compared to model-free or greedy baselines?
• RQ3 How does the cascading Q-network approach perform for large combinatorial action spaces in recommender settings?
• RQ4 Can the learned user model adapt quickly to new dynamics or limited interactions?

Key findings

• GAN-PW and GAN-LSTM achieve superior predictive precision over baselines on multiple real-world datasets, with GAN-PW offering a favorable efficiency-accuracy balance.
• The GAN-based approach yields higher cumulative rewards and CTRs for the recommended items compared to several strong baselines (e.g., W&D-LR, W&D-CCF, GAN-Greedy, GAN-RWD1, GAN-GDQN, GAN-CDQN).
• GAN-CDQN consistently outperforms alternative RL policies, achieving the highest reported rewards and CTRs in the reported table (e.g., for k=3 and k=5).
• The cascading Q-network framework enables O(k|I|) computation to select top-k items, making combinatorial recommendations scalable to large candidate pools.
• The GAN-based model demonstrates faster adaptation (lower sample complexity) to new dynamics than model-free approaches, leveraging the learned environment model.