[Paper Review] Deep Reinforcement Learning for List-wise Recommendations

Xiangyu Zhao, Liang Zhang+4 more|arXiv (Cornell University)|Dec 30, 2017

Recommender Systems and Techniques32 references109 citations

TL;DR

This paper proposes a deep reinforcement learning framework (LIRD) for list-wise recommendations, leveraging an actor-critic architecture with an online environment simulator to train and evaluate offline before online deployment, showing gains over baselines on real e-commerce data.

ABSTRACT

Recommender systems play a crucial role in mitigating the problem of information overload by suggesting users' personalized items or services. The vast majority of traditional recommender systems consider the recommendation procedure as a static process and make recommendations following a fixed strategy. In this paper, we propose a novel recommender system with the capability of continuously improving its strategies during the interactions with users. We model the sequential interactions between users and a recommender system as a Markov Decision Process (MDP) and leverage Reinforcement Learning (RL) to automatically learn the optimal strategies via recommending trial-and-error items and receiving reinforcements of these items from users' feedbacks. In particular, we introduce an online user-agent interacting environment simulator, which can pre-train and evaluate model parameters offline before applying the model online. Moreover, we validate the importance of list-wise recommendations during the interactions between users and agent, and develop a novel approach to incorporate them into the proposed framework LIRD for list-wide recommendations. The experimental results based on a real-world e-commerce dataset demonstrate the effectiveness of the proposed framework.

Motivation & Objective

• Motivate the need for dynamic, long-term optimization in recommender systems beyond static, short-term strategies.
• Model user–recommender interactions as an MDP to maximize cumulative rewards over time.
• Develop an online environment simulator to enable offline pre-training and evaluation before online deployment.
• Introduce a list-wise, scalable RL framework (LIRD) that handles large and dynamic item spaces.
• Demonstrate the effectiveness of list-wise recommendations on real-world e-commerce data.

Proposed method

• Model the recommender as an MDP with state s as user browsing history, action a as a list of K recommended items, reward r from user feedback, and discount factor gamma.
• Employ an online environment simulator that maps (state, action) pairs to rewards using historical memory and cosine similarity, enabling offline training.
• Use an actor-critic architecture where the Actor generates state-specific weight vectors to score items, producing a list-wise action; the Critic estimates Q(s,a) via a deep Q-network approximation.
• Train with Deep Deterministic Policy Gradient (DDPG) using experience replay, target networks, and prioritized sampling.
• Adopt a two-stage training procedure: generate transitions from interactions, then update Actor and Critic networks with mini-batches.
• Evaluate list-wise strategy by varying K and compare against CF, FM, DNN, RNN, and DQN baselines on a real dataset.

Experimental results

Research questions

• RQ1 Does the proposed framework outperform representative baselines in item recommendation tasks?
• RQ2 How does list-wise recommendation (varying K) impact performance in long-term scenarios?
• RQ3 Can the online simulator provide reliable offline pre-training, reducing the gap to online deployment?

Key findings

• The proposed framework outperforms baselines in both short and long sessions, with stronger gains in long sessions due to long-term reward optimization.
• List-wise recommendations (K=4) yield better performance than other K values, indicating a balance between capturing item correlations and avoiding noise.
• LIRD trains faster than DQN while achieving similar or better performance, due to reduced computation by avoiding evaluating all actions.
• The online simulator enables offline training and evaluation, mitigating the offline–online gap and facilitating parameter initialization for online use.
• Using historical user-item embeddings and item history improves modeling of user preferences and scalability.