[Paper Review] A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions

Xiaocong Chen, Lina Yao+3 more|arXiv (Cornell University)|Sep 8, 2021

Reinforcement Learning in Robotics134 references20 citations

TL;DR

This survey provides a comprehensive, systematic review of deep reinforcement learning (DRL) in recommender systems, categorizing existing methods, analyzing their strengths and limitations, and identifying emerging trends and open challenges. It offers a unified taxonomy, highlights key DRL algorithms like A2C, TD3, and SAC, and outlines future research directions for advancing interactive, dynamic recommendation systems.

ABSTRACT

In light of the emergence of deep reinforcement learning (DRL) in recommender systems research and several fruitful results in recent years, this survey aims to provide a timely and comprehensive overview of the recent trends of deep reinforcement learning in recommender systems. We start with the motivation of applying DRL in recommender systems. Then, we provide a taxonomy of current DRL-based recommender systems and a summary of existing methods. We discuss emerging topics and open issues, and provide our perspective on advancing the domain. This survey serves as introductory material for readers from academia and industry into the topic and identifies notable opportunities for further research.

Motivation & Objective

• To provide a timely and systematic overview of deep reinforcement learning (DRL) applications in recommender systems, addressing the gap in existing surveys that lack depth on DRL-specific techniques.
• To address the limitations of traditional deep learning-based recommenders in capturing dynamic user preferences due to distribution shift and static training data.
• To classify and analyze existing DRL-based recommender systems using a novel, structured taxonomy to clarify methodological differences and design choices.
• To identify emerging topics such as offline DRL, meta-DRL, and actor-critic improvements, and to highlight open issues in generalization, exploration, and real-world deployment.
• To guide researchers and practitioners by outlining promising future research directions, including improved generalization, sample efficiency, and robustness in dynamic environments.

Proposed method

• Proposes a taxonomy of DRL-based recommender systems, classifying methods by environment construction, state representation, and policy learning mechanisms.
• Reviews core DRL algorithms including Advantage Actor-Critic (A2C), Twin-Delayed DDPG (TD3), Trust Region Policy Optimization (TRPO), Proximal Policy Optimization (PPO), and Soft Actor-Critic (SAC), emphasizing their role in stabilizing training and improving sample efficiency.
• Analyzes offline DRL (batch DRL) as a method to train from historical interaction data without online interaction, enabling generalization across new scenarios.
• Explores meta-DRL, which uses memory units (e.g., RNNs) to store and transfer knowledge across tasks, reducing data requirements and improving few-shot adaptation.
• Examines the use of advantage functions and entropy regularization (as in SAC) to balance exploration and exploitation in policy learning.
• Integrates insights from off-policy and on-policy training paradigms, comparing their suitability for recommendation tasks under different data availability and interaction constraints.

Experimental results

Research questions

• RQ1 How can deep reinforcement learning effectively model dynamic user preferences that shift over time, overcoming the distribution shift problem in static training data?
• RQ2 What are the key architectural and algorithmic components that distinguish successful DRL-based recommender systems, and how do they contribute to performance and stability?
• RQ3 How do offline and meta-DRL approaches improve generalization and sample efficiency in recommendation systems with limited or no real-time interaction?
• RQ4 What are the major challenges in applying DRL to real-world recommender systems, particularly regarding exploration-exploitation trade-offs, policy stability, and scalability?
• RQ5 What future research directions are most promising for advancing DRL-based recommendation systems in terms of robustness, interpretability, and deployment in industrial settings?

Key findings

• This survey presents the first comprehensive and systematic review of DRL in recommender systems, offering a structured taxonomy and classification of existing methods.
• Actor-critic methods such as A2C, TD3, PPO, and SAC significantly improve training stability and sample efficiency compared to basic policy gradient methods.
• Offline DRL enables training from large historical datasets without online interaction, making it suitable for industrial applications where real-time deployment is costly or risky.
• Meta-DRL shows promise in few-shot adaptation by leveraging memory mechanisms to transfer knowledge across tasks, reducing data dependency.
• SAC’s use of entropy regularization enhances exploration, making it particularly effective in environments with sparse rewards or complex action spaces.
• Despite progress, challenges remain in handling distributional shift, ensuring policy generalization, and achieving robust performance in real-world, dynamic recommendation scenarios.