[Paper Review] A Brief Survey of Deep Reinforcement Learning

Kai Arulkumaran, Marc Peter Deisenroth+2 more|arXiv (Cornell University)|Aug 19, 2017

Reinforcement Learning in Robotics121 references750 citations

TL;DR

This paper surveys deep reinforcement learning (DRL), detailing how deep nets enable reinforcement learning to scale to high-dimensional problems, reviews key value-based and policy-based DRL methods (e.g., DQN, TRPO, A3C), and discusses applications, benchmarks, challenges, and future research directions.

ABSTRACT

Deep reinforcement learning is poised to revolutionise the field of AI and represents a step towards building autonomous systems with a higher level understanding of the visual world. Currently, deep learning is enabling reinforcement learning to scale to problems that were previously intractable, such as learning to play video games directly from pixels. Deep reinforcement learning algorithms are also applied to robotics, allowing control policies for robots to be learned directly from camera inputs in the real world. In this survey, we begin with an introduction to the general field of reinforcement learning, then progress to the main streams of value-based and policy-based methods. Our survey will cover central algorithms in deep reinforcement learning, including the deep $Q$-network, trust region policy optimisation, and asynchronous advantage actor-critic. In parallel, we highlight the unique advantages of deep neural networks, focusing on visual understanding via reinforcement learning. To conclude, we describe several current areas of research within the field.

Motivation & Objective

• Motivate and define reinforcement learning and its challenges.
• Explain how deep learning enables scaling RL to high-dimensional problems.
• Review core DRL paradigms: value-based, policy-based, and actor-critic methods.
• Highlight seminal DRL successes and common benchmarks.
• Discuss ongoing research directions and practical considerations in DRL.

Proposed method

• Present a structured overview of RL basics and Markov decision processes.
• Describe value-function and policy-search frameworks and their equations.
• Introduce deep DRL techniques like DQN, experience replay, and target networks.
• Explain improvements to Q-learning (e.g., Double Q-learning, distributional DQN) and policy gradient methods (e.g., actor-critic).
• Discuss planning versus learning, model-based versus model-free approaches, and sample efficiency.
• Survey applications and benchmarks (e.g., Atari, robotics) and future challenges.

Experimental results

Research questions

• RQ1 What are the primary DRL approaches for learning from high-dimensional inputs?
• RQ2 How do value-based and policy-based DRL methods compare and complement each other?
• RQ3 What are the key techniques that stabilized DRL training (e.g., experience replay, target networks)?
• RQ4 What benchmarks and applications demonstrate DRL capabilities and limitations?
• RQ5 What are the open challenges and directions for future DRL research?

Key findings

• DRL enables learning control policies directly from high-dimensional sensory inputs, such as images.
• The Atari benchmark and AlphaGo-like success illustrate DRL’s potential beyond handcrafted features.
• Techniques like experience replay and target networks were crucial to stabilizing DRL training.
• Hybrid actor-critic methods combine value functions with policy optimization to balance bias and variance.
• Deep nets provide powerful representations that mitigate the curse of dimensionality in RL.
• DRL applications span robotics, gaming, and visuomotor tasks, signaling broad potential and remaining challenges.