[Paper Review] A Survey of Deep Reinforcement Learning in Video Games

Kun Shao, Zhentao Tang+3 more|arXiv (Cornell University)|Dec 23, 2019

Reinforcement Learning in Robotics91 references148 citations

TL;DR

This survey reviews DRL foundations (value-based, policy gradient, model-based) and summarizes their achievements and challenges across 2D/3D games, including Atari, ViZDoom, StarCraft, and Dota-like environments.

ABSTRACT

Deep reinforcement learning (DRL) has made great achievements since proposed. Generally, DRL agents receive high-dimensional inputs at each step, and make actions according to deep-neural-network-based policies. This learning mechanism updates the policy to maximize the return with an end-to-end method. In this paper, we survey the progress of DRL methods, including value-based, policy gradient, and model-based algorithms, and compare their main techniques and properties. Besides, DRL plays an important role in game artificial intelligence (AI). We also take a review of the achievements of DRL in various video games, including classical Arcade games, first-person perspective games and multi-agent real-time strategy games, from 2D to 3D, and from single-agent to multi-agent. A large number of video game AIs with DRL have achieved super-human performance, while there are still some challenges in this domain. Therefore, we also discuss some key points when applying DRL methods to this field, including exploration-exploitation, sample efficiency, generalization and transfer, multi-agent learning, imperfect information, and delayed spare rewards, as well as some research directions.

Motivation & Objective

• Motivate why DRL is effective for game AI and its unique challenges.
• Catalog and compare DRL methods (value-based, policy gradient, model-based) and their key algorithms.
• Survey DRL achievements across game genres (2D/3D, single-agent/multi-agent).
• Discuss practical challenges (exploration, sample efficiency, generalization, transfer, imperfect information).
• Outline future directions for applying DRL in game AI domains.

Proposed method

• Describe background for DL and RL and formalize MDP notation.
• Summarize major DRL method families and representative algorithms (DQN, Rainbow, IQN, SAC, MuZero, etc.).
• Present key architectural and algorithmic innovations (experience replay, target networks, actor-critic, distributional RL, planning models).
• Discuss training platforms and evaluation benchmarks for DRL in games.
• Provide comparative analysis of performance trends and empirical results in Atari and other game domains.

Experimental results

Research questions

• RQ1 What DRL methods are most effective for game AI across different game genres and settings?
• RQ2 How do value-based, policy gradient, and model-based DRL methods compare in terms of sample efficiency, stability, and generalization in video games?
• RQ3 What platforms, benchmarks, and experimental setups best facilitate progress in DRL for games?
• RQ4 What are the main challenges and future directions for applying DRL to single-agent and multi-agent game environments?

Key findings

• DRL methods achieve super-human performance in many video games across 2D and 3D environments.
• Atari benchmarks show dramatic gains: Rainbow and Ape-X variants substantially outperform earlier DQN approaches.
• Distributional and multi-agent extensions (C51, QR-DQN, IQN, Ape-X DQN/DQfD) yield large improvements in mean and median Atari scores.
• Model-based approaches (MuZero, TreeQN) push planning-based improvements in DRL for complex domains.
• First-person and RTS domains present additional challenges like partial observability and multi-agent coordination, motivating further advances in memory, exploration, and transfer.