[Paper Review] A Distributional Perspective on Reinforcement Learning

Marc G. Bellemare, Will Dabney+1 more|arXiv (Cornell University)|Jul 21, 2017

Reinforcement Learning in RoboticsComputer Science38 references241 citations

TL;DR

The paper argues for modeling the full distribution of returns (value distributions) in reinforcement learning, introduces a distributional Bellman framework with contraction in Wasserstein distance for policy evaluation, analyzes instability in the control setting, and presents a discrete distributional learning algorithm (categorical DQN) that achieves strong Atari results.

ABSTRACT

In this paper we argue for the fundamental importance of the value distribution: the distribution of the random return received by a reinforcement learning agent. This is in contrast to the common approach to reinforcement learning which models the expectation of this return, or value. Although there is an established body of literature studying the value distribution, thus far it has always been used for a specific purpose such as implementing risk-aware behaviour. We begin with theoretical results in both the policy evaluation and control settings, exposing a significant distributional instability in the latter. We then use the distributional perspective to design a new algorithm which applies Bellman's equation to the learning of approximate value distributions. We evaluate our algorithm using the suite of games from the Arcade Learning Environment. We obtain both state-of-the-art results and anecdotal evidence demonstrating the importance of the value distribution in approximate reinforcement learning. Finally, we combine theoretical and empirical evidence to highlight the ways in which the value distribution impacts learning in the approximate setting.

Motivation & Objective

• Motivate a distributional view of reinforcement learning by focusing on the distribution of returns Z rather than just its expectation Q.
• Characterize the theoretical behavior of distributional Bellman operators in policy evaluation and control.
• Develop a practical algorithm to learn approximate value distributions and assess its empirical performance on Atari games.

Proposed method

• Define the value distribution Z under a policy and formulate distributional Bellman equations.
• Use the Wasserstein metric to analyze contraction properties of distributional Bellman operators in policy evaluation (T^π).
• Show instability in the distributional optimality operator in the control setting, including non-contraction and non-fixed-point issues.
• Propose a parametric discrete distribution (atoms on a fixed grid) to model Z and a projection-based Bellman update onto the support (multiclass classification).
• Train a categorical distributional DQN (C51) by minimizing KL divergence between projected Bellman updates and the current distribution.
• Evaluate on the Arcade Learning Environment with Atari 2600 games, comparing to DQN-family baselines.

Experimental results

Research questions

• RQ1 Does modeling the full value distribution provide theoretical and empirical advantages over learning only the expected return?
• RQ2 Is the distributional Bellman operator a contraction in a suitable metric in policy evaluation and in control settings?
• RQ3 Can a tractable, distributional approximation (via discretized distributions) be learned effectively and yield improved performance on complex tasks like Atari games?

Key findings

• The distributional Bellman operator is a γ-contraction in the maximal Wasserstein metric for policy evaluation, with convergence to the true value distribution Z^π.
• In the control setting, the distributional optimality operator is not a contraction in any metric over distributions and may fail to have a fixed point, indicating instability with greedy updates.
• Learning the full value distribution can preserve multimodality and lead to more stable learning under function approximation and nonstationary policies.
• A discrete, parameterized value distribution learned via projection (categorical algorithm) can outperform DQN on several Atari games, achieving state-of-the-art results on some titles.
• Increasing the number of atoms in the distribution generally improves performance, with substantial gains over DQN in multiple games.
• The approach helps propagate rare or sparse rewards more effectively, improving performance on sparse-reward games.