[Paper Review] An Analysis of Categorical Distributional Reinforcement Learning

Mark Rowland, Marc G. Bellemare+3 more|arXiv (Cornell University)|Feb 22, 2018

Reinforcement Learning in Robotics17 references40 citations

TL;DR

The paper develops a theoretical framework for categorical distributional reinforcement learning (CDRL), proving convergence of sample-based CDRL methods using a projection step grounded in the Cramér distance, and quantifying approximation errors due to discretization.

ABSTRACT

Distributional approaches to value-based reinforcement learning model the entire distribution of returns, rather than just their expected values, and have recently been shown to yield state-of-the-art empirical performance. This was demonstrated by the recently proposed C51 algorithm, based on categorical distributional reinforcement learning (CDRL) [Bellemare et al., 2017]. However, the theoretical properties of CDRL algorithms are not yet well understood. In this paper, we introduce a framework to analyse CDRL algorithms, establish the importance of the projected distributional Bellman operator in distributional RL, draw fundamental connections between CDRL and the Cramér distance, and give a proof of convergence for sample-based categorical distributional reinforcement learning algorithms.

Motivation & Objective

• Motivate and formalize the study of distributional RL where returns are distributions rather than expectations.
• Introduce a unified framework for categorical distributional RL (CDRL) and identify key approximations.
• Analyze the impact of discretization and projection on learning dynamics and convergence.
• Establish convergence results for sample-based CDRL algorithms in both policy evaluation and control.

Proposed method

• Define return distributions and the distributional Bellman operator.
• Introduce a parametric family of categorical distributions over a fixed support.
• Use a projection operator Pi_C to map back to the parametric family after the Bellman backup.
• Employ a stochastic approximation scheme with gradient (KL) updates or a mixture update for learning.
• Prove contractivity of the projected operator in the Cramér distance (ell_2) and derive convergence results.
• Provide finite-grid error bounds and discuss implications for discretization granularity.

Experimental results

Research questions

• RQ1 Does the projected distributional Bellman operator (with a Cramér-distance projection) remain contractive and ensure convergence?
• RQ2 How does discretization (categorical support) affect the approximation to true return distributions and the optimal policy?
• RQ3 Can sample-based CDRL with stochastic updates converge in policy evaluation and in control (Q-learning) to meaningful limits?
• RQ4 What ensures that the learning process preserves optimality when using distributional updates with projection?
• RQ5 What is the role of projection in the convergence and stability of CDRL algorithms?

Key findings

• The composed operator Pi_C T^π is a sqrt(γ)-contraction under the Cramér distance, yielding a unique fixed point η_C.
• The limiting distribution η_C approximates the true η_π with error bound that scales with the grid spacing and 1/(1−γ).
• A stochastic mixture update with Robbins–Monro stepsizes converges almost surely to η_C in policy evaluation.
• Under certain conditions, the corresponding categorical Q-learning converges to a limit η_C^* whose greedy policy is the optimal policy π^*.
• The Cramér projection plays a crucial role by acting as an orthogonal projection in a Hilbert-like space, enabling contraction and convergence analyses.
• The framework reveals how discretization and projection interact to produce tractable, provably convergent CDRL algorithms.