[Paper Review] Implicit Quantile Networks for Distributional Reinforcement Learning

Will Dabney, Georg Ostrovski+2 more|arXiv (Cornell University)|Jun 14, 2018

Reinforcement Learning in RoboticsComputer Science46 references197 citations

TL;DR

IQN learns the full quantile function of return distributions to enable distributional and risk-sensitive reinforcement learning, achieving strong Atari-57 results and competitive performance with Rainbow.

ABSTRACT

In this work, we build on recent advances in distributional reinforcement learning to give a generally applicable, flexible, and state-of-the-art distributional variant of DQN. We achieve this by using quantile regression to approximate the full quantile function for the state-action return distribution. By reparameterizing a distribution over the sample space, this yields an implicitly defined return distribution and gives rise to a large class of risk-sensitive policies. We demonstrate improved performance on the 57 Atari 2600 games in the ALE, and use our algorithm's implicitly defined distributions to study the effects of risk-sensitive policies in Atari games.

Motivation & Objective

• Motivate distributional RL as modeling the full return distribution instead of only its mean.
• Propose a flexible IQN approach that learns an implicit quantile function via reparameterization.
• Enable risk-sensitive policies by allowing distortion risk measures over the learned distribution.
• Demonstrate improved performance on Atari-57 and explore robustness and risk-sensitive behavior.

Proposed method

• Model the state-action quantile function Z_tau(x,a) by reparameterizing a base sample tau from U([0,1]) through a differentiable embedding.
• Train using an IQN loss that averages a quantile regression loss over pairs of tau samples, minimizing a Wasserstein-consistent objective.
• Use a DQN-like architecture with an additional tau-embedding φ(tau) that interacts multiplicatively with state features to produce Z_tau(x,a).
• Allow distortion risk measures beta to induce risk-sensitive policies by mapping tau through beta (e.g., CPW, Wang, CVaR) and optimizing distorted expectations.
• Explore architectural variants and hyperparameters (N, N') controlling the number of tau samples per update to balance data efficiency and learning speed.

Experimental results

Research questions

• RQ1 Can learning the full quantile function of returns improve distributional RL over fixed-quantile methods like QR-DQN?
• RQ2 Does implicit quantile representation enable effective risk-sensitive policies via distortion risk measures?
• RQ3 How do architectural choices and sampling parameters (N, N') affect data efficiency and long-term performance?
• RQ4 How does IQN perform on large-scale benchmarks like Atari-57 compared to state-of-the-art methods such as Rainbow and QR-DQN?

Key findings

• IQN significantly outperforms QR-DQN and nearly matches Rainbow on Atari-57 without combining orthogonal improvements.
• IQN achieves 1000+% mean human-normalized score and strong median performance, with IQN often narrowing the gap to Rainbow on hard games.
• Varying the number of tau samples N and N' shows N=8 and N'=8 providing strong long-term performance with diminishing returns beyond that.
• Risk-averse distortion measures can yield improved performance on some games, while risk-seeking measures may underperform on others, indicating nuanced effects of risk preferences.
• IQN with risk-neutral evaluation still delivers substantial gains over baseline methods, and the approach halves the remaining distance to Rainbow on Atari-57.