[Paper Review] QUOTA: The Quantile Option Architecture for Reinforcement Learning

Shangtong Zhang, Borislav Mavrin+3 more|arXiv (Cornell University)|Nov 5, 2018

Reinforcement Learning in Robotics4 citations

TL;DR

QUOTA proposes a novel reinforcement learning architecture that uses quantiles of the value distribution for decision-making, enabling adaptive exploration through optimistic (high quantile) and pessimistic (low quantile) strategies. By integrating these quantile-based options within a hierarchical framework, QUOTA improves sample efficiency and performance across Atari games and Roboschool robot simulators, outperforming standard DDPG and QR-DDPG baselines.

ABSTRACT

In this paper, we propose the Quantile Option Architecture (QUOTA) for exploration based on recent advances in distributional reinforcement learning (RL). In QUOTA, decision making is based on quantiles of a value distribution, not only the mean. QUOTA provides a new dimension for exploration via making use of both optimism and pessimism of a value distribution. We demonstrate the performance advantage of QUOTA in both challenging video games and physical robot simulators.

Motivation & Objective

• To address the limitation of mean-based value estimation in reinforcement learning, which fails to leverage full return distribution for efficient exploration.
• To develop a method that dynamically selects between optimistic and pessimistic exploration strategies using quantiles of the value distribution.
• To extend the option framework to incorporate quantile-based decision-making for both discrete- and continuous-action problems.
• To empirically validate that quantile-based exploration improves sample efficiency and final performance in complex environments.

Proposed method

• QUOTA uses a distributional RL approach where state-action values are represented as a set of quantiles, rather than just the mean.
• Action selection is based on a specific quantile of the value distribution, with high quantiles enabling optimistic exploration and low quantiles enabling pessimistic exploration.
• A high-level option policy selects which quantile to use at each step, allowing dynamic switching between exploration strategies.
• For continuous-action problems, QUOTA introduces multiple quantile actors, each specialized to maximize a specific quantile of the value distribution.
• The method employs quantile regression loss to train the value network and a modified DDPG-style algorithm with experience replay and target networks.
• The option value function is trained using a bootstrap target that combines immediate reward and future option value, with a termination probability to model option duration.

Experimental results

Research questions

• RQ1 Can leveraging the full distribution of returns, rather than just the mean, improve exploration in reinforcement learning?
• RQ2 Does using high quantiles (optimism) or low quantiles (pessimism) lead to better sample efficiency in different environments?
• RQ3 Can a hierarchical option framework that selects among quantile-based policies adaptively improve performance across diverse tasks?
• RQ4 How does QUOTA compare to standard DDPG and QR-DDPG in terms of final performance and learning stability?

Key findings

• QUOTA significantly outperformed QR-DDPG and DDPG on 49 Atari games, with median scores of 2555.80, 2364.60, and 1730.33 respectively across three independent runs.
• In Roboschool environments, QUOTA achieved higher average returns than DDPG and QR-DDPG, with evaluation curves showing faster convergence and better final performance.
• The method demonstrated consistent performance gains in tasks like Montezuma’s Revenge and Private Eye, where exploration is particularly challenging.
• QUOTA’s ability to switch between optimistic and pessimistic strategies enabled faster learning in environments where mean-based methods failed to explore efficiently.
• The use of quantile actors in continuous control tasks enabled stable training and improved sample efficiency, as shown by the learning curves in Roboschool.