[Paper Review] Safe and Efficient Off-Policy Reinforcement Learning

Rémi Munos, Thomas Stepleton+2 more|arXiv (Cornell University)|Jun 8, 2016

Reinforcement Learning in Robotics22 references93 citations

TL;DR

This paper introduces Retrace(λ), a novel off-policy reinforcement learning algorithm that combines the safety of tree-backup with the efficiency of return-based learning. It ensures low-variance, stable convergence to the optimal Q-value function Q* without requiring the GLIE assumption, and proves convergence for Watkins' Q(λ) — an open problem since 1989. The method uses importance sampling with a capped correction term to handle arbitrary behavior policies while efficiently utilizing on-policy data.

ABSTRACT

In this work, we take a fresh look at some old and new algorithms for off-policy, return-based reinforcement learning. Expressing these in a common form, we derive a novel algorithm, Retrace($λ$), with three desired properties: (1) it has low variance; (2) it safely uses samples collected from any behaviour policy, whatever its degree of "off-policyness"; and (3) it is efficient as it makes the best use of samples collected from near on-policy behaviour policies. We analyze the contractive nature of the related operator under both off-policy policy evaluation and control settings and derive online sample-based algorithms. We believe this is the first return-based off-policy control algorithm converging a.s. to $Q^*$ without the GLIE assumption (Greedy in the Limit with Infinite Exploration). As a corollary, we prove the convergence of Watkins' Q($λ$), which was an open problem since 1989. We illustrate the benefits of Retrace($λ$) on a standard suite of Atari 2600 games.

Motivation & Objective

• Address the fundamental trade-off between return-based learning (low variance, good with function approximation) and bootstrapping (efficient for off-policy data), aiming to combine both advantages.
• Develop a method that is safe for arbitrary off-policy behavior policies, avoiding divergence due to high off-policyness.
• Ensure high sample efficiency, especially when behavior and target policies are similar, by leveraging full returns from experience replay.
• Eliminate the need for the GLIE assumption in off-policy control, enabling convergence without infinite exploration.
• Provide a theoretical proof of convergence for Watkins' Q(λ), a long-standing open problem in reinforcement learning.

Proposed method

• Formalize off-policy return-based algorithms using a common mathematical framework based on λ-return and importance sampling.
• Propose Retrace(λ), which uses a capped importance sampling ratio min(1, π(a|x)/μ(a|x)) to stabilize learning and ensure convergence under arbitrary behavior policies.
• Integrate the Retrace operator into online, sample-based learning algorithms suitable for experience replay and deep RL.
• Derive a contractive operator for policy evaluation and control, proving almost-sure convergence to Q* without GLIE.
• Use the λ-return extension to balance bootstrapping and Monte Carlo return estimation, enabling flexible and stable learning.
• Apply the algorithm to deep Q-networks in the Atari 2600 environment, demonstrating robust performance across diverse games.

Experimental results

Research questions

• RQ1 How can we design an off-policy return-based reinforcement learning algorithm that remains stable and convergent even when the behavior policy is far from the target policy?
• RQ2 What modifications to existing return-based algorithms are necessary to ensure convergence to the optimal Q-function Q* without requiring the GLIE assumption?
• RQ3 Can the Retrace(λ) algorithm achieve both high sample efficiency in near-on-policy settings and robustness in highly off-policy settings?
• RQ4 Does Retrace(λ) provide a solution to the long-open problem of proving convergence for Watkins' Q(λ) algorithm?
• RQ5 Can Retrace(λ) be effectively applied in deep reinforcement learning settings with experience replay and function approximation?

Key findings

• Retrace(λ) is the first online, return-based, off-policy control algorithm that converges almost surely to the optimal Q-value function Q* without requiring the GLIE assumption.
• The algorithm achieves stable and low-variance learning by capping the importance sampling ratio at 1, ensuring safety even with highly off-policy behavior policies.
• In experiments on the Atari 2600 suite, Retrace(λ) outperformed both Tree-backup(λ) and standard DQN, with higher average scores across multiple games.
• Retrace(λ) demonstrated superior performance in games like Atlantis, Space Invaders, and Video Pinball, achieving final scores of 2,110,401, 6,096, and 228,283 respectively — significantly outperforming DQN and Tree-backup.
• The paper proves that Watkins’ Q(λ) converges almost surely to Q*, resolving an open problem in reinforcement learning that had persisted since 1989.
• Retrace(λ) effectively balances the trade-off between bootstrapping and return estimation, making it both efficient in near-on-policy regimes and safe in highly off-policy settings.