[Paper Review] Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learning

Edoardo Cetin, Oya Çeliktutan|arXiv (Cornell University)|Oct 7, 2021

Reinforcement Learning in Robotics4 citations

TL;DR

This paper introduces Generalized Pessimism Learning (GPL), a novel method that learns a dynamic penalty to correct overestimation bias in off-policy deep reinforcement learning via dual TD-learning. By integrating GPL with SAC and DrQ, the approach achieves state-of-the-art performance on both proprioceptive and pixel-based benchmarks with minimal computational overhead, outperforming prior methods in sample efficiency and training speed.

ABSTRACT

Off-policy deep reinforcement learning algorithms commonly compensate for overestimation bias during temporal-difference learning by utilizing pessimistic estimates of the expected target returns. In this work, we propose Generalized Pessimism Learning (GPL), a strategy employing a novel learnable penalty to enact such pessimism. In particular, we propose to learn this penalty alongside the critic with dual TD-learning, a new procedure to estimate and minimize the magnitude of the target returns bias with trivial computational cost. GPL enables us to accurately counteract overestimation bias throughout training without incurring the downsides of overly pessimistic targets. By integrating GPL with popular off-policy algorithms, we achieve state-of-the-art results in both competitive proprioceptive and pixel-based benchmarks.

Motivation & Objective

• To address overestimation bias in off-policy deep reinforcement learning, which undermines policy learning stability and sample efficiency.
• To develop a learnable, adaptive mechanism for pessimism that avoids the limitations of fixed or heuristic penalty strategies.
• To improve exploration by introducing pessimism annealing, which leverages epistemic uncertainty to guide directed exploration in early training.
• To integrate the proposed method with established off-policy algorithms like SAC and DrQ to achieve robust, scalable performance gains.
• To demonstrate that the method achieves state-of-the-art results with negligible computational overhead across diverse benchmark environments.

Proposed method

• Proposes Generalized Pessimism Learning (GPL), a learnable penalty mechanism that corrects overestimation bias in critic targets during temporal-difference learning.
• Introduces dual TD-learning, a novel optimization procedure that estimates and minimizes bias in critic predictions using dual gradient descent.
• Computes the penalty weight as a function of epistemic uncertainty, measured via the expected Wasserstein distance between return distributions predicted by multiple critic networks.
• Employs pessimism annealing, which starts with a biased, risk-seeking target to encourage exploration in high-uncertainty states during early training.
• Integrates GPL with Soft Actor-Critic (SAC) and Data-regularized Q (DrQ) algorithms, forming GPL-SAC and GPL-DrQ, respectively.
• Uses a shared, grouped parameterization across multiple critic networks to improve computational efficiency and scalability on modern hardware.

Experimental results

Research questions

• RQ1 Can a learnable, dynamic penalty mechanism effectively correct overestimation bias in off-policy deep reinforcement learning without relying on fixed or heuristic strategies?
• RQ2 How does dual TD-learning compare to standard TD-learning in estimating and minimizing bias in critic predictions?
• RQ3 To what extent does pessimism annealing improve exploration and sample efficiency in early training phases?
• RQ4 Can GPL achieve state-of-the-art performance across both proprioceptive and pixel-based environments with minimal computational overhead?
• RQ5 How does the choice of critic architecture (e.g., standard vs. distributional) affect the performance and efficiency of GPL?

Key findings

• GPL-SAC achieves a Humanoid score of 5000 in under 100,000 environment steps, more than nine times faster than standard SAC.
• GPL-SAC outperforms both model-free and model-based state-of-the-art algorithms on challenging MuJoCo benchmarks, including in low-data regimes.
• GPL-DrQ improves performance over the recent DrQv2 baseline on pixel-based environments from the DeepMind Control Suite.
• The computational overhead of GPL is trivial: training time increases by only 2.5% when adding dual TD-learning and pessimism annealing compared to baseline SAC.
• Increasing the number of critic networks from 2 to 20 increases training time by less than 12%, indicating sub-linear scaling and strong hardware efficiency.
• GPL-QR-SAC with $W_1$-based uncertainty estimation performs similarly to GPL-SAC, suggesting that distributional critics do not yield major gains in low-stochasticity environments like OpenAI Gym.