[Paper Review] Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction

Aviral Kumar, Justin Fu+2 more|arXiv (Cornell University)|Jun 3, 2019

Reinforcement Learning in Robotics37 references234 citations

TL;DR

The paper identifies bootstrapping error from out-of-distribution actions as a key instability in off-policy Q-learning with static datasets and introduces BEAR, a distribution-constrained offline RL method that reduces error accumulation and yields robust performance across varied off-policy data.

ABSTRACT

Off-policy reinforcement learning aims to leverage experience collected from prior policies for sample-efficient learning. However, in practice, commonly used off-policy approximate dynamic programming methods based on Q-learning and actor-critic methods are highly sensitive to the data distribution, and can make only limited progress without collecting additional on-policy data. As a step towards more robust off-policy algorithms, we study the setting where the off-policy experience is fixed and there is no further interaction with the environment. We identify bootstrapping error as a key source of instability in current methods. Bootstrapping error is due to bootstrapping from actions that lie outside of the training data distribution, and it accumulates via the Bellman backup operator. We theoretically analyze bootstrapping error, and demonstrate how carefully constraining action selection in the backup can mitigate it. Based on our analysis, we propose a practical algorithm, bootstrapping error accumulation reduction (BEAR). We demonstrate that BEAR is able to learn robustly from different off-policy distributions, including random and suboptimal demonstrations, on a range of continuous control tasks.

Motivation & Objective

• Motivate learning from large static off-policy datasets without further environment interaction.
• Analyze bootstrapping error due to out-of-distribution actions in Q-learning.
• Develop a practical off-policy algorithm that controls error propagation via action support constraints.
• Provide theoretical insights and performance guarantees for distribution-constrained backups.

Proposed method

• Formulate distribution-constrained backups that maximize over policies within a support set Pi within the data distribution.
• Introduce suboptimality constant alpha(Pi) and concentrability C(Pi) to bound off-policy performance.
• Propose BEAR: use a Q-ensemble and select actions via the minimum Q across ensemble within Pi_epsilon (support-constrained set).
• Approximate Pi_epsilon using differentiable MMD-based constraint to match the support of the behavior policy.
• Solve the constrained policy improvement with dual gradient methods and sample-based MMD estimation.
• Tie BEAR to distribution-constrained backups by restricting the policy search to the data support while maintaining performance.

Experimental results

Research questions

• RQ1 Can off-policy Q-learning stabilize when learning from a fixed, off-policy dataset with no interaction?
• RQ2 How does constraining backups to the data support affect error propagation and overall performance?
• RQ3 Do distribution-constrained backups generalize across datasets from random, suboptimal, and optimal policies?
• RQ4 Does an offline RL method based on BEAR outperform existing approaches like BCQ and TD3 across diverse continuous-control tasks?

Key findings

• BEAR-QL consistently outperforms BCQ and naïve off-policy RL on medium-quality data across MuJoCo tasks.
• BEAR-QL achieves robust performance on random and near-optimal datasets, often matching or exceeding dataset returns.
• Constraining backups to the data support via MMD-based constraints yields more stable learning than KL-divergence or unconstrained approaches.
• BEAR maintains competitive performance in difficult environments (e.g., Humanoid-v2) under various data conditions.
• A two- or multi-Q ensemble with conservative policy improvement improves robustness to dataset composition.