[Paper Review] Provably Good Batch Reinforcement Learning Without Great Exploration

Yao Liu, Adith Swaminathan+2 more|arXiv (Cornell University)|Jul 16, 2020

Reinforcement Learning in Robotics38 references36 citations

TL;DR

The paper introduces conservative, marginalized-behavior-supported batch RL algorithms (MBS-PI and MBS-QI) that guarantee near-optimal policies within the data-supported region, without relying on strong concentrability assumptions. It provides theory and empirical evaluation showing improved stability and performance over baselines in batch settings.

ABSTRACT

Batch reinforcement learning (RL) is important to apply RL algorithms to many high stakes tasks. Doing batch RL in a way that yields a reliable new policy in large domains is challenging: a new decision policy may visit states and actions outside the support of the batch data, and function approximation and optimization with limited samples can further increase the potential of learning policies with overly optimistic estimates of their future performance. Recent algorithms have shown promise but can still be overly optimistic in their expected outcomes. Theoretical work that provides strong guarantees on the performance of the output policy relies on a strong concentrability assumption, that makes it unsuitable for cases where the ratio between state-action distributions of behavior policy and some candidate policies is large. This is because in the traditional analysis, the error bound scales up with this ratio. We show that a small modification to Bellman optimality and evaluation back-up to take a more conservative update can have much stronger guarantees. In certain settings, they can find the approximately best policy within the state-action space explored by the batch data, without requiring a priori assumptions of concentrability. We highlight the necessity of our conservative update and the limitations of previous algorithms and analyses by illustrative MDP examples, and demonstrate an empirical comparison of our algorithm and other state-of-the-art batch RL baselines in standard benchmarks.

Motivation & Objective

• Motivate batch RL when data only partially covers the state-action space and conventional methods risk overly optimistic estimates.
• Propose conservative Bellman backups that restrict updates to well-supported state-action pairs.
• Develop Marginalized Behavior Supported Policy Iteration (MBS-PI) and Q-Iteration (MBS-QI) with theoretical guarantees.
• Provide empirical evaluation showing robustness and improved performance over baselines in discrete and continuous domains.

Proposed method

• Introduce a density-based filter z(s,a; mu_hat,b) to restrict Bellman backups to well-supported state-action pairs.
• Define a z-constrained Bellman evaluation operator tilde{T}^{pi} and a tilde{T} operator for Q-learning that use z to avoid extrapolation.
• Formulate MBS-PI and MBS-QI as constrained API and AVI variants that optimize only over data-supported actions.
• Use least-squares empirical loss l_D(f;f',p) to fit Q-functions within the constrained operator framework.
• Provide theoretical guarantees: compare outputs to the best covered policy within the constrained set Pi_all, with bounds that scale with a hyperparameter b (via C = U/b).
• Discuss practical aspects of choosing the threshold b and estimating the behavior density  mu.

Experimental results

Research questions

• RQ1 Can batch RL algorithms guarantee good performance while avoiding reliance on strong concentrability assumptions?
• RQ2 Does constraining Bellman backups to data-supported state-action pairs yield provable guarantees and improved empirical performance in both discrete and continuous domains?
• RQ3 How do MBS-PI and MBS-QI compare to existing baselines (e.g., BCQL, SPIBB, BEAR, BC) under varying data coverage?
• RQ4 What is the role of the threshold parameter b in trading off conservatism and optimism, and how should it be set in practice?
• RQ5 Under what conditions does the constrained approach achieve near-optimal performance within the data-supported policy class?

Key findings

• MBS-PI and MBS-QI can identify approximately best policies within the state-action space explored by the batch data without requiring concentrability assumptions.
• The z-constrained Bellman operators reduce updates for poorly supported regions, mitigating overly optimistic bootstrapping common in prior batch RL methods.
• Theoretical guarantees show the value gap to the best covered policy scales with the data density bound and the threshold parameter b, plus standard dependence on n and (1-γ) factors.
• Empirical results in discrete CartPole and continuous MuJoCo tasks demonstrate improved stability and performance over baselines across varying data coverage scenarios.
• The framework provides safe-policy-improvement style guarantees in discrete settings and demonstrates robustness to estimation errors in the behavior density mu_hat.
• The hyperparameter b is central to balancing conservatism and data utilization, with guidance on practical selection and diagnostic checks.