[Paper Review] Safe Policy Improvement with Baseline Bootstrapping

Romain Laroche, Paul Trichelair+1 more|arXiv (Cornell University)|Dec 19, 2017

Information and Cyber Security82 citations

TL;DR

This paper proposes SPIBB, a safe batch RL method that bootsstraps the trained policy with a baseline in uncertain state-action pairs, with theoretical safety guarantees and a practical model-free variant called SPIBB-DQN.

ABSTRACT

This paper considers Safe Policy Improvement (SPI) in Batch Reinforcement Learning (Batch RL): from a fixed dataset and without direct access to the true environment, train a policy that is guaranteed to perform at least as well as the baseline policy used to collect the data. Our approach, called SPI with Baseline Bootstrapping (SPIBB), is inspired by the knows-what-it-knows paradigm: it bootstraps the trained policy with the baseline when the uncertainty is high. Our first algorithm, $Π_b$-SPIBB, comes with SPI theoretical guarantees. We also implement a variant, $Π_{\leq b}$-SPIBB, that is even more efficient in practice. We apply our algorithms to a motivational stochastic gridworld domain and further demonstrate on randomly generated MDPs the superiority of SPIBB with respect to existing algorithms, not only in safety but also in mean performance. Finally, we implement a model-free version of SPIBB and show its benefits on a navigation task with deep RL implementation called SPIBB-DQN, which is, to the best of our knowledge, the first RL algorithm relying on a neural network representation able to train efficiently and reliably from batch data, without any interaction with the environment.

Motivation & Objective

• Motivate safe policy improvement in offline batch RL where the true environment is unavailable.
• Introduce the SPIBB framework that bootstraps decisions in uncertain state-action pairs using a baseline policy.
• Provide theoretical guarantees of near-safe improvement and develop computationally efficient variants.
• Extend SPIBB to model-free and neural-network based settings for scalable applications.
• Demonstrate empirical safety and performance advantages across gridworld, random MDPs, and a deep RL navigation task.

Proposed method

• Define a percentile/safety criterion reformulation that constrains improvements to be safe against the baseline across an uncertainty set.
• Introduce bootstrapped state-action sets where rare (x,a) pairs are forced to follow the baseline (Pi_b-SPIBB).
• Develop a model-based policy iteration under the Pi_b-SPIBB constraint with convergence guarantees.
• Provide a more permissive Pi_≤b-SPIBB variant that relaxes the bootstrapping constraint for practical performance.
• Offer a model-free version with SPIBB targets, and SPIBB-DQN that uses pseudo-count inspired bootstrapping for neural networks.
• Present related theoretical bounds tying counts N_◇ to safety and improvement.

Experimental results

Research questions

• RQ1 Can we guarantee a policy that improves upon a given baseline in an offline batch RL setting without interacting with the environment?
• RQ2 How can we leverage baseline bootstrapping to safely constrain policy improvement in uncertain state-action pairs?
• RQ3 What are the trade-offs between model-based Pi_b-SPIBB and the more practical Pi_≤b-SPIBB in terms of safety and performance?
• RQ4 Can SPIBB be extended to model-free and deep RL settings to enable scalable offline improvements?
• RQ5 How do SPIBB variants perform across diverse domains (gridworld, random MDPs, navigation tasks) relative to existing SPI methods?

Key findings

• SPIBB provides a provable safe policy improvement over the baseline with high probability in finite MDPs.
• Pi_b-SPIBB converges to a Pi_b-optimal policy in the estimated MDP and yields a zeta-approximate safe improvement over the baseline.
• Pi_≤b-SPIBB often achieves better or comparable improvement while maintaining practical safety and efficiency.
• A model-free SPIBB variant and SPIBB-DQN enable safe training from batch data without environment interaction.
• Empirical results show SPIBB variants outperform several baselines in mean performance and CVaR safety across gridworlds and random MDPs, and SPIBB-DQN demonstrates safe improvement in a continuous navigation task.