[Paper Review] Provably Efficient Safe Exploration via Primal-Dual Policy Optimization

Dong-Sheng Ding, Xiaohan Wei+3 more|arXiv (Cornell University)|Mar 1, 2020

Reinforcement Learning in Robotics62 references35 citations

TL;DR

The paper introduces OPDOP, an optimistic primal-dual proximal policy optimization algorithm for constrained CMDPs with linear function approximation, achieving sublinear regret and constraint violation under bandit feedback. It provides the first provably efficient online policy optimization guarantees for CMDPs with safe exploration in the function-approximation setting.

ABSTRACT

We study the Safe Reinforcement Learning (SRL) problem using the Constrained Markov Decision Process (CMDP) formulation in which an agent aims to maximize the expected total reward subject to a safety constraint on the expected total value of a utility function. We focus on an episodic setting with the function approximation where the Markov transition kernels have a linear structure but do not impose any additional assumptions on the sampling model. Designing SRL algorithms with provable computational and statistical efficiency is particularly challenging under this setting because of the need to incorporate both the safety constraint and the function approximation into the fundamental exploitation/exploration tradeoff. To this end, we present an \underline{O}ptimistic \underline{P}rimal-\underline{D}ual Proximal Policy \underline{OP}timization (OPDOP) algorithm where the value function is estimated by combining the least-squares policy evaluation and an additional bonus term for safe exploration. We prove that the proposed algorithm achieves an $ ilde{O}(d H^{2.5}\sqrt{T})$ regret and an $ ilde{O}(d H^{2.5}\sqrt{T})$ constraint violation, where $d$ is the dimension of the feature mapping, $H$ is the horizon of each episode, and $T$ is the total number of steps. These bounds hold when the reward/utility functions are fixed but the feedback after each episode is bandit. Our bounds depend on the capacity of the state-action space only through the dimension of the feature mapping and thus our results hold even when the number of states goes to infinity. To the best of our knowledge, we provide the first provably efficient online policy optimization algorithm for CMDP with safe exploration in the function approximation setting.

Motivation & Objective

• Motivate Safe Reinforcement Learning (SRL) under constraints and unknown transitions using CMDP formulation in an episodic setting.
• Develop an online, sample-efficient algorithm that handles safety constraints while learning from streaming, bandit feedback.
• Provide theoretical guarantees: sublinear regret and constraint violation with respect to the best policy in hindsight.
• Show that the bounds depend on feature dimension rather than state space size, enabling infinite or very large state spaces.

Proposed method

• Propose OPDOP: an optimistic primal-dual proximal policy optimization algorithm with three stages per episode: policy improvement, dual update, and policy evaluation.
• Use a LM-like Lagrangian with dual variable Y and perform policy improvement via a mirror-descent style update that is separable across states, incorporating a KL-divergence regularization term.
• Incorporate an optimism via Upper Confidence Bound (UCB) bonuses in value-function estimation through Least-Squares Temporal Difference (LSTD) style updates.
• Estimate reward r and utility g with linear function approximation and compute Q and V functions with UCB bonuses to guide exploration.
• Update the dual variable Y with a projected gradient step to balance reward maximization and constraint satisfaction, ensuring feasibility in the long run.
• Utilize LSTD (Algorithm 2) to estimate (P_h V_{…,h+1}) terms and employ feature maps psi and phi to represent transitions and rewards in a linear MDP setting.

Experimental results

Research questions

• RQ1 Can we design a provably efficient online policy optimization algorithm for CMDPs with unknown transitions under function approximation?
• RQ2 What are the regret and constraint-violation bounds when learning under bandit feedback in episodic CMDPs with linear structure?
• RQ3 How can safe exploration be achieved when the safety constraint function is unknown and revealed only via interaction?
• RQ4 How do bounds scale with the feature dimension d, horizon H, and total steps T, and do they remain valid for infinite state spaces?
• RQ5 Can an optimistic, primal-dual mirror-descent approach yield tractable, provable guarantees without oracle solvers in the constrained RL setting?

Key findings

• OPDOP achieves sublinear regret and sublinear constraint violation: both scale as Õ(d H^{2.5} sqrt(T)) under bandit feedback.
• Bounds depend on the feature dimension d rather than the (potentially infinite) state space, enabling applicability to large or infinite state spaces.
• First provably efficient online policy optimization result for CMDPs with safe exploration in the function-approximation setting.
• Algorithm combines least-squares policy evaluation with an additional safe-exploration bonus and an online primal-dual mirror-descent update.
• Dual updates balance reward maximization and constraint satisfaction, ensuring feasibility in expectation and sublinear violation across episodes.