[Paper Review] Exploration-Exploitation in Constrained MDPs

Yonathan Efroni, Shie Mannor+1 more|arXiv (Cornell University)|Mar 4, 2020

Reinforcement Learning in Robotics45 references31 citations

TL;DR

The paper analyzes exploration strategies for online learning in constrained Markov Decision Processes (CMDPs), proving sublinear regret bounds for both objective and constraint violation across multiple algorithms, with LP-based methods offering stronger guarantees than dual-based approaches.

ABSTRACT

In many sequential decision-making problems, the goal is to optimize a utility function while satisfying a set of constraints on different utilities. This learning problem is formalized through Constrained Markov Decision Processes (CMDPs). In this paper, we investigate the exploration-exploitation dilemma in CMDPs. While learning in an unknown CMDP, an agent should trade-off exploration to discover new information about the MDP, and exploitation of the current knowledge to maximize the reward while satisfying the constraints. While the agent will eventually learn a good or optimal policy, we do not want the agent to violate the constraints too often during the learning process. In this work, we analyze two approaches for learning in CMDPs. The first approach leverages the linear formulation of CMDP to perform optimistic planning at each episode. The second approach leverages the dual formulation (or saddle-point formulation) of CMDP to perform incremental, optimistic updates of the primal and dual variables. We show that both achieves sublinear regret w.r.t.\ the main utility while having a sublinear regret on the constraint violations. That being said, we highlight a crucial difference between the two approaches; the linear programming approach results in stronger guarantees than in the dual formulation based approach.

Motivation & Objective

• Motivate learning in CMDPs where long-term constraints shape feasible policies.
• Develop and analyze algorithms that balance exploration and constraint satisfaction.
• Provide regret guarantees for both objective performance and constraint violations.
• Compare LP-based optimistic planning with dual-based approaches in constrained settings.

Proposed method

• Extend UCRL2-style optimistic planning to CMDPs via an extended occupancy-measure LP.
• Introduce OptCMDP that builds plausible CMDPs and plans optimistically.
• Develop OptCMDP-bonus that merges uncertainties into an exploration bonus and solves a single CMDP.</br>Use an optimistic dual (and primal-dual) approach (OptDual-CMDP, OptPrimalDual-CMDP) with Lagrangian/Mirror Descent updates.
• Provide sublinear regret bounds for both objective and constraint violations for each method.
• Utilize confidence intervals for transitions and costs using Bernstein/Hoeffding bounds to construct optimistic CMDPs.

Experimental results

Research questions

• RQ1 How should an agent explore in unknown CMDPs while satisfying long-term constraints?
• RQ2 What are the regret guarantees (both objective and constraint violations) for different CMDP learning strategies?
• RQ3 How do LP-based optimistic planning and dual-based approaches compare in terms of theoretical guarantees in CMDPs?
• RQ4 Can exploration bonuses or dual updates yield computationally efficient yet provably safe learning in CMDPs?

Key findings

• OptCMDP achieves sublinear regret for both optimality and constraint violation: Reg_+(K,c) and Reg_+(K,d) are bounded by tilde-O(sqrt(S N H^4 K)).
• OptCMDP-bonus attains the same sublinear regret order as OptCMDP with a more computationally efficient single-CMDP solve.
• OptDual-CMDP provides regret bounds that depend on problem parameters and the Lagrange multiplier radius ρ, with constraint regret having a (1+1/ρ) factor.
• OptPrimalDual-CMDP offers similar sublinear regret guarantees with additional I and H-dependent terms.
• Table 1 summarizes the regret guarantees for each algorithm, showing LP-based methods generally achieving stronger guarantees than dual-based approaches.