[Paper Review] A Sample-Efficient Algorithm for Episodic Finite-Horizon MDP with Constraints

Krishna C. Kalagarla, Rahul Jain+1 more|arXiv (Cornell University)|Sep 23, 2020

Formal Methods in Verification12 citations

TL;DR

This paper proposes UC-CFH, a sample-efficient online algorithm for episodic finite-horizon constrained MDPs with unknown transition probabilities. By leveraging optimistic planning via linear programming in occupancy measures, it achieves a PAC guarantee with sample complexity $ tilde{ m O}ig{(}\frac{|\mathcal{S}||\mathcal{A}|C^{2}H^{2}}{\epsilon^{2}}\log\frac{1}{\delta}\big{)}$, where $C$ is the maximum number of successor states per state-action pair, ensuring $\epsilon$-optimal performance and constraint satisfaction with high probability.

ABSTRACT

Constrained Markov Decision Processes (CMDPs) formalize sequential decision-making problems whose objective is to minimize a cost function while satisfying constraints on various cost functions. In this paper, we consider the setting of episodic fixed-horizon CMDPs. We propose an online algorithm which leverages the linear programming formulation of finite-horizon CMDP for repeated optimistic planning to provide a probably approximately correct (PAC) guarantee on the number of episodes needed to ensure an $ε$-optimal policy, i.e., with resulting objective value within $ε$ of the optimal value and satisfying the constraints within $ε$-tolerance, with probability at least $1-δ$. The number of episodes needed is shown to be of the order $ ilde{\mathcal{O}}\big(\frac{|S||A|C^{2}H^{2}}{ε^{2}}\log\frac{1}δ\big)$, where $C$ is the upper bound on the number of possible successor states for a state-action pair. Therefore, if $C \ll |S|$, the number of episodes needed have a linear dependence on the state and action space sizes $|S|$ and $|A|$, respectively, and quadratic dependence on the time horizon $H$.

Motivation & Objective

• To address the challenge of learning near-optimal policies in episodic finite-horizon MDPs with constraints when transition probabilities are unknown.
• To provide a Probably Approximately Correct (PAC) guarantee on both performance and constraint satisfaction during online learning.
• To achieve sample efficiency by minimizing the number of episodes required to reach $\epsilon$-optimal policies.
• To extend optimism-in-the-face-of-uncertainty to constrained MDPs using a novel linear programming formulation in occupancy measures.
• To reduce sample complexity dependence on the time horizon $H$ from cubic to quadratic, improving over prior regret-based formulations.

Proposed method

• The algorithm uses an optimistic planning approach based on the principle of optimism-in-the-face-of-uncertainty to balance exploration and exploitation.
• It constructs a set of statistically plausible transition models from observed trajectories, updating confidence intervals over time.
• Optimistic policy selection is formulated as a linear program (LP) in occupancy measures, incorporating constraints on expected cumulative costs.
• The algorithm executes the optimistic policy over multiple episodes until state-action pairs are visited sufficiently often, then updates visitation counts and repeats.
• It leverages concentration inequalities to bound model uncertainty and ensures high-probability guarantees on policy quality and constraint satisfaction.
• The sample complexity is derived using a recursive analysis of confidence intervals and a novel parameterization involving $m_1$ and $m$, leading to a tight bound in terms of $|\mathcal{S}|$, $|\mathcal{A}|$, $C$, $H$, $\epsilon$, and $\delta$.

Experimental results

Research questions

• RQ1 What is the minimum number of episodes required to guarantee an $\epsilon$-optimal policy in episodic finite-horizon CMDPs with unknown transitions?
• RQ2 Can optimism-in-the-face-of-uncertainty be effectively extended to constrained MDPs with provable PAC guarantees?
• RQ3 How does the sample complexity scale with the state space size $|\mathcal{S}|$, action space size $|\mathcal{A}|$, time horizon $H$, and constraint structure?
• RQ4 Can the dependence on the time horizon $H$ be reduced from cubic to quadratic in the sample complexity bound?
• RQ5 What is the impact of the maximum number of successor states $C$ on the overall sample efficiency of the learning algorithm?

Key findings

• The proposed UC-CFH algorithm achieves a sample complexity of $\tilde{\mathcal{O}}\big{(}\frac{|\mathcal{S}||\mathcal{A}|C^{2}H^{2}}{\epsilon^{2}}\log\frac{1}{\delta}\big{)}$ for achieving an $\epsilon$-optimal policy with probability at least $1-\delta$.
• The sample complexity exhibits quadratic dependence on the time horizon $H$, improving upon prior cubic bounds derived from regret-based formulations.
• When $C \ll |\mathcal{S}|$, the algorithm achieves linear dependence on $|\mathcal{S}|$ and $|\mathcal{A}|$, making it scalable to large state and action spaces.
• The algorithm ensures that constraints are satisfied within $\epsilon$-tolerance with high probability, providing a formal guarantee on constraint violation.
• The analysis establishes that the number of episodes during which the agent plays an $\epsilon$-suboptimal policy is bounded by the derived sample complexity with high probability.
• The method achieves these guarantees without requiring a generative model, relying only on observed trajectories, which enhances practical applicability.