[Paper Review] On the convergence of optimistic policy iteration for stochastic shortest path problem

Yuanlong Chen|arXiv (Cornell University)|Aug 27, 2018

Optimization and Search Problems4 references3 citations

TL;DR

This paper establishes convergence of optimistic policy iteration for the stochastic shortest path problem under discounted and undiscounted (α=1) settings, using Monte Carlo and TD(λ) methods for policy evaluation. It proves that the algorithm converges to the optimal cost-to-go function under proper policies, with convergence guaranteed via contraction arguments and bounded error propagation in value updates.

ABSTRACT

In this paper, we prove some convergence results of a special case of optimistic policy iteration algorithm for stochastic shortest path problem. We consider both Monte Carlo and $TD(λ)$ methods for the policy evaluation step under the condition that the termination state will eventually be reached almost surely.

Motivation & Objective

• To establish convergence of optimistic policy iteration for the stochastic shortest path problem with discount factor α=1.
• To analyze the convergence behavior when policy evaluation uses Monte Carlo or TD(λ) methods under proper policies.
• To prove that the algorithm converges to the optimal cost-to-go vector despite approximate policy evaluation.
• To extend prior results on policy iteration convergence to the optimistic setting with bounded error in value updates.

Proposed method

• Uses the standard dynamic programming operators T and Tμ for value and policy evaluation, respectively.
• Applies the maximum norm and weighted maximum norm to analyze convergence of value vectors.
• Employs contraction arguments and comparison principles via sequences Yt and Ytl to bound error propagation.
• Introduces a stopped process v^l(t) to handle martingale-like noise terms and ensure convergence under bounded variance.
• Uses the key inequality T_{μ_t}^{k+1}J_t ≤ TJ_t + ε for large t and k, derived from boundedness and limsup(c_t) ≤ 0.
• Establishes convergence via recursive update: J_{t+1} ≤ (1−γ_t)J_t + γ_t(TJ_t + ε) + γ_tω_t, with ε arbitrary small.

Experimental results

Research questions

• RQ1 Does optimistic policy iteration converge when policy evaluation is approximate via Monte Carlo or TD(λ) methods?
• RQ2 Can convergence be guaranteed under the α=1 case of the stochastic shortest path problem with proper policies?
• RQ3 How does bounded error in policy evaluation affect the convergence of the value function to the optimal solution?
• RQ4 What conditions ensure that the sequence of value vectors converges to the optimal cost-to-go vector despite approximation errors?
• RQ5 Can the convergence proof be extended to the optimistic policy iteration framework with general policy evaluation schemes?

Key findings

• The algorithm converges to the optimal cost-to-go vector J* under the assumption that all policies are proper and α=1.
• For any ε>0, there exists a time t(ε) such that for all t≥t(ε), the approximate policy evaluation satisfies (1−λ)∑λ^k T_{μ_t}^{k+1}J_t ≤ TJ_t + εe.
• The sequence of value vectors J_t converges to J* as t→∞, with the limsup of the error c_t bounded by 0 due to diminishing noise and error terms.
• The use of a stopped process v^l(t) ensures that the error sequence v_t does not disrupt convergence, leading to Y_t converging to εe and thus Y_t → 0 as ε→0.
• The convergence proof relies on a comparison argument using Y_t, which is shown to converge to εe, implying that the error in the value update vanishes in the limit.
• The result holds for both Monte Carlo and TD(λ) policy evaluation, provided the policy evaluation error remains bounded and the policy is proper.