[Paper Review] Implicit Finite-Horizon Approximation and Efficient Optimal Algorithms for Stochastic Shortest Path

Liyu Chen, Mehdi Jafarnia-Jahromi+2 more|arXiv (Cornell University)|Jun 15, 2021

Advanced Bandit Algorithms Research30 references5 citations

TL;DR

This paper introduces a novel implicit finite-horizon approximation technique to design minimax-optimal regret algorithms for the Stochastic Shortest Path (SSP) problem. It proposes two new algorithms: LCB-Advantage-SSP, the first model-free SSP algorithm with minimax optimal regret under strictly positive costs, and SVI-SSP, a model-based algorithm that achieves minimax optimality even with zero-cost actions, both with highly sparse updates and parameter-free variants.

ABSTRACT

We introduce a generic template for developing regret minimization algorithms in the Stochastic Shortest Path (SSP) model, which achieves minimax optimal regret as long as certain properties are ensured. The key of our analysis is a new technique called implicit finite-horizon approximation, which approximates the SSP model by a finite-horizon counterpart only in the analysis without explicit implementation. Using this template, we develop two new algorithms: the first one is model-free (the first in the literature to our knowledge) and minimax optimal under strictly positive costs; the second one is model-based and minimax optimal even with zero-cost state-action pairs, matching the best existing result from [Tarbouriech et al., 2021b]. Importantly, both algorithms admit highly sparse updates, making them computationally more efficient than all existing algorithms. Moreover, both can be made completely parameter-free.

Motivation & Objective

• Address the lack of efficient, model-free regret minimization algorithms in the Stochastic Shortest Path (SSP) setting with minimax optimal regret.
• Overcome the computational inefficiency of existing model-based algorithms that rely on full-planning and have high space complexity.
• Develop a generic algorithmic template that achieves minimax optimal regret under mild analytical conditions.
• Design algorithms that support sparse updates and can be made completely parameter-free without sacrificing regret bounds.
• Enable optimal performance in SSP environments with zero-cost state-action pairs, a challenge not fully addressed by prior work.

Proposed method

• Propose a new analysis framework called implicit finite-horizon approximation, which approximates SSP as a finite-horizon MDP only in theoretical analysis, not in algorithmic implementation.
• Design LCB-Advantage-SSP as a model-free algorithm using reference-advantage decomposition and a doubling-based empirical upper bound on the value function to enable parameter-free operation.
• Develop SVI-SSP as a model-based algorithm that uses one-step planning instead of full-planning, reducing time complexity and enabling logarithmic updates per state-action pair.
• Incorporate a Bernstein-style bonus term in SVI-SSP to bound variance recursively, ensuring minimax optimality even when $c_{\min} = 0$.
• Ensure both algorithms achieve minimax optimal regret by satisfying key structural properties in the analysis, leveraging the implicit finite-horizon approximation framework.
• Use a reference-advantage decomposition with frequent reference value updates to reduce sample complexity and improve lower-order regret terms.

Experimental results

Research questions

• RQ1 Can a model-free algorithm be designed for SSP that achieves minimax optimal regret under strictly positive costs?
• RQ2 Can a model-based SSP algorithm be developed that maintains minimax optimality even when some state-action pairs have zero cost?
• RQ3 Can both model-free and model-based SSP algorithms be made computationally efficient through sparse updates and reduced planning overhead?
• RQ4 Can the proposed algorithms be made completely parameter-free without degrading regret performance?
• RQ5 Can the implicit finite-horizon approximation technique enable horizon-free regret bounds while avoiding the space complexity blowup of explicit horizon-based methods?

Key findings

• LCB-Advantage-SSP achieves a regret bound of $\tilde{\mathcal{O}}(B_{\star}\sqrt{SAK} + B_{\star}^{5}S^{2}A/c_{\min}^{4})$, which is minimax optimal when $c_{\min} > 0$.
• SVI-SSP achieves a regret bound of $\tilde{\mathcal{O}}(B_{\star}\sqrt{SAK} + B_{\star}S^{2}A)$, matching the best existing result from Tarbouriech et al. (2021b), even when $c_{\min} = 0$.
• Both LCB-Advantage-SSP and SVI-SSP achieve space complexity of $\tilde{\mathcal{O}}(SA)$, significantly lower than existing model-based algorithms with $\Omega(S^2A)$ space.
• SVI-SSP performs only logarithmic updates per state-action pair and uses one-step planning, making it the most computationally efficient among all compared algorithms in experiments.
• Empirical results show that SVI-SSP outperforms EB-SSP and UC-SSP in terms of regret and update efficiency, with the lowest total update time in both RandomMDP and GridWorld environments.
• Q-learning with $\epsilon$-greedy exploration exhibits linear regret, confirming the inefficacy of naive exploration in SSP settings, while LCB-Advantage-SSP and SVI-SSP show strong performance with sparse, efficient updates.