[Paper Review] Regret Bounds for Stochastic Shortest Path Problems with Linear Function Approximation

Daniel Vial, Advait Parulekar+2 more|arXiv (Cornell University)|May 4, 2021

Advanced Bandit Algorithms Research27 references4 citations

TL;DR

This paper presents the first statistically and computationally efficient reinforcement learning algorithm for stochastic shortest path problems using linear function approximation (LFA), achieving sublinear regret with stationary policies. It introduces Optimistic Approximate Fixed Points (OAFPs) and provides regret bounds scaling as $\sqrt{K}$, $K^{3/4}$, or $K^{5/6}$ depending on assumptions, under minimal conditions including positive minimal cost and proper policies.

ABSTRACT

We propose an algorithm that uses linear function approximation (LFA) for stochastic shortest path (SSP). Under minimal assumptions, it obtains sublinear regret, is computationally efficient, and uses stationary policies. To our knowledge, this is the first such algorithm in the LFA literature (for SSP or other formulations). Our algorithm is a special case of a more general one, which achieves regret square root in the number of episodes given access to a certain computation oracle.

Motivation & Objective

• Address the lack of efficient, stationary-policy algorithms in linear function approximation for stochastic shortest path (SSP) problems.
• Develop a computationally efficient algorithm that achieves sublinear regret independent of state and action space size.
• Provide theoretical guarantees for regret scaling under minimal assumptions, including positive minimal cost and existence of proper policies.
• Bridge the gap between theoretical efficiency and practical deployment by using stationary policies.
• Extend prior work on finite-horizon LFA to infinite-horizon SSP settings with minimal structural assumptions.

Proposed method

• Propose a novel framework based on Optimistic Approximate Fixed Points (OAFPs), which are d-dimensional vectors with small Bellman error relative to a data-driven operator $\hat{G}_t$.
• Design Algorithm 1 that uses an oracle to compute OAFPs and update policies, reducing regret minimization to OAFP computation.
• Introduce a confidence set construction using subgaussian concentration and matrix concentration inequalities to bound estimation errors.
• Use a least-squares regression with regularization and adaptive feature weighting to estimate the optimal weight vector $w^\star$ in the LFA setting.
• Leverage net covering arguments and determinant bounds to control the complexity of function approximation over feature space.
• Establish regret bounds via a reduction to OAFP computation quality, with oracle access enabling $\sqrt{K}$ regret under favorable conditions.

Experimental results

Research questions

• RQ1 Can a statistically and computationally efficient algorithm be designed for stochastic shortest path problems using linear function approximation with stationary policies?
• RQ2 What regret scaling is achievable under minimal assumptions such as positive minimal cost and existence of proper policies?
• RQ3 How does the quality of the OAFP oracle affect the overall regret bound in the SSP setting?
• RQ4 Can sublinear regret be achieved without requiring knowledge of the horizon or non-stationary policies?
• RQ5 Under what conditions on the feature representation (e.g., orthogonality) can $\sqrt{K}$ regret be achieved?

Key findings

• The proposed algorithm achieves $\sqrt{K}$ regret when features are orthogonal in a specific sense, under Assumptions 1–2.
• With only Assumptions 1–2, the regret scales as $K^{5/6}$, representing a significant improvement over prior work.
• When all stationary policies are proper, the regret bound improves to $K^{3/4}$, demonstrating the impact of policy structure on performance.
• The algorithm is the first to achieve both statistical and computational efficiency in the LFA setting for SSP with stationary policies.
• The framework reduces regret minimization to OAFP computation, and the paper provides oracle implementations that yield the stated regret bounds.