[Paper Review] Large-Scale Markov Decision Problems via the Linear Programming Dual

Yasin Abbasi-Yadkori, Peter L. Bartlett+2 more|arXiv (Cornell University)|Jan 6, 2019

Reinforcement Learning in Robotics27 references7 citations

TL;DR

This paper proposes a novel approach to solving large-scale Markov decision processes (MDPs) by optimizing over a low-dimensional family of policies derived from approximate occupancy measures via the dual of a linear programming formulation. It introduces an efficient stochastic subgradient algorithm that scales with the subspace dimension rather than the state space, achieving provable excess loss bounds in both average and discounted cost settings, with empirical validation in a queueing network application.

ABSTRACT

We consider the problem of controlling a fully specified Markov decision process (MDP), also known as the planning problem, when the state space is very large and calculating the optimal policy is intractable. Instead, we pursue the more modest goal of optimizing over some small family of policies. Specifically, we show that the family of policies associated with a low-dimensional approximation of occupancy measures yields a tractable optimization. Moreover, we propose an efficient algorithm, scaling with the size of the subspace but not the state space, that is able to find a policy with low excess loss relative to the best policy in this class. To the best of our knowledge, such results did not exist in the literature previously. We bound excess loss in the average cost and discounted cost cases, which are treated separately. Preliminary experiments show the effectiveness of the proposed algorithms in a queueing application.

Motivation & Objective

• Address the intractability of solving large-scale MDPs when the state space is too large for standard dynamic programming or linear programming methods.
• Develop a tractable optimization framework that scales with a low-dimensional policy class rather than the full state space.
• Provide theoretical excess loss bounds relative to the best policy in the parametric class for both average and discounted cost MDPs.
• Design an efficient algorithm that avoids reliance on samples from the optimal policy's distribution, unlike prior Approximate Linear Programming (ALP) methods.
• Demonstrate empirical effectiveness through experiments in a queueing network with a 10,000-fold reduction in feature dimension.

Proposed method

• Formulate the MDP planning problem using the dual of a linear program, focusing on occupancy measures and their low-dimensional approximation via a feature matrix Φ.
• Parameterize policies through linear functions in the dual space, enabling optimization over a low-dimensional subspace of policies.
• Use stochastic subgradient descent to minimize a surrogate loss function that combines the objective and constraint violations of the dual LP.
• Introduce a randomized algorithm that samples constraints on-the-fly, ensuring scalability to large state spaces by depending only on the subspace dimension.
• Leverage the structure of the dual LP to derive excess loss bounds in both average and discounted cost settings using novel proof techniques.
• Incorporate a backwards simulator to access states that transition into any given state, enabling constraint sampling and policy evaluation.

Experimental results

Research questions

• RQ1 Can we achieve provable excess loss bounds for large-scale MDPs when optimizing over a low-dimensional policy class?
• RQ2 Is it possible to design an algorithm that scales with the subspace dimension rather than the full state space size in MDP planning?
• RQ3 How can we derive theoretical performance guarantees in both average cost and discounted cost MDP settings using dual LP formulations?
• RQ4 Can we avoid the dependency on samples from the optimal policy’s distribution, as in prior ALP methods, while still maintaining convergence and error bounds?
• RQ5 How effective is the proposed method in practice compared to heuristic policies in real-world applications like queueing networks?

Key findings

• The proposed algorithm achieves excess loss bounds relative to the best policy in the parametric class for both average and discounted cost MDPs, with bounds derived using novel proof techniques.
• The method scales with the dimension of the policy subspace, not the size of the state space, enabling efficient computation in large-scale problems.
• In a queueing network with 372 features (a 10^4 reduction from the original state space), the algorithm outperformed two common heuristics (LONGER and LBFS) in terms of average loss.
• The stochastic subgradient descent algorithm successfully minimized the surrogate loss, leading to lower average losses, as shown in the right plot of Figure 5.
• The approach differs fundamentally from prior ALP methods by working with stationary distributions rather than value functions, enabling new theoretical and algorithmic developments.
• Empirical results confirm that minimizing the surrogate loss function leads to policies with lower actual average loss, validating the theoretical framework.