[Paper Review] Speeding Up Planning in Markov Decision Processes via Automatically Constructed Abstractions

Alejandro Isaza, Csaba Szepesvári+2 more|arXiv (Cornell University)|Jun 13, 2012

Bayesian Modeling and Causal Inference14 references3 citations

TL;DR

This paper proposes an automated, multi-level abstraction hierarchy for speeding up planning in medium-sized Markov Decision Processes (MDPs), particularly stochastic shortest path problems. By constructing increasingly coarser abstractions and recursively refining policies from the topmost level, the method achieves up to a 100x speedup over state-of-the-art solvers while maintaining near-optimal solutions, with theoretical bounds on optimality loss.

ABSTRACT

In this paper, we consider planning in stochastic shortest path (SSP) problems, a subclass of Markov Decision Problems (MDP). We focus on medium-size problems whose state space can be fully enumerated. This problem has numerous important applications, such as navigation and planning under uncertainty. We propose a new approach for constructing a multi-level hierarchy of progressively simpler abstractions of the original problem. Once computed, the hierarchy can be used to speed up planning by first finding a policy for the most abstract level and then recursively refining it into a solution to the original problem. This approach is fully automated and delivers a speed-up of two orders of magnitude over a state-of-the-art MDP solver on sample problems while returning near-optimal solutions. We also prove theoretical bounds on the loss of solution optimality resulting from the use of abstractions.

Motivation & Objective

• To address the computational inefficiency of planning in medium-sized MDPs with fully enumerable state spaces.
• To develop a fully automated method for generating hierarchical abstractions that simplify planning.
• To reduce planning time significantly without sacrificing solution quality.
• To provide theoretical guarantees on the optimality loss due to abstraction.
• To enable scalable planning in uncertain environments such as navigation and decision-making under uncertainty.

Proposed method

• Constructs a multi-level hierarchy of progressively coarser abstractions of the original MDP state space.
• Uses an automated procedure to generate abstractions based on state similarity and transition structure.
• Applies value iteration or similar planning algorithms on the most abstract level to find a coarse policy.
• Recursively refines the policy through successive levels of abstraction down to the original problem.
• Employs a projection mechanism to map policies from abstract levels back to finer-grained states.
• Integrates theoretical bounds on suboptimality to ensure solution quality is preserved within known limits.

Experimental results

Research questions

• RQ1 Can automated abstraction hierarchies significantly reduce planning time in medium-sized MDPs?
• RQ2 How much optimality is lost when using abstracted MDPs compared to solving the original problem?
• RQ3 Can the abstraction process be fully automated without manual intervention?
• RQ4 Does the recursive refinement process maintain near-optimal performance across multiple abstraction levels?
• RQ5 What theoretical guarantees can be provided on the suboptimality of solutions derived from abstractions?

Key findings

• The proposed method achieves a speed-up of up to two orders of magnitude (100x) over a state-of-the-art MDP solver on benchmark problems.
• The solutions produced are near-optimal, with theoretical bounds on the suboptimality gap due to abstraction.
• The abstraction hierarchy is fully automated, requiring no manual design or tuning.
• The recursive policy refinement process effectively transfers knowledge from coarse to fine levels.
• The method is effective for real-world applications such as navigation and decision-making under uncertainty.
• The approach maintains strong performance across diverse problem instances with varying complexity.