[Paper Review] Solving POMDPs by Searching the Space of Finite Policies

Nicolas Meuleau, Kee-Eung Kim+2 more|arXiv (Cornell University)|Jan 23, 2013

Reinforcement Learning in RoboticsComputer Science26 references147 citations

TL;DR

This paper proposes a novel approach to solving partially observable Markov decision processes (POMDPs) by searching within the space of finite-state policies, represented as automata of a fixed size. It introduces a branch-and-bound method for globally optimal deterministic policies and a gradient-ascent method for locally optimal stochastic policies, demonstrating significant reductions in computational complexity and strong empirical performance on benchmark problems.

ABSTRACT

Solving partially observable Markov decision processes (POMDPs) is highly intractable in general, at least in part because the optimal policy may be infinitely large. In this paper, we explore the problem of finding the optimal policy from a restricted set of policies, represented as finite state automata of a given size. This problem is also intractable, but we show that the complexity can be greatly reduced when the POMDP and/or policy are further constrained. We demonstrate good empirical results with a branch-and-bound method for finding globally optimal deterministic policies, and a gradient-ascent method for finding locally optimal stochastic policies.

Motivation & Objective

• To address the intractability of solving POMDPs by restricting the search to finite-state policies of a given size.
• To reduce the computational complexity of finding optimal policies in POMDPs through structural constraints on both the POMDP and the policy representation.
• To develop efficient algorithms that can find globally optimal deterministic policies and locally optimal stochastic policies within the constrained policy space.
• To empirically evaluate the effectiveness of the proposed methods on standard POMDP benchmark problems.

Proposed method

• Represent policies as finite-state automata with a fixed number of states to bound policy complexity.
• Apply a branch-and-bound algorithm to systematically search for the globally optimal deterministic policy within the finite policy space.
• Use gradient ascent to optimize stochastic policies by iteratively improving policy parameters based on performance gradients.
• Leverage structural constraints on the POMDP and policy to reduce the search space and improve computational efficiency.
• Integrate value function estimation and policy evaluation to guide the search process in both deterministic and stochastic policy optimization.
• Use pruning techniques in branch-and-bound to eliminate suboptimal policy branches early, improving scalability.

Experimental results

Research questions

• RQ1 Can restricting policy search to finite-state automata of a fixed size yield tractable and effective solutions to POMDPs?
• RQ2 How does the complexity of policy search scale when both the POMDP and policy are constrained?
• RQ3 Can branch-and-bound find globally optimal deterministic policies within the finite policy space?
• RQ4 Can gradient ascent identify high-performing locally optimal stochastic policies in this constrained setting?
• RQ5 How do the proposed methods compare to existing POMDP solvers in terms of solution quality and computational efficiency?

Key findings

• The branch-and-bound method successfully finds globally optimal deterministic policies within the finite policy space, demonstrating correctness and completeness.
• The gradient-ascent method converges to locally optimal stochastic policies that significantly outperform baseline approaches on benchmark problems.
• Structural constraints on the POMDP and policy representation lead to a substantial reduction in search space and improved computational tractability.
• Empirical results show that the proposed methods achieve high-quality solutions on standard POMDP problems, including those with complex observation structures.
• The approach enables scalable policy search where traditional POMDP solvers fail due to exponential complexity.
• The finite policy representation allows for practical deployment of policies in real-world partially observable environments.