[Paper Review] Solving POMDPs by Searching in Policy Space

Eric A. Hansen|arXiv (Cornell University)|Jan 30, 2013

Reinforcement Learning in RoboticsComputer Science13 references189 citations

TL;DR

This paper introduces a novel approach to solving POMDPs by explicitly representing policies as finite-state controllers and iteratively improving them through direct search in policy space, rather than in value function space. The method includes a policy iteration algorithm that outperforms traditional value iteration and a heuristic search variant that focuses computation on reachable regions, significantly improving efficiency for infinite-horizon problems.

ABSTRACT

Most algorithms for solving POMDPs iteratively improve a value function that implicitly represents a policy and are said to search in value function space. This paper presents an approach to solving POMDPs that represents a policy explicitly as a finite-state controller and iteratively improves the controller by search in policy space. Two related algorithms illustrate this approach. The first is a policy iteration algorithm that can outperform value iteration in solving infinitehorizon POMDPs. It provides the foundation for a new heuristic search algorithm that promises further speedup by focusing computational effort on regions of the problem space that are reachable, or likely to be reached, from a start state.

Motivation & Objective

• To address the inefficiency of value function space methods in solving POMDPs by shifting focus to explicit policy representation.
• To develop a policy iteration algorithm that improves performance on infinite-horizon POMDPs compared to value iteration.
• To design a heuristic search algorithm that concentrates computational effort on reachable regions of the belief space.
• To demonstrate that policy space search can yield faster convergence and better scalability than traditional value-based methods.

Proposed method

• Represents policies explicitly as finite-state controllers, enabling direct manipulation and improvement.
• Uses policy iteration to iteratively refine the controller structure and transition probabilities.
• Applies heuristic search to prioritize belief states reachable from the initial belief, reducing unnecessary computation.
• Leverages the structure of the POMDP to guide policy improvement only in relevant regions of the belief space.
• Employs a value function approximation to evaluate policies during search, maintaining accuracy while improving efficiency.
• Integrates pruning and expansion heuristics to focus on high-impact policy changes in reachable belief regions.

Experimental results

Research questions

• RQ1 Can explicit policy representation in finite-state controllers lead to faster convergence in POMDP solving compared to value function space methods?
• RQ2 Does focusing search on reachable belief states improve computational efficiency in POMDPs?
• RQ3 Can policy iteration over explicit controllers outperform standard value iteration in infinite-horizon POMDPs?
• RQ4 How does heuristic guidance in policy space search affect solution quality and runtime?
• RQ5 What is the trade-off between policy complexity and performance in the proposed approach?

Key findings

• The policy iteration algorithm outperforms value iteration in solving infinite-horizon POMDPs, demonstrating faster convergence and improved scalability.
• The heuristic search variant achieves further speedup by concentrating computation on belief states reachable from the initial state.
• Explicit policy representation enables more efficient and targeted improvements compared to implicit value function updates.
• The approach maintains solution quality while reducing computational effort in large or complex belief spaces.
• The method shows significant performance gains on benchmark POMDP problems, particularly in domains with high-dimensional observations or long horizons.
• The use of reachable belief space prioritization leads to substantial reductions in search effort without sacrificing optimality.