[Paper Review] Adaptive Belief Discretization for POMDP Planning

Divya Grover, Christos Dimitrakakis|arXiv (Cornell University)|Apr 15, 2021

Formal Methods in Verification14 references4 citations

TL;DR

This paper proposes an adaptive belief discretization method for online POMDP planning that reduces memory and computational complexity by dynamically refining belief space covers based on lookahead depth. It provides theoretical bounds on planning error and covering number, enabling a computationally efficient solver (FMP) that matches or exceeds state-of-the-art performance with minimal tuning.

ABSTRACT

Partially Observable Markov Decision Processes (POMDP) is a widely used model to represent the interaction of an environment and an agent, under state uncertainty. Since the agent does not observe the environment state, its uncertainty is typically represented through a probabilistic belief. While the set of possible beliefs is infinite, making exact planning intractable, the belief space's complexity (and hence planning complexity) is characterized by its covering number. Many POMDP solvers uniformly discretize the belief space and give the planning error in terms of the (typically unknown) covering number. We instead propose an adaptive belief discretization scheme, and give its associated planning error. We furthermore characterize the covering number with respect to the POMDP parameters. This allows us to specify the exact memory requirements on the planner, needed to bound the value function error. We then propose a novel, computationally efficient solver using this scheme. We demonstrate that our algorithm is highly competitive with the state of the art in a variety of scenarios.

Motivation & Objective

• To address the high memory and computational cost of exact POMDP planning due to the infinite belief space.
• To reduce planning error by replacing uniform belief discretization with an adaptive scheme tailored to lookahead depth.
• To provide explicit theoretical bounds on the covering number of the belief space in terms of POMDP parameters.
• To design a planner with direct control over planning error through tunable discretization parameters.
• To develop a computationally efficient online POMDP solver that maintains strong performance with minimal hyperparameter tuning.

Proposed method

• Proposes an adaptive discretization scheme that generates belief space covers with size dependent on lookahead depth and error tolerance.
• Uses epsilon-covers to approximate the belief space, where cover size is minimized via dynamic refinement based on depth-dependent error bounds.
• Derives theoretical upper bounds on the covering number of the belief space using POMDP parameters such as state space size, action space size, and discount factor.
• Integrates the adaptive discretization into a novel online planner (FMP) that builds deep lookahead trees with controlled error.
• Employs a belief tree search strategy that prioritizes regions of the belief space with higher uncertainty or potential value, guided by the adaptive cover.
• Tunes the discretization sequence $\epsilon_d$ and initial resolution $h_0$ to balance error and computational cost within online planning limits.

Experimental results

Research questions

• RQ1 How can belief space discretization be made adaptive to lookahead depth to reduce memory and computational complexity in POMDP planning?
• RQ2 What is the theoretical relationship between the covering number of the belief space and the underlying POMDP parameters?
• RQ3 Can an adaptive discretization scheme provide tighter bounds on planning error compared to uniform discretization?
• RQ4 How does the proposed method’s error bound relate to the planner’s memory and computational requirements?
• RQ5 To what extent can a simple, low-tuning planner achieve performance competitive with state-of-the-art POMDP solvers?

Key findings

• The FMP planner achieves competitive performance on RockSample and RandomPOMDP environments, with average discounted rewards matching or exceeding those of POMCP and DESPOT, despite using a slower Python implementation.
• On the RS[15,15] environment, FMP achieved a discounted reward of 15.67 ± 0.37, outperforming POMCP (15.32 ± 0.28) and DESPOT (18.64 ± 0.28) in the reported results.
• For RandomPOMDP, FMP performed best in low-sparsity settings (e.g., RP[30,0.9] with 6.36), showing strong performance when many beliefs exist initially.
• The planner’s performance was stable across environments with minimal tuning, requiring only $h_0$ and $\epsilon_d$ sequences, unlike POMCP and DESPOT which require multiple complex hyperparameters.
• The theoretical analysis shows that the covering number of the belief space is bounded in terms of POMDP parameters, enabling precise memory and error control.
• The adaptive discretization scheme significantly reduces the size of the belief cover compared to uniform methods, directly lowering memory and computational overhead.