[Paper Review] Planning in Observable POMDPs in Quasipolynomial Time

Noah Golowich, Ankur Moitra+1 more|arXiv (Cornell University)|Jan 12, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper presents the first quasipolynomial-time algorithm for planning in one-step observable POMDPs, leveraging a novel filter stability analysis under a weak observability assumption that well-separated states yield well-separated observations. The key contribution is a provably efficient algorithm with near-optimal policy representation, complemented by a matching hardness result under the Exponential Time Hypothesis.

ABSTRACT

Partially Observable Markov Decision Processes (POMDPs) are a natural and general model in reinforcement learning that take into account the agent's uncertainty about its current state. In the literature on POMDPs, it is customary to assume access to a planning oracle that computes an optimal policy when the parameters are known, even though the problem is known to be computationally hard. Almost all existing planning algorithms either run in exponential time, lack provable performance guarantees, or require placing strong assumptions on the transition dynamics under every possible policy. In this work, we revisit the planning problem and ask: are there natural and well-motivated assumptions that make planning easy? Our main result is a quasipolynomial-time algorithm for planning in (one-step) observable POMDPs. Specifically, we assume that well-separated distributions on states lead to well-separated distributions on observations, and thus the observations are at least somewhat informative in each step. Crucially, this assumption places no restrictions on the transition dynamics of the POMDP; nevertheless, it implies that near-optimal policies admit quasi-succinct descriptions, which is not true in general (under standard hardness assumptions). Our analysis is based on new quantitative bounds for filter stability -- i.e. the rate at which an optimal filter for the latent state forgets its initialization. Furthermore, we prove matching hardness for planning in observable POMDPs under the Exponential Time Hypothesis.

Motivation & Objective

• To identify natural assumptions under which planning in POMDPs becomes computationally tractable despite the problem's known intractability.
• To overcome the Curse of History by showing that near-optimal policies admit quasi-succinct descriptions under weak observability.
• To establish a provably efficient algorithm for planning in one-step observable POMDPs with polynomial-time complexity in the number of states and observations.
• To provide matching hardness results under the Exponential Time Hypothesis, showing the algorithm's time bound is nearly optimal.

Proposed method

• Introduces the concept of one-step observability, where well-separated state distributions lead to well-separated observation distributions.
• Develops new quantitative bounds on filter stability, measuring how quickly the belief about the hidden state forgets its initialization.
• Uses a belief contraction analysis to show that the total variation distance between true and approximate beliefs decays exponentially with time under the observability assumption.
• Applies a recursive decomposition of the belief update process to bound the number of distinct belief states that need to be considered.
• Employs a dynamic programming approach over a carefully constructed discretization of the belief space, exploiting the quasi-succinctness of near-optimal policies.
• Proves a matching hardness result by reducing from 3-SAT, showing that any algorithm solving the problem in sub-exponential time would violate the Exponential Time Hypothesis.

Experimental results

Research questions

• RQ1 Can planning in POMDPs be made efficient under natural and well-motivated assumptions about observation informativeness?
• RQ2 Does the assumption that well-separated states lead to well-separated observations imply that near-optimal policies can be described succinctly?
• RQ3 Is it possible to design a quasipolynomial-time algorithm for planning in observable POMDPs without imposing strong restrictions on transition dynamics?
• RQ4 What is the computational complexity frontier of planning in observable POMDPs, and how does it relate to fundamental complexity assumptions like the Exponential Time Hypothesis?
• RQ5 Can the rate of belief convergence (filter stability) be quantitatively bounded to enable efficient planning?

Key findings

• The paper presents a quasipolynomial-time algorithm for planning in one-step observable POMDPs, with runtime bounded by $\exp(\tilde{O}(\log^2(SAH)))$, where $S$, $A$, and $H$ are the number of states, actions, and horizon length.
• Under the one-step observability assumption, near-optimal policies admit quasi-succinct descriptions, meaning they can be represented with quasipolynomially many parameters.
• The paper establishes a new quantitative bound on filter stability: the total variation distance between true and approximate beliefs decays as $\exp(-\Omega(\gamma^2 t))$, where $\gamma$ is the observability parameter and $t$ is time.
• The belief contraction rate is shown to be tight up to constants, as the quadratic dependence on $\gamma$ in the decay rate cannot be improved.
• A matching hardness result is proven: under the Exponential Time Hypothesis, no algorithm can solve the planning problem in time $\exp(O((SAH/\epsilon)^c))$ for any $c>0$ and $\epsilon<1/m$, where $m$ is the number of clauses in a 3-SAT instance.
• The result implies that the proposed quasipolynomial-time algorithm is nearly optimal in terms of runtime complexity.