[Paper Review] Qualitative Possibilistic Mixed-Observable MDPs

Nicolas Drougard, Florent Teichteil-Königsbuch+2 more|arXiv (Cornell University)|Sep 26, 2013

Reinforcement Learning in Robotics16 references3 citations

TL;DR

This paper introduces qualitative possibilistic mixed-observable MDPs (pi-MOMDPs), a tractable extension of possibilistic POMDPs that leverages partial observability to reduce computational complexity. By assuming some state variables are fully observable, the authors propose a value iteration algorithm that achieves optimal policies under infinite-horizon planning, outperforming traditional probabilistic POMDPs in target recognition tasks with imprecise observations.

ABSTRACT

Possibilistic and qualitative POMDPs (pi-POMDPs) are counterparts of POMDPs used to model situations where the agent's initial belief or observation probabilities are imprecise due to lack of past experiences or insufficient data collection. However, like probabilistic POMDPs, optimally solving pi-POMDPs is intractable: the finite belief state space exponentially grows with the number of system's states. In this paper, a possibilistic version of Mixed-Observable MDPs is presented to get around this issue: the complexity of solving pi-POMDPs, some state variables of which are fully observable, can be then dramatically reduced. A value iteration algorithm for this new formulation under infinite horizon is next proposed and the optimality of the returned policy (for a specified criterion) is shown assuming the existence of a "stay" action in some goal states. Experimental work finally shows that this possibilistic model outperforms probabilistic POMDPs commonly used in robotics, for a target recognition problem where the agent's observations are imprecise.

Motivation & Objective

• To address the intractability of solving qualitative possibilistic POMDPs (pi-POMDPs) due to exponential growth in belief space.
• To reduce computational complexity by exploiting partial observability, where some state variables are fully observable.
• To develop a value iteration algorithm for infinite-horizon planning in this new framework.
• To demonstrate the superiority of the proposed model over standard probabilistic POMDPs in real-world robotics applications with imprecise observations.

Proposed method

• Formalizes a possibilistic version of Mixed-Observable MDPs (MOMDPs), where some state variables are fully observable, reducing belief space complexity.
• Adapts value iteration to the qualitative possibilistic framework, using possibility theory instead of probability to represent uncertainty.
• Employs a belief update mechanism that combines possibility distributions over unobserved states with deterministic observations of observed states.
• Introduces a 'stay' action in goal states to ensure convergence and optimality of the value iteration process.
• Defines a criterion for policy optimality based on possibility theory, ensuring maximal possibility of reaching goal states.
• Employs a finite-state abstraction that avoids exponential growth by exploiting the mixed-observability structure.

Experimental results

Research questions

• RQ1 Can mixed-observability reduce the computational complexity of solving qualitative possibilistic POMDPs?
• RQ2 Is it possible to design a value iteration algorithm for infinite-horizon pi-MOMDPs that guarantees policy optimality?
• RQ3 How does the performance of the proposed possibilistic model compare to standard probabilistic POMDPs in real-world recognition tasks?
• RQ4 Does the inclusion of a 'stay' action in goal states ensure convergence and optimality in the possibilistic setting?

Key findings

• The proposed pi-MOMDP framework significantly reduces the complexity of solving qualitative POMDPs by exploiting partial observability of state variables.
• The value iteration algorithm converges to an optimal policy under the specified criterion, assuming the existence of a 'stay' action in goal states.
• Experimental results show that the possibilistic model outperforms standard probabilistic POMDPs in a target recognition task with imprecise observations.
• The model achieves better performance in terms of possibility of success, particularly when observation data is sparse or unreliable.