[Paper Review] Learning a Machine for the Decision in a Partially Observable Markov Universe

Frédéric Dambreville|arXiv (Cornell University)|Aug 11, 2004

Machine Learning and Algorithms6 references3 citations

TL;DR

This paper proposes a cross-entropy optimization method to learn a hierarchical hidden Markov model (HHMM) that approximates an optimal decision policy in partially observable Markov decision processes (POMDPs). By modeling actions as a parameterized probabilistic function of observations, the approach directly learns near-optimal strategies without dynamic programming, achieving strong performance on a multi-agent tracking task with adaptive cooperation strategies.

ABSTRACT

In this paper, we are interested in optimal decisions in a partially observable Markov universe. Our viewpoint departs from the dynamic programming viewpoint: we are directly approximating an optimal strategic tree depending on the observation. This approximation is made by means of a parameterized probabilistic law. In this paper, a particular family of hidden Markov models, with input and output, is considered as a learning framework. A method for optimizing the parameters of these HMMs is proposed and applied. This optimization method is based on the cross-entropic principle.

Motivation & Objective

• To address the intractability of exact POMDP solutions by directly approximating optimal decision trees using parameterized probabilistic laws.
• To develop a learning framework based on hierarchical HMMs with input and output for modeling action policies conditioned on observations.
• To optimize HHMM parameters using the cross-entropy method for near-optimal policy learning in POMDPs.
• To demonstrate that hierarchical structure improves convergence and performance over non-hierarchical models in complex decision tasks.

Proposed method

• A hierarchical HMM (HHMM) is used to model the policy, where the hidden states represent machine memory and the output represents actions conditioned on observations.
• The policy is parameterized as a conditional probability $ h(x|y) $, approximating the optimal stochastic policy $ \pi(x|y) $.
• The cross-entropy method is applied to iteratively update HHMM parameters by minimizing the cross-entropy between the estimated policy and the optimal policy.
• The learning process uses a reward-based evaluation function to guide parameter updates toward higher-performing policies.
• The method avoids dynamic programming by directly optimizing the policy using Monte Carlo sampling and importance sampling within the cross-entropy framework.
• The hierarchical structure enables multi-level information diffusion, allowing compact modeling of complex decision dependencies.

Experimental results

Research questions

• RQ1 Can a parameterized probabilistic model based on hierarchical HMMs effectively approximate the optimal decision policy in a POMDP?
• RQ2 Does the cross-entropy method enable efficient and stable learning of such policies without relying on dynamic programming?
• RQ3 How does hierarchical structure affect convergence speed and solution quality in POMDP policy learning?
• RQ4 Can the learned policy discover multiple distinct, context-sensitive strategies, such as cooperative or deceptive behaviors?

Key findings

• The cross-entropy method successfully optimized the HHMM parameters, achieving a mean evaluation of 69 under the best policy, indicating strong near-optimality.
• The 4-level HHMM (Λ=4) achieved the highest performance under both weak (65) and strong (66) criteria, outperforming lower-level models.
• Despite higher complexity, the 4-level HHMM required only 758 parameters, significantly fewer than the 4320-parameter 2-level model, showing better efficiency.
• The algorithm demonstrated global convergence, with multiple runs converging to similar optimal values, indicating robustness.
• The learned policy discovered two distinct strategies: cooperative tracking and one mobile blocking escape paths, demonstrating adaptive, context-aware behavior.
• The method proved effective even with limited state spaces, suggesting promise for more complex, continuous, or hybrid discrete-continuous problems.