[Paper Review] A Spectral Approach to Off-Policy Evaluation for POMDPs

Yash Nair, Nan Jiang|arXiv (Cornell University)|Sep 22, 2021

Optimization and Search Problems15 references4 citations

TL;DR

This paper proposes a spectral method for off-policy evaluation in POMDPs that relaxes the strict invertibility assumptions of prior work by leveraging multi-step observable proxies for hidden states. By extending past and future histories and using eigendecomposition, the approach improves estimation accuracy and generalizability, while also enabling a new Importance Sampling algorithm with weaker rank, positivity, and distinctness conditions instead of sufficiency assumptions.

ABSTRACT

We consider off-policy evaluation (OPE) in Partially Observable Markov Decision Processes, where the evaluation policy depends only on observable variables but the behavior policy depends on latent states (Tennenholtz et al. (2020a)). Prior work on this problem uses a causal identification strategy based on one-step observable proxies of the hidden state, which relies on the invertibility of certain one-step moment matrices. In this work, we relax this requirement by using spectral methods and extending one-step proxies both into the past and future. We empirically compare our OPE methods to existing ones and demonstrate their improved prediction accuracy and greater generality. Lastly, we derive a separate Importance Sampling (IS) algorithm which relies on rank, distinctness, and positivity conditions, and not on the strict sufficiency conditions of observable trajectories with respect to the reward and hidden-state structure required by Tennenholtz et al. (2020a).

Motivation & Objective

• Address the limitations of prior off-policy evaluation (OPE) methods in POMDPs that rely on invertible one-step moment matrices.
• Improve generalizability by allowing richer observation spaces where the observation space exceeds the hidden state space.
• Develop a method that estimates joint probabilities instead of conditional probabilities to avoid issues with rare or zero-probability events.
• Extend observable proxies into both past and future trajectories to reduce reliance on strong invertibility assumptions.
• Derive a new Importance Sampling algorithm based on rank, positivity, and distinctness conditions, avoiding the strict sufficiency requirements of prior work.

Proposed method

• Use spectral decomposition techniques inspired by PSRs and HMMs to identify latent state distributions from observable trajectories.
• Construct multi-step observable proxies by extending the history and future windows around the confounder to improve rank conditions.
• Employ eigendecomposition of moment matrices (e.g., $P^r_{i,z_i}$, $Q^r_{i,z_i,z_{i+1}}$) to identify hidden state distributions without requiring matrix invertibility.
• Identify $P^b(U_{0:H}| au^o_H)$ using the full-rank property of $P^b(R_{0:H}|U_{0:H})$ and Moore-Penrose inversion of observable joint distributions.
• Formulate a new Importance Sampling estimator using identified weights $W(v, au^o)$, derived from $P^b(r_i|u_i)$, $P^b(u_{0:H}| au^o)$, and $ ilde{ ho}_b(u_{0:H})$.
• Ensure consistent ordering of hidden states via distinctness assumptions (Assumption 10), enabling correct alignment of probability estimates.

Experimental results

Research questions

• RQ1 Can we relax the invertibility requirement of one-step moment matrices in POMDP OPE by using multi-step observable proxies?
• RQ2 Does extending observable proxies into both past and future trajectories improve the identifiability and accuracy of off-policy value estimation?
• RQ3 Can we derive a new Importance Sampling algorithm that avoids the sufficiency assumptions of prior work by relying only on rank, positivity, and distinctness?
• RQ4 How does the performance of the proposed spectral OPE estimator compare to existing methods in terms of predictive accuracy and robustness?
• RQ5 To what extent does the method generalize to POMDPs with larger observation spaces than hidden state spaces?

Key findings

• The proposed spectral OPE method achieves improved predictive accuracy compared to prior methods, especially in settings with high-dimensional observations.
• The method relaxes the need for invertibility of one-step moment matrices by using multi-step histories and futures as proxies, enhancing applicability to real-world POMDPs.
• By estimating joint probabilities instead of conditionals, the method avoids issues with rare or zero-probability conditioning events.
• The eigendecomposition-based identification procedure enables consistent estimation of $P^b(r_i|u_i)$ and $P^b(u_{0:H}| au^o_H)$ under weaker assumptions than prior work.
• The new Importance Sampling algorithm is identifiable under rank, positivity, and distinctness conditions, avoiding the strict sufficiency assumptions required by Tennenholtz et al. (2020a).
• Empirical validation confirms that the extended proxy approach leads to more stable and accurate OPE estimates across multiple environments.