[Paper Review] Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling

Tengyang Xie, Yifei Ma+1 more|arXiv (Cornell University)|Jan 1, 2019

Advanced Causal Inference Techniques53 citations

TL;DR

This paper proposes a marginalized importance sampling (MIS) estimator for off-policy evaluation in nonstationary episodic MDPs with long horizons and large action spaces. By recursively estimating state marginal distributions, MIS achieves a mean-squared error bound with polynomial dependence on horizon H, matching the Cramer-Rao lower bound up to an H factor, and demonstrates superior empirical performance in challenging RL environments.

ABSTRACT

Motivated by the many real-world applications of reinforcement learning (RL) that require safe-policy iterations, we consider the problem of off-policy evaluation (OPE) --- the problem of evaluating a new policy using the historical data obtained by different behavior policies --- under the model of nonstationary episodic Markov Decision Processes (MDP) with a long horizon and a large action space. Existing importance sampling (IS) methods often suffer from large variance that depends exponentially on the RL horizon $H$. To solve this problem, we consider a marginalized importance sampling (MIS) estimator that recursively estimates the state marginal distribution for the target policy at every step. MIS achieves a mean-squared error of $$ \frac{1}{n} \sum_{t=1}^H\mathbb{E}_{\mu}\left[\frac{d_t^\pi(s_t)^2}{d_t^\mu(s_t)^2} \Var_{\mu}\left[\frac{\pi_t(a_t|s_t)}{\mu_t(a_t|s_t)}\big( V_{t+1}^\pi(s_{t+1}) + r_t\big) \middle| s_t ight] ight] + ilde{O}(n^{-1.5}) $$ where $\mu$ and $\pi$ are the logging and target policies, $d_t^{\mu}(s_t)$ and $d_t^{\pi}(s_t)$ are the marginal distribution of the state at $t$th step, $H$ is the horizon, $n$ is the sample size and $V_{t+1}^\pi$ is the value function of the MDP under $\pi$. The result matches the Cramer-Rao lower bound in [Jiang and Li, 2016] up to a multiplicative factor of $H$. To the best of our knowledge, this is the first OPE estimation error bound with a polynomial dependence on $H$. Besides theory, we show empirical superiority of our method in time-varying, partially observable, and long-horizon RL environments.

Motivation & Objective

• To address high variance in off-policy evaluation (OPE) under long-horizon, large-action-space MDPs.
• To reduce the exponential dependence on horizon H seen in traditional importance sampling (IS) methods.
• To develop an OPE estimator with theoretically grounded error bounds that scale polynomially with H.
• To achieve theoretical and empirical improvements over existing IS-based OPE methods in nonstationary and partially observable environments.

Proposed method

• The method introduces a marginalized importance sampling (MIS) estimator that recursively estimates the state marginal distribution under the target policy at each time step.
• It leverages the ratio of state marginal probabilities $ \frac{d_t^\pi(s_t)^2}{d_t^\mu(s_t)^2} $ to reweight importance sampling contributions.
• The estimator incorporates the value function $ V_{t+1}^\pi(s_{t+1}) $ and immediate reward $ r_t $ within the importance weight computation.
• The approach uses a recursive decomposition of the value function to stabilize estimation and reduce variance.
• Theoretical analysis bounds the mean-squared error as $ \frac{1}{n} \sum_{t=1}^H \mathbb{E}_\mu\left[ \frac{d_t^\pi(s_t)^2}{d_t^\mu(s_t)^2} \Var_\mu\left[ \frac{\pi_t(a_t|s_t)}{\mu_t(a_t|s_t)} (V_{t+1}^\pi(s_{t+1}) + r_t) \mid s_t \right] \right] + \tilde{O}(n^{-1.5}) $.
• The method is designed to be applicable in time-varying, partially observable, and long-horizon reinforcement learning settings.

Experimental results

Research questions

• RQ1 Can we reduce the variance of off-policy evaluation in long-horizon MDPs beyond the exponential dependence on H seen in standard IS?
• RQ2 Does marginalized importance sampling yield a theoretical error bound that scales polynomially with the horizon H?
• RQ3 How does MIS compare empirically to existing OPE methods in partially observable and time-varying RL environments?
• RQ4 To what extent does MIS achieve the Cramer-Rao lower bound for OPE in nonstationary episodic MDPs?
• RQ5 Can MIS maintain low variance and high accuracy in large-action-space and long-horizon settings?

Key findings

• The MIS estimator achieves a mean-squared error bound with polynomial dependence on the horizon H, a significant improvement over the exponential dependence in standard IS.
• The theoretical error bound matches the Cramer-Rao lower bound from [Jiang and Li, 2016] up to a multiplicative factor of H.
• The method is the first OPE estimator to achieve such a bound with polynomial H dependence in nonstationary episodic MDPs.
• Empirical results show MIS outperforms existing methods in time-varying, partially observable, and long-horizon RL environments.
• The estimator achieves $ \tilde{O}(n^{-1.5}) $-style convergence rate in addition to the main error term, indicating strong finite-sample performance.
• The recursive estimation of state marginal distributions effectively reduces variance propagation across time steps in long-horizon settings.