[Paper Review] From Importance Sampling to Doubly Robust Policy Gradient

Jiawei Huang, Nan Jiang|arXiv (Cornell University)|Oct 20, 2019

Optimization and Search Problems18 references4 citations

TL;DR

This paper establishes a unifying framework that derives policy gradient (PG) estimators from off-policy evaluation (OPE) techniques, particularly the doubly robust (DR) estimator. By applying finite differences to DR-based value estimators, the authors derive a general PG form that subsumes state-of-the-art variance reduction methods and achieves lower variance, approaching the Cramér-Rao lower bound in deterministic MDPs.

ABSTRACT

We show that on-policy policy gradient (PG) and its variance reduction variants can be derived by taking finite difference of function evaluations supplied by estimators from the importance sampling (IS) family for off-policy evaluation (OPE). Starting from the doubly robust (DR) estimator (Jiang & Li, 2016), we provide a simple derivation of a very general and flexible form of PG, which subsumes the state-of-the-art variance reduction technique (Cheng et al., 2019) as its special case and immediately hints at further variance reduction opportunities overlooked by existing literature. We analyze the variance of the new DR-PG estimator, compare it to existing methods as well as the Cramer-Rao lower bound of policy gradient, and empirically show its effectiveness.

Motivation & Objective

• To uncover a deep theoretical connection between on-policy policy gradient and off-policy importance sampling techniques.
• To derive a general, flexible policy gradient estimator by applying finite differences to the doubly robust (DR) OPE estimator.
• To achieve lower variance in policy gradient estimation by leveraging side information and structural properties of the DR estimator.
• To demonstrate that the proposed DR-PG estimator approaches the Cramér-Rao lower bound for policy gradient variance in deterministic MDPs.

Proposed method

• The policy gradient is derived as the finite difference of the expected return, where the return is estimated using any method from the importance sampling (IS) family.
• The doubly robust (DR) estimator from Jiang & Li (2016) is used as the foundation for constructing a new, general policy gradient estimator.
• The method incorporates both a model-based value estimate and an importance-weighted correction, enabling robustness to errors in either component.
• The resulting DR-PG estimator includes a trajectory-wise control variate and allows for gradient-dependent corrections via $ abla_{m{ heta}} ilde{Q}$ and $ ilde{G}_2$ terms.
• The framework naturally subsumes prior methods, including Cheng et al. (2019)’s trajectory-wise control variate, as special cases.
• The method is implemented with practical modifications such as $ heta = 0.9$, $ ho_{[0:t]}$, and bootstrapped value estimators $ ilde{V}, ilde{Q}$.

Experimental results

Research questions

• RQ1 Can policy gradient estimation be systematically derived from off-policy evaluation estimators, particularly the doubly robust estimator?
• RQ2 Does the DR-based policy gradient estimator achieve lower variance than existing state-of-the-art methods?
• RQ3 Can the proposed framework reveal new variance reduction opportunities overlooked by prior work?
• RQ4 How close is the variance of the DR-PG estimator to the Cramér-Rao lower bound in deterministic MDPs?
• RQ5 What role do gradient-dependent corrections and side information play in improving estimator efficiency?

Key findings

• The DR-PG estimator achieves variance that is significantly lower than baseline methods, including per-step IS and state-action-dependent baselines.
• The proposed DR-PG estimator subsumes the state-of-the-art method by Cheng et al. (2019) as a special case, confirming its generality and flexibility.
• The variance of the DR-PG estimator is shown to approach the Cramér-Rao lower bound in deterministic MDPs, indicating near-optimal efficiency.
• The inclusion of gradient terms $ abla_{m{ heta}} ilde{Q}$ and $ ilde{G}_2$ enables further variance reduction beyond existing control variate techniques.
• Empirical results confirm that the DR-PG estimator outperforms all baselines in terms of sample efficiency and variance reduction across multiple environments.
• The method demonstrates that the connection between IS and PG is not superficial, but foundational—allowing unified derivation and analysis of variance-reduced PG estimators.