[Paper Review] Triply Robust Off-Policy Evaluation

Anqi Liu, Hao Liu+2 more|arXiv (Cornell University)|Nov 13, 2019

Advanced Causal Inference Techniques6 references4 citations

TL;DR

This paper proposes a triply robust off-policy evaluation method for contextual bandits by framing off-policy evaluation as a covariate shift problem and applying deep robust regression to improve the direct method component. The approach enhances both direct methods and doubly robust estimators, achieving superior performance—especially when the logging policy is unknown—through improved bias and variance control, with up to 50% relative error reduction in challenging settings.

ABSTRACT

We propose a robust regression approach to off-policy evaluation (OPE) for contextual bandits. We frame OPE as a covariate-shift problem and leverage modern robust regression tools. Ours is a general approach that can be used to augment any existing OPE method that utilizes the direct method. When augmenting doubly robust methods, we call the resulting method Triply Robust. We prove upper bounds on the resulting bias and variance, as well as derive novel minimax bounds based on robust minimax analysis for covariate shift. Our robust regression method is compatible with deep learning, and is thus applicable to complex OPE settings that require powerful function approximators. Finally, we demonstrate superior empirical performance across the standard OPE benchmarks, especially in the case where the logging policy is unknown and must be estimated from data.

Motivation & Objective

• To address the challenge of off-policy evaluation in contextual bandits where the logging policy is often unknown or poorly estimated.
• To improve the bias and variance of direct methods (DM) used in off-policy evaluation by leveraging robust regression under covariate shift.
• To extend robustness to doubly robust methods by integrating robust direct methods, resulting in a triply robust estimator.
• To provide theoretical guarantees on bias, variance, and minimax risk for the proposed framework.
• To demonstrate empirical superiority across standard benchmarks, especially in high-variance or unknown logging policy scenarios.

Proposed method

• Formulates off-policy evaluation as a covariate shift problem, where the target policy's context-action distribution differs from the logging policy's distribution.
• Applies modern robust regression techniques to the direct method component, explicitly modeling the shift between logging and target policy distributions.
• Introduces a triply robust (TR) estimator by combining robust direct methods with inverse propensity scoring in a doubly robust framework.
• Uses deep neural networks as function approximators to enable application in complex, high-dimensional settings.
• Derives novel minimax bounds based on worst-case covariate shift to analyze the robustness of the proposed estimators.
• Employs a general framework that can augment existing OPE methods by replacing their standard direct method with a robust regression-based alternative.

Experimental results

Research questions

• RQ1 Can robust regression techniques designed for covariate shift be effectively applied to improve off-policy evaluation in contextual bandits?
• RQ2 How does integrating robust regression into the direct method component affect the bias and variance of off-policy estimators?
• RQ3 What are the theoretical guarantees—particularly in terms of bias, variance, and minimax risk—of the resulting triply robust estimator?
• RQ4 How does the proposed method perform empirically when the logging policy is unknown or must be estimated from data?
• RQ5 Can the framework be extended to improve existing doubly robust and other state-of-the-art OPE methods?

Key findings

• The triply robust estimator consistently outperforms standard doubly robust methods, especially when the logging policy is unknown or estimated from data, achieving up to 50% relative error reduction.
• In settings with high-variance logging policies, the robust direct method (DM-R) significantly reduces error compared to standard DM, and this benefit transfers to the TR estimator.
• When the logging policy is known and uniform, TR achieves lower variance than DR, demonstrating improved stability.
• The DM-R and TR families of methods outperform their non-robust counterparts across all benchmark environments, including Vehicle and CIFAR10.
• The robust regression framework improves performance even when the logging policy is estimated via a classification model, highlighting its practical robustness.
• Theoretical analysis confirms improved bias and variance bounds for the proposed estimators, with novel minimax bounds derived under worst-case covariate shift.