[Paper Review] Confident Off-Policy Evaluation and Selection through Self-Normalized Importance Weighting

Ilja Kuzborskij, Claire Vernade+2 more|arXiv (Cornell University)|Jun 18, 2020

Advanced Causal Inference Techniques29 references10 citations

TL;DR

This paper proposes a novel high-probability lower bound for off-policy evaluation in contextual bandits using self-normalized importance weighting (SN), combining a semi-empirical Efron-Stein tail inequality with multiplicative bias control. The method achieves tighter confidence intervals and superior policy selection performance compared to state-of-the-art baselines on synthetic and real-world datasets.

ABSTRACT

We consider off-policy evaluation in the contextual bandit setting for the purpose of obtaining a robust off-policy selection strategy, where the selection strategy is evaluated based on the value of the chosen policy in a set of proposal (target) policies. We propose a new method to compute a lower bound on the value of an arbitrary target policy given some logged data in contextual bandits for a desired coverage. The lower bound is built around the so-called Self-normalized Importance Weighting (SN) estimator. It combines the use of a semi-empirical Efron-Stein tail inequality to control the concentration and a new multiplicative (rather than additive) control of the bias. The new approach is evaluated on a number of synthetic and real datasets and is found to be superior to its main competitors, both in terms of tightness of the confidence intervals and the quality of the policies chosen.

Motivation & Objective

• Address the challenge of reliable off-policy evaluation and selection in contextual bandits with limited, logged data.
• Develop a high-probability lower bound on the value of a target policy that remains tight even under heavy-tailed importance weights.
• Improve policy selection robustness by providing confidence bounds that control both bias and variance in off-policy estimation.
• Overcome limitations of standard importance weighting and inverse propensity scoring in misaligned behavior and target policy scenarios.
• Enable practical deployment of off-policy selection in real-world applications such as personalized medicine and online advertising.

Proposed method

• Leverages self-normalized importance weighting (SN) as a core estimator to stabilize variance and ensure all moments are bounded.
• Introduces a semi-empirical Efron-Stein inequality to control the tail concentration of the SN estimator.
• Applies a multiplicative bias correction instead of additive bias control, improving finite-sample performance.
• Derives a finite-sample, high-probability lower bound on the value of a target policy using SN estimation.
• Uses empirical likelihood and moment-generating function techniques to calibrate the confidence bounds.
• Employs Gibbs-fitting to tune hyperparameters and improve robustness across diverse datasets.

Experimental results

Research questions

• RQ1 Can a high-probability lower bound be constructed for off-policy evaluation that remains tight under heavy-tailed importance weights?
• RQ2 How does multiplicative bias control in SN estimation compare to additive bias correction in terms of finite-sample performance?
• RQ3 Can the proposed method outperform existing off-policy evaluation baselines in policy selection accuracy and confidence interval tightness?
• RQ4 How effective is the semi-empirical Efron-Stein inequality in controlling concentration for SN estimators in practice?
• RQ5 Does the method maintain robustness across diverse real-world and synthetic datasets with varying policy misalignment?

Key findings

• The proposed method achieves the tightest confidence intervals among all evaluated baselines across all datasets, with the ESLB (Efron-Stein Lower Bound) for SN consistently outperforming alternatives.
• On the Letter dataset, the ESLB for Gibbs-fitted-SN achieved a coverage of 0.997 ± 0.004, significantly outperforming the ideal bound (0.903 ± 0.013) in terms of empirical coverage quality.
• For the kropt dataset, the Cheb- SN estimator with Gibbs-fitted-SN achieved a value of 0.840 ± 0.053, demonstrating strong performance in high-dimensional settings.
• The method outperforms all competitors in policy selection, with the lowest mean squared error in value estimation across all target policies.
• The use of multiplicative bias control leads to more stable and accurate bounds compared to additive corrections, especially in high-variance regimes.
• Empirical results show that SN-based estimators with ESLB achieve near-ideal coverage (e.g., 0.999 ± 0.002 on OptDigits) even when behavior and target policies are misaligned.