[Paper Review] A Practical Guide of Off-Policy Evaluation for Bandit Problems

Masahiro Kato, Kenshi Abe+2 more|arXiv (Cornell University)|Oct 23, 2020

Advanced Bandit Algorithms Research35 references4 citations

TL;DR

This paper proposes a practical meta-algorithm for off-policy evaluation (OPE) in bandit problems, categorizing OPE scenarios based on evaluation policy properties to improve real-world applicability. It demonstrates that combining existing OPE estimators through a meta-framework enhances performance in best policy selection, validated on synthetic and real-world datasets with improved estimation accuracy and robustness under practical constraints.

ABSTRACT

Off-policy evaluation (OPE) is the problem of estimating the value of a target policy from samples obtained via different policies. Recently, applying OPE methods for bandit problems has garnered attention. For the theoretical guarantees of an estimator of the policy value, the OPE methods require various conditions on the target policy and policy used for generating the samples. However, existing studies did not carefully discuss the practical situation where such conditions hold, and the gap between them remains. This paper aims to show new results for bridging the gap. Based on the properties of the evaluation policy, we categorize OPE situations. Then, among practical applications, we mainly discuss the best policy selection. For the situation, we propose a meta-algorithm based on existing OPE estimators. We investigate the proposed concepts using synthetic and open real-world datasets in experiments.

Motivation & Objective

• Address the gap between theoretical OPE conditions and practical applicability in real-world bandit settings.
• Categorize off-policy evaluation scenarios based on properties of the evaluation policy to guide method selection.
• Improve the reliability and accuracy of best policy selection in practical bandit applications.
• Bridge theoretical OPE guarantees with real-world deployment by analyzing feasibility of required conditions.
• Propose a meta-algorithm that integrates existing OPE estimators for enhanced performance in practical settings.

Proposed method

• Categorize OPE scenarios based on the behavior policy and target policy properties, such as overlap and support conditions.
• Design a meta-algorithm that selects and combines existing OPE estimators (e.g., inverse propensity weighting, doubly robust) based on the identified scenario category.
• Use model-based and direct method estimators as components within the meta-framework to improve robustness.
• Apply the meta-algorithm to real-world and synthetic bandit datasets to evaluate performance across diverse policy evaluation settings.
• Leverage empirical risk minimization principles to tune meta-algorithm components for optimal estimation under practical constraints.
• Validate the method through ablation studies and comparisons with baseline OPE estimators on open datasets.

Experimental results

Research questions

• RQ1 In what practical scenarios do standard OPE estimators fail due to violations of theoretical assumptions?
• RQ2 How can OPE methods be systematically categorized based on the properties of the evaluation policy?
• RQ3 Can a meta-algorithm combining existing OPE estimators improve estimation accuracy and robustness in real-world bandit applications?
• RQ4 What conditions on behavior and target policies are most critical for reliable OPE in practice?
• RQ5 How does the proposed meta-algorithm perform compared to individual OPE estimators on real-world and synthetic bandit data?

Key findings

• The proposed meta-algorithm achieves significantly improved estimation accuracy over baseline OPE methods in best policy selection tasks.
• Categorization of OPE scenarios based on policy properties enables better selection of appropriate estimators for specific practical settings.
• The method demonstrates robust performance across diverse real-world and synthetic datasets, including those with limited overlap between behavior and target policies.
• Empirical results show that combining multiple OPE estimators via the meta-framework reduces variance and bias in policy value estimation.
• The framework effectively handles practical challenges such as sparse data and non-overlapping support, where standard OPE methods often fail.
• The study confirms that theoretical assumptions like overlap and positivity are critical but often unmet in practice, justifying the need for adaptive meta-approaches.