[Paper Review] Sample-efficient Nonstationary Policy Evaluation for Contextual Bandits

Miroslav Dudı́k, Dumitru Erhan+2 more|arXiv (Cornell University)|Oct 16, 2012

Advanced Bandit Algorithms Research25 references3 citations

TL;DR

This paper proposes a sample-efficient offline policy evaluation method for contextual bandits that unifies importance weighting, doubly robust estimation, and nonstationary policy evaluation. By carefully managing the bias-variance tradeoff and leveraging target policy randomness, it reduces variance and improves sample efficiency, demonstrating superior performance on synthetic and real-world datasets with up to an order of magnitude better data utilization.

ABSTRACT

We present and prove properties of a new offline policy evaluator for an exploration learning setting which is superior to previous evaluators. In particular, it simultaneously and correctly incorporates techniques from importance weighting, doubly robust evaluation, and nonstationary policy evaluation approaches. In addition, our approach allows generating longer histories by careful control of a bias-variance tradeoff, and further decreases variance by incorporating information about randomness of the target policy. Empirical evidence from synthetic and realworld exploration learning problems shows the new evaluator successfully unifies previous approaches and uses information an order of magnitude more efficiently.

Motivation & Objective

• To address the challenge of evaluating policies in contextual bandit settings with limited offline data.
• To improve sample efficiency in offline policy evaluation by combining multiple established techniques.
• To reduce variance in policy evaluation through explicit modeling of target policy randomness.
• To enable generation of longer effective data histories via controlled bias-variance tradeoff.
• To unify and outperform prior approaches in both synthetic and real-world exploration learning problems.

Proposed method

• Combines importance weighting with doubly robust estimation to improve accuracy and reduce variance in offline policy evaluation.
• Incorporates nonstationary policy evaluation techniques to handle time-varying behavior policies.
• Introduces a bias-variance tradeoff mechanism that allows the generation of longer effective data histories from limited data.
• Leverages knowledge of the target policy's stochasticity to further reduce variance in the evaluation estimator.
• Employs a unified estimator that integrates all three components—importance weighting, double robustness, and nonstationarity—into a single coherent framework.
• Uses a modified empirical risk minimization approach that accounts for both behavior policy and target policy distributions.

Experimental results

Research questions

• RQ1 Can a unified offline policy evaluation method simultaneously improve sample efficiency and reduce variance in contextual bandit settings?
• RQ2 How does incorporating target policy randomness affect the variance of policy evaluation estimators?
• RQ3 To what extent can the bias-variance tradeoff be controlled to generate longer effective data histories from limited data?
• RQ4 Does combining importance weighting, doubly robust estimation, and nonstationary evaluation lead to consistent improvements over individual methods?
• RQ5 How does the proposed method compare to prior approaches in both synthetic and real-world contextual bandit problems?

Key findings

• The proposed method achieves significantly lower variance than baseline approaches by incorporating target policy randomness.
• The method enables effective use of data up to an order of magnitude more efficiently than previous methods in both synthetic and real-world experiments.
• The unified estimator outperforms individual techniques like importance weighting and doubly robust estimation in terms of accuracy and stability.
• Controlled bias-variance tradeoff allows the generation of longer effective histories, improving evaluation reliability.
• Empirical results on real-world exploration learning problems confirm the method's superiority in sample efficiency and robustness.
• The method demonstrates consistent improvement across multiple evaluation metrics, including mean squared error and confidence interval coverage.