[Paper Review] Doubly Robust Policy Evaluation and Learning

Miroslav Dudı́k, John Langford+1 more|arXiv (Cornell University)|Mar 23, 2011

Advanced Bandit Algorithms Research13 references302 citations

TL;DR

This paper proposes a doubly robust (DR) approach for policy evaluation and learning in contextual bandit settings, combining reward modeling and inverse propensity scoring to achieve unbiased estimates when either model is accurate. The method reduces variance and improves accuracy over existing techniques, with empirical results showing consistent 13.6% average RMSE reduction in value estimation and superior policy optimization.

ABSTRACT

We study decision making in environments where the reward is only partially observed, but can be modeled as a function of an action and an observed context. This setting, known as contextual bandits, encompasses a wide variety of applications including health-care policy and Internet advertising. A central task is evaluation of a new policy given historic data consisting of contexts, actions and received rewards. The key challenge is that the past data typically does not faithfully represent proportions of actions taken by a new policy. Previous approaches rely either on models of rewards or models of the past policy. The former are plagued by a large bias whereas the latter have a large variance. In this work, we leverage the strength and overcome the weaknesses of the two approaches by applying the doubly robust technique to the problems of policy evaluation and optimization. We prove that this approach yields accurate value estimates when we have either a good (but not necessarily consistent) model of rewards or a good (but not necessarily consistent) model of past policy. Extensive empirical comparison demonstrates that the doubly robust approach uniformly improves over existing techniques, achieving both lower variance in value estimation and better policies. As such, we expect the doubly robust approach to become common practice.

Motivation & Objective

• Address the challenge of accurate policy evaluation in contextual bandits where historical data does not reflect the new policy's action distribution.
• Overcome the limitations of direct method (high bias if reward model is poor) and inverse propensity scoring (high variance if behavior policy model is poor).
• Develop a unified framework that ensures unbiased estimation when either the reward model or the behavior policy model is accurate.
• Demonstrate that the doubly robust approach uniformly improves estimation accuracy and policy optimization over existing methods.

Proposed method

• Apply the doubly robust estimation technique to contextual bandit policy evaluation, combining a reward model and a behavior policy model.
• Use a weighted combination of inverse propensity scores and reward model predictions to form an unbiased estimator when either component is correct.
• Formulate the doubly robust estimator as: $\hat{V}_{\text{DR}} = \sum_i \frac{\mathbf{1}(a_i = a) \cdot r_i}{\hat{e}(a|x_i)} + \hat{\varrho}(x_i) \cdot \left(1 - \frac{\mathbf{1}(a_i = a)}{\hat{e}(a|x_i)} \right)$, where $\hat{e}$ is the estimated behavior policy and $\hat{\varrho}$ the reward model.
• Optimize policies using direct loss minimization with gradient updates on policy weights, incorporating DR estimates to guide learning.
• Use ridge regression to train the reward model $\hat{\varrho}(x)$, and estimate behavior policy probabilities $\hat{e}(a|x)$ via logistic regression or similar.
• Evaluate performance on both synthetic benchmarks and a large-scale real-world dataset from Yahoo! News, comparing DR to IPS and direct method.

Experimental results

Research questions

• RQ1 Can a doubly robust estimator improve policy value estimation in contextual bandits when either the reward model or the behavior policy model is inaccurate?
• RQ2 How does the doubly robust method compare to inverse propensity scoring and direct methods in terms of bias, variance, and estimation accuracy?
• RQ3 Does the use of a doubly robust estimator lead to better policy optimization in practice?
• RQ4 What is the impact of model quality on the performance of the doubly robust estimator in non-asymptotic settings?

Key findings

• The doubly robust estimator consistently reduces estimation error compared to inverse propensity scoring, with an average RMSE reduction of 13.6% across experiments.
• The DR estimator achieves lower variance than IPS, especially with smaller dataset sizes, accelerating convergence to the true policy value.
• The method maintains low bias even when one of the models (reward or behavior policy) is misspecified, demonstrating robustness.
• Empirical results show that DR-based policy learning outperforms both IPS and direct method in policy optimization, yielding better-performing policies.
• In real-world experiments on a large-scale Yahoo! News dataset, DR achieved significant improvements in value estimation accuracy, particularly in low-data regimes.
• The theoretical analysis shows that bias and variance of the DR estimator depend on deviations of both models from the truth, providing a principled understanding of its performance.