[Paper Review] Efficient Optimal Learning for Contextual Bandits

Miroslav Dudı́k, Daniel Hsu+5 more|arXiv (Cornell University)|Jun 13, 2011

Advanced Bandit Algorithms Research16 references119 citations

TL;DR

This paper presents the first efficient algorithm for contextual bandits that achieves optimal regret with polylogarithmic runtime in the number of policies. By reducing the problem to cost-sensitive classification and using an oracle learner, the method attains regret $O(\sqrt{TK\ln N})$ in $\mathrm{polylog}(N)$ time, exponentially faster than prior optimal regret algorithms.

ABSTRACT

We address the problem of learning in an online setting where the learner repeatedly observes features, selects among a set of actions, and receives reward for the action taken. We provide the first efficient algorithm with an optimal regret. Our algorithm uses a cost sensitive classification learner as an oracle and has a running time $\mathrm{polylog}(N)$, where $N$ is the number of classification rules among which the oracle might choose. This is exponentially faster than all previous algorithms that achieve optimal regret in this setting. Our formulation also enables us to create an algorithm with regret that is additive rather than multiplicative in feedback delay as in all previous work.

Motivation & Objective

• Address the computational bottleneck in contextual bandit learning, where previous optimal regret algorithms required linear time in the number of policies.
• Enable efficient learning in large policy spaces by leveraging cost-sensitive classification oracles.
• Achieve optimal regret scaling while maintaining computational efficiency, overcoming the exponential runtime of prior methods.
• Provide a framework that transforms any cost-sensitive classification learner into an optimal contextual bandit algorithm.
• Eliminate the dependence on multiplicative feedback delay in regret bounds, achieving additive dependence instead.

Proposed method

• Reduce the contextual bandit problem to a sequence of cost-sensitive classification problems using a novel reduction technique.
• Use a cost-sensitive classification oracle to select policies at each round, avoiding explicit maintenance of a measure over all policies.
• Apply the ellipsoid method to solve a relaxed convex program that ensures regret optimality, with constraints on policy weights and expected rewards.
• Construct separating hyperplanes via convex function evaluation to guide the ellipsoid algorithm toward feasible solutions.
• Round the final solution to a discrete distribution over policies using a perceptron-based rounding procedure with bounded error.
• Ensure polylogarithmic runtime by limiting the number of ellipsoid iterations and oracle calls through careful parameterization and concentration bounds.

Experimental results

Research questions

• RQ1 Can we achieve optimal regret in contextual bandits with computational efficiency that scales polylogarithmically in the number of policies?
• RQ2 Is it possible to eliminate the multiplicative dependence on feedback delay in regret bounds while maintaining optimality?
• RQ3 Can we reduce the contextual bandit problem to cost-sensitive classification without sacrificing regret guarantees?
• RQ4 How can we efficiently search large policy spaces using only oracle access to classification learners?
• RQ5 What is the minimal computational overhead required to achieve optimal regret in the i.i.d. contextual bandit setting?

Key findings

• The proposed algorithm achieves optimal regret of $O(\sqrt{TK\ln N})$ with $\mathrm{polylog}(N)$ runtime, where $N$ is the number of policies.
• The algorithm runs in time $O(t^5 K^4 \log^2(tK/\delta))$ for $t$ time steps, which is exponentially faster than previous optimal regret algorithms.
• The regret bound is additive in feedback delay, unlike prior work that had multiplicative dependence, improving robustness to delayed feedback.
• The method uses only a cost-sensitive classification oracle, making it modular and extensible to future improvements in classification learning.
• The ellipsoid method successfully solves the relaxed convex program in polylogarithmic time, with provable feasibility and optimality guarantees.
• The rounding procedure ensures that the final policy distribution is close to the optimal solution, with $\|W_P - W\| \leq 2\delta$.