[Paper Review] Taming the Monster: A Fast and Simple Algorithm for Contextual Bandits

Alekh Agarwal, Daniel Hsu+4 more|arXiv (Cornell University)|Feb 4, 2014

Advanced Bandit Algorithms Research21 references313 citations

TL;DR

This paper introduces a novel, efficient algorithm for contextual bandits that achieves optimal regret bounds with only sublinear oracle calls—specifically, $ tilde{O}(σ√{KT/\ln|Π|})$—by combining a sparse policy distribution via coordinate descent and an adaptive epoch structure. The method leverages a cost-sensitive classification oracle, enabling practical deployment with significantly reduced computational overhead compared to prior optimal-regret algorithms.

ABSTRACT

We present a new algorithm for the contextual bandit learning problem, where the learner repeatedly takes one of $K$ actions in response to the observed context, and observes the reward only for that chosen action. Our method assumes access to an oracle for solving fully supervised cost-sensitive classification problems and achieves the statistically optimal regret guarantee with only $ ilde{O}(\sqrt{KT/\log N})$ oracle calls across all $T$ rounds, where $N$ is the number of policies in the policy class we compete against. By doing so, we obtain the most practical contextual bandit learning algorithm amongst approaches that work for general policy classes. We further conduct a proof-of-concept experiment which demonstrates the excellent computational and prediction performance of (an online variant of) our algorithm relative to several baselines.

Motivation & Objective

• To develop a computationally efficient contextual bandit algorithm that achieves optimal regret bounds for general policy classes.
• To reduce the number of oracle calls required for optimal regret from polynomial in T to sublinear, specifically $ tilde{O}(√{KT/\ln|Π|})$.
• To enable practical deployment of contextual bandit algorithms by minimizing computational overhead while maintaining statistical optimality.
• To design a method that scales efficiently with large policy classes without linear dependence on $|Π|$.

Proposed method

• The algorithm uses a coordinate descent procedure to compute a very sparse distribution over policies, enabling efficient sampling.
• It introduces a novel epoch structure that updates the policy distribution infrequently, reducing computational frequency.
• The method relies on an optimization oracle that solves cost-sensitive classification problems to select high-reward policies.
• A randomized exploration strategy with a minimum exploration probability $\mu$ ensures sufficient coverage of actions without sacrificing performance.
• An online variant of the algorithm is developed, using doubly robust reward estimation and online learning of cost-sensitive oracles.
• The algorithm maintains a distribution over policies that is updated only at selected epochs, minimizing oracle calls while preserving regret guarantees.

Experimental results

Research questions

• RQ1 Can we achieve optimal regret in contextual bandits with sublinear oracle calls relative to the number of rounds T?
• RQ2 How can we maintain statistical optimality while drastically reducing computational complexity in large policy classes?
• RQ3 Can a sparse policy distribution be computed efficiently and still ensure low regret in the presence of exploration?
• RQ4 What structural design (e.g., epoch scheduling) enables both low regret and low oracle usage?
• RQ5 Is it possible to design a practical contextual bandit algorithm that scales well with large policy classes and real-world data?

Key findings

• The algorithm achieves the statistically optimal regret bound of $\ttilde{O}(\sqrt{KT/\ln(|\Pi|/\delta)})$ with high probability.
• The total number of oracle calls over $T$ rounds is $\ttilde{O}(\sqrt{KT/\ln(|\Pi|/\delta)})$, which is sublinear in $T$.
• The net running time is $\ttilde{O}(T^{1.5}\sqrt{K\log|\Pi|})$, representing a significant improvement over prior methods like Randomized UCB.
• The algorithm's computational complexity is drastically reduced compared to $\ttilde{O}(T^6)$ in Randomized UCB, making it practical for real-world deployment.
• The online variant of the algorithm demonstrates low computational cost and high reward performance in proof-of-concept experiments.
• Theoretical analysis shows that any distribution $Q$ satisfying the required constraints must have support size $\Omega(\sqrt{K\tau_M / \ln(\tau_M|\Pi|/\delta)})$, proving the necessity of sparsity.