[Paper Review] Provably Optimal Algorithms for Generalized Linear Contextual Bandits

Lihong Li, Yun Gang Lu+1 more|arXiv (Cornell University)|Feb 28, 2017

Advanced Bandit Algorithms Research141 citations

TL;DR

The paper introduces GLM-based UCB algorithms for contextual bandits with generalized linear rewards, achieving near-optimal regret and providing a sharp finite-sample MLE confidence bound.

ABSTRACT

Contextual bandits are widely used in Internet services from news recommendation to advertising, and to Web search. Generalized linear models (logistical regression in particular) have demonstrated stronger performance than linear models in many applications where rewards are binary. However, most theoretical analyses on contextual bandits so far are on linear bandits. In this work, we propose an upper confidence bound based algorithm for generalized linear contextual bandits, which achieves an $ ilde{O}(\sqrt{dT})$ regret over $T$ rounds with $d$ dimensional feature vectors. This regret matches the minimax lower bound, up to logarithmic terms, and improves on the best previous result by a $\sqrt{d}$ factor, assuming the number of arms is fixed. A key component in our analysis is to establish a new, sharp finite-sample confidence bound for maximum-likelihood estimates in generalized linear models, which may be of independent interest. We also analyze a simpler upper confidence bound algorithm, which is useful in practice, and prove it to have optimal regret for certain cases.

Motivation & Objective

• Motivate contextual bandits with generalized linear rewards beyond linear models.
• Develop provable algorithms with strong regret guarantees in GLM contexts.
• Provide a new finite-sample confidence bound for GLM maximum likelihood estimates.
• Offer a practically useful UCB-based algorithm and an optimality-focused variant.

Proposed method

• Propose SupCB-GLM, an upper confidence bound algorithm for GLM contextual bandits that achieves ~O(sqrt(d T)) regret.
• Derive a sharp finite-sample, direction-wise confidence bound for GLM MLE (non-asymptotic normality-type result).
• Analyze a simpler UCB-GLM algorithm with optimal regret under reasonable conditions.
• Introduce SupCB-GLM as an instance of creating independent samples to enable tighter analysis (inspired by Auer 2002).
• Provide proofs that leverage self-normalized martingales and a novel confidence bound to bound regret.
• Discuss computational and practical considerations, including a more efficient variant without projection steps.

Experimental results

Research questions

• RQ1 Can we design an upper confidence bound algorithm for generalized linear contextual bandits that matches minimax regret (up to logarithmic factors) in the fixed-arm regime?
• RQ2 What finite-sample, direction-wise confidence bounds can be established for GLM maximum likelihood estimators to support tight regret analysis?
• RQ3 How do GLM-based UCB algorithms compare to existing GLM bandit approaches in terms of regret, particularly in high-dimensional settings?
• RQ4 Can a practical, simpler UCB variant achieve near-optimal regret without heavy computation like projection steps?
• RQ5 Is it possible to construct independent-sample-based methods (SupCB-GLM) that attain near-optimal regret for small action sets?

Key findings

• The GLM UCB algorithm SupCB-GLM attains regret ~O(d sqrt(T)) up to logarithmic factors, matching the minimax lower bound for infinite actions (and improving over prior GLM results).
• A sharp finite-sample confidence bound is proven for GLM MLE that holds in every direction, enabling direction-specific regret analysis.
• A simpler UCB-GLM algorithm is shown to achieve optimal regret under reasonable assumptions and is more practical than projection-based methods.
• SupCB-GLM achieves near-optimal regret ~O(sqrt(d T log K)) for finite action sets, with overall bound showing tightness up to logs.
• The analysis highlights the advantage of direction-wise confidence bounds over traditional ellipsoidal (l2) bounds in GLM bandits.