[Paper Review] Sequential Batch Learning in Finite-Action Linear Contextual Bandits

Yanjun Han, Zhengqing Zhou+4 more|arXiv (Cornell University)|Apr 14, 2020

Advanced Bandit Algorithms ResearchDecision Sciences55 references31 citations

TL;DR

The paper analyzes sequential batch learning in finite-action linear contextual bandits, deriving upper and lower regret bounds under adversarial and stochastic contexts and proposing corresponding algorithms.

ABSTRACT

We study the sequential batch learning problem in linear contextual bandits with finite action sets, where the decision maker is constrained to split incoming individuals into (at most) a fixed number of batches and can only observe outcomes for the individuals within a batch at the batch's end. Compared to both standard online contextual bandits learning or offline policy learning in contexutal bandits, this sequential batch learning problem provides a finer-grained formulation of many personalized sequential decision making problems in practical applications, including medical treatment in clinical trials, product recommendation in e-commerce and adaptive experiment design in crowdsourcing. We study two settings of the problem: one where the contexts are arbitrarily generated and the other where the contexts are extit{iid} drawn from some distribution. In each setting, we establish a regret lower bound and provide an algorithm, whose regret upper bound nearly matches the lower bound. As an important insight revealed therefrom, in the former setting, we show that the number of batches required to achieve the fully online performance is polynomial in the time horizon, while for the latter setting, a pure-exploitation algorithm with a judicious batch partition scheme achieves the fully online performance even when the number of batches is less than logarithmic in the time horizon. Together, our results provide a near-complete characterization of sequential decision making in linear contextual bandits when batch constraints are present.

Motivation & Objective

• Motivate and formalize sequential batch learning where only batch-end rewards are observed.
• Characterize how a fixed number of batches M affects regret in linear contextual bandits with finite actions.
• Develop algorithms and prove regret upper and lower bounds for both adversarial and stochastic context settings.

Proposed method

• Formulate sequential batch learning with a grid of M batches and batched policy, extending online contextual bandits to batch-constrained feedback.
• Propose a sequential batch UCB (SBUCB) algorithm that updates theta estimates at batch ends and uses upper confidence bounds within each batch.
• Provide a master algorithm to handle dependence issues and establish feasibility of confidence bounds.
• Derive upper and lower regret bounds for adversarial contexts showing polylog factors in T and dependence on M.
• Analyze stochastic contexts with a pure-exploitation algorithm and derive corresponding regret bounds.
• Present problem-dependent regret bounds and discuss high-probability guarantees.

Experimental results

Research questions

• RQ1 How does restricting feedback to M batches affect regret in finite-action linear contextual bandits under adversarial contexts?
• RQ2 What is the near-optimal regret rate achievable with a sequential batch UCB algorithm in the adversarial-context setting?
• RQ3 How do stochastic contexts change the optimal batching strategy and achievable regret?
• RQ4 What are the tight lower bounds that show the necessity of a certain number of batches for optimal regret?
• RQ5 How can pure-exploitation strategies perform under stochastic contexts and what are their regret characteristics?

Key findings

• In the adversarial-context setting, there exists a sequential batch algorithm achieving polylog(T) times (sqrt(dT) + dT/M) expected regret.
• A lower bound shows regret at least c*(sqrt(dT) + min{T sqrt(d)/M, T/ sqrt(M)}) for K=2, matching the upper bound up to polylog and constant factors.
• This implies Theta(sqrt(dT)) batches suffice to attain fully online regret, and O(sqrt(Td)) batches are enough in low-dimension.
• In the stochastic-context setting, a pure-exploitation algorithm can achieve the minimax regret tilde Theta(sqrt(dT)) with as few as polylog(T) batches, specifically near log log(T/d^2).
• For stochastic contexts, the upper and lower bounds are within polylog factors, indicating near-minimax optimality with significantly fewer batches than in the adversarial case.