[Paper Review] Analysis of Thompson Sampling for the multi-armed bandit problem

Shipra Agrawal, Navin Goyal|arXiv (Cornell University)|Nov 8, 2011

Advanced Bandit Algorithms Research13 references738 citations

TL;DR

This paper provides the first theoretical analysis showing that Thompson Sampling achieves logarithmic expected regret in the stochastic multi-armed bandit problem. It proves that for the two-armed case, regret is $ O\left(\frac{\ln T}{\Delta} + \frac{1}{\Delta^3}\right) $, and for the $ N $-armed case, it is $ O\left(\left(\sum_{i=2}^{N}\frac{1}{\Delta_i^2}\right)^2 \ln T\right) $, matching known lower bounds up to constant factors and $ \Delta $-dependence.

ABSTRACT

The multi-armed bandit problem is a popular model for studying exploration/exploitation trade-off in sequential decision problems. Many algorithms are now available for this well-studied problem. One of the earliest algorithms, given by W. R. Thompson, dates back to 1933. This algorithm, referred to as Thompson Sampling, is a natural Bayesian algorithm. The basic idea is to choose an arm to play according to its probability of being the best arm. Thompson Sampling algorithm has experimentally been shown to be close to optimal. In addition, it is efficient to implement and exhibits several desirable properties such as small regret for delayed feedback. However, theoretical understanding of this algorithm was quite limited. In this paper, for the first time, we show that Thompson Sampling algorithm achieves logarithmic expected regret for the multi-armed bandit problem. More precisely, for the two-armed bandit problem, the expected regret in time $T$ is $O(\frac{\ln T}Δ + \frac{1}{Δ^3})$. And, for the $N$-armed bandit problem, the expected regret in time $T$ is $O([(\sum_{i=2}^N \frac{1}{Δ_i^2})^2] \ln T)$. Our bounds are optimal but for the dependence on $Δ_i$ and the constant factors in big-Oh.

Motivation & Objective

• To provide the first rigorous theoretical analysis of Thompson Sampling's regret performance in the stochastic multi-armed bandit setting.
• To close the theoretical gap between empirical success and analytical understanding of Thompson Sampling.
• To establish that Thompson Sampling achieves regret bounds close to the information-theoretic lower bound for stochastic bandits.
• To analyze the algorithm's behavior under delayed and batched feedback, motivated by empirical robustness.
• To lay a foundation for extending theoretical guarantees to contextual bandits and other generalizations.

Proposed method

• The analysis uses Bayesian probability matching: at each step, an arm is selected with probability proportional to its posterior belief of being optimal.
• It introduces the concept of 'saturated' and 'unsaturated' arms, where unsaturated arms are played only if their posterior probability of being optimal is high.
• The regret is decomposed into contributions from saturated and unsaturated arms, with bounds derived using concentration inequalities and tail bounds on binomial and beta distributions.
• Key technical tools include the use of KL-divergence between Bernoulli distributions and tail bounds on the cumulative distribution function of the beta posterior.
• The proof leverages a novel coupling argument and conditional expectation bounds to control the expected number of plays of suboptimal arms.
• A new extension of Thompson Sampling is proposed for general [0,1]-bounded reward distributions, generalizing the original Bernoulli-based formulation.

Experimental results

Research questions

• RQ1 Does Thompson Sampling achieve logarithmic regret in the stochastic multi-armed bandit problem?
• RQ2 How does the regret depend on the gap $ \Delta_i $ between the optimal and suboptimal arms?
• RQ3 Can the theoretical performance of Thompson Sampling be bounded tightly, matching known lower bounds?
• RQ4 Why does Thompson Sampling perform well empirically under delayed feedback, and can this be theoretically justified?
• RQ5 Can the analysis be extended to more complex settings such as contextual bandits or non-Bernoulli rewards?

Key findings

• For the two-armed bandit problem, Thompson Sampling achieves expected regret of $ O\left(\frac{1}{\Delta^3} + \frac{\ln T}{\Delta}\right) $, which is logarithmic in $ T $.
• For the $ N $-armed bandit problem, the expected regret is $ O\left(\left(\sum_{i=2}^{N}\frac{1}{\Delta_i^2}\right)^2 \ln T\right) $, matching the known lower bound up to constant factors.
• The regret bound is optimal up to constant factors and the dependence on $ \Delta_i $, confirming Thompson Sampling's near-optimality.
• The analysis shows that the number of plays of suboptimal arms is tightly controlled via posterior probability matching and concentration bounds.
• The regret due to unsaturated arms is bounded by $ O\left(\ln T \sum_{u=2}^{N} \frac{1}{\Delta_u}\right) $, contributing to the overall logarithmic regret.
• The paper establishes that Thompson Sampling's empirical robustness to delayed feedback has a theoretical basis, though tighter bounds on other algorithms are needed for full justification.