[Paper Review] On the Prior Sensitivity of Thompson Sampling

Che-Yu Liu, Lihong Li|arXiv (Cornell University)|Jun 10, 2015

Advanced Bandit Algorithms Research25 references6 citations

TL;DR

This paper provides a rigorous theoretical analysis of Thompson Sampling's sensitivity to prior distributions in stochastic multi-armed bandit problems. It establishes tight $O(\sqrt{T/p})$ and $O(\sqrt{(1-p)T})$ regret bounds for poor- and good-prior cases, respectively, where $p$ is the prior probability of the true model, using martingale theory and novel structural properties of Thompson Sampling.

ABSTRACT

The empirically successful Thompson Sampling algorithm for stochastic bandits has drawn much interest in understanding its theoretical properties. One important benefit of the algorithm is that it allows domain knowledge to be conveniently encoded as a prior distribution to balance exploration and exploitation more effectively. While it is generally believed that the algorithm's regret is low (high) when the prior is good (bad), little is known about the exact dependence. In this paper, we fully characterize the algorithm's worst-case dependence of regret on the choice of prior, focusing on a special yet representative case. These results also provide insights into the general sensitivity of the algorithm to the choice of priors. In particular, with $p$ being the prior probability mass of the true reward-generating model, we prove $O(\sqrt{T/p})$ and $O(\sqrt{(1-p)T})$ regret upper bounds for the bad- and good-prior cases, respectively, as well as \emph{matching} lower bounds. Our proofs rely on the discovery of a fundamental property of Thompson Sampling and make heavy use of martingale theory, both of which appear novel in the literature, to the best of our knowledge.

Motivation & Objective

• To understand the theoretical dependence of Thompson Sampling's regret on the choice of prior distribution.
• To characterize the worst-case regret when the prior is either poorly informed (low $p$) or well-informed (high $p$) about the true model.
• To establish matching upper and lower bounds on regret to quantify sensitivity to prior quality.
• To identify fundamental structural properties of Thompson Sampling that enable the analysis, particularly via martingale methods.

Proposed method

• Analyzes a two-model, two-action bandit setting as a representative case to study prior sensitivity.
• Uses martingale theory to derive bounds on posterior concentration and action selection probabilities.
• Applies Bayesian updating rules recursively to model posterior distributions over models based on observed rewards.
• Derives recurrence inequalities for regret by comparing expected regret under different prior masses.
• Employs a coupling argument between models to bound the difference in regret under varying priors.
• Validates theoretical findings with empirical experiments on Bernoulli bandits with varying prior masses $p$.

Experimental results

Research questions

• RQ1 How does the regret of Thompson Sampling scale with the prior probability $p$ of the true model?
• RQ2 What are the tight upper and lower bounds on regret when the prior is either good ($p \approx 1$) or bad ($p \approx 0$)?
• RQ3 Can the dependence of regret on prior quality be fully characterized in a representative bandit setting?
• RQ4 What fundamental structural properties of Thompson Sampling enable such a precise regret analysis?

Key findings

• For a poor prior with prior mass $p$ on the true model, the regret is bounded above by $O(\sqrt{T/p})$, and this bound is tight.
• For a good prior with prior mass $p$ on the true model, the regret is bounded above by $O(\sqrt{(1-p)T})$, and this bound is also tight.
• The paper establishes matching lower bounds, confirming that the regret scales as $\Theta(\sqrt{T/p})$ and $\Theta(\sqrt{(1-p)T})$ for poor- and good-prior cases, respectively.
• The analysis reveals that the regret is highly sensitive to prior quality, with regret increasing as $\sqrt{1/p}$ when the prior is weak and as $\sqrt{1-p}$ when the prior is strong.
• Empirical results confirm the predicted linear scaling of regret with $\sqrt{1/p}$ and $\sqrt{1-p}$, supporting the tightness of the theoretical bounds.
• The use of martingale theory and novel structural insights into Thompson Sampling’s behavior provide tools potentially applicable to broader analysis of the algorithm.