[Paper Review] On the Performance of Thompson Sampling on Logistic Bandits

Shi Dong, Tengyu Ma+1 more|arXiv (Cornell University)|May 12, 2019

Advanced Bandit Algorithms Research16 references4 citations

TL;DR

This paper establishes a Bayesian regret bound of $\tilde{O}(d\sqrt{T})$ for Thompson sampling on logistic bandits, independent of the slope parameter $\beta$, under the condition that the action and parameter sets are identical. It further generalizes this result using the fragility dimension $\eta$ and worst-case optimal log-odds $\lambda$, showing that regret scales as $\tilde{O}(\sqrt{d\eta T}/\lambda)$, and proves that no algorithm can achieve sub-polynomial regret in $T$ without dependence on $\eta$ or $\lambda$. The analysis introduces a novel information ratio bound for logistic bandits, resolving a long-standing open question about $\beta$-dependence in prior bounds.

ABSTRACT

We study the logistic bandit, in which rewards are binary with success probability $\exp(βa^ op θ) / (1 + \exp(βa^ op θ))$ and actions $a$ and coefficients $θ$ are within the $d$-dimensional unit ball. While prior regret bounds for algorithms that address the logistic bandit exhibit exponential dependence on the slope parameter $β$, we establish a regret bound for Thompson sampling that is independent of $β$. Specifically, we establish that, when the set of feasible actions is identical to the set of possible coefficient vectors, the Bayesian regret of Thompson sampling is $ ilde{O}(d\sqrt{T})$. We also establish a $ ilde{O}(\sqrt{dηT}/λ)$ bound that applies more broadly, where $λ$ is the worst-case optimal log-odds and $η$ is the "fragility dimension," a new statistic we define to capture the degree to which an optimal action for one model fails to satisfice for others. We demonstrate that the fragility dimension plays an essential role by showing that, for any $ε> 0$, no algorithm can achieve $\mathrm{poly}(d, 1/λ)\cdot T^{1-ε}$ regret.

Motivation & Objective

• To resolve the open question of whether Thompson sampling on logistic bandits can achieve regret independent of the slope parameter $\beta$, which previous bounds showed with exponential dependence.
• To develop a new information-theoretic analysis technique for bounding the information ratio in logistic bandits, extending prior work on linear and independent bandits.
• To define and analyze the fragility dimension $\eta$, a new statistic capturing the sensitivity of optimal actions across models, and show its essential role in regret bounds.
• To establish tight regret lower bounds, proving that no algorithm can achieve $T^{1-\epsilon}$ regret without polynomial dependence on $\eta$ and $\lambda$, for any $\epsilon > 0$.

Proposed method

• Proposes a novel information ratio bound tailored to logistic bandits, leveraging the structure of the logistic link function and the geometry of the parameter and action spaces.
• Introduces the fragility dimension $\eta$ as the maximum number of models for which the optimal action in one model yields success probability $\leq 50\%$ in another.
• Uses a Bayesian regret framework and derives bounds via a careful analysis of the mutual information between actions and the true parameter $\theta$, exploiting concentration and anti-concentration properties.
• Establishes a key inequality involving the information ratio and the log-odds of action-parameter pairs, leading to a $\tilde{O}(d\sqrt{T})$ bound when $\mathcal{A} = \Theta$, the action and parameter sets are identical.
• Generalizes the bound to the case $\mathcal{A} \neq \Theta$ by introducing $\lambda$, the worst-case optimal log-odds, and shows the regret scales as $\tilde{O}(\sqrt{d\eta T}/\lambda)$.
• Employs a constructive lower bound argument using spherical codes and inner product constraints to show that $\eta$ can grow exponentially with $d$ in certain regimes, and that this growth is unavoidable for sub-polynomial regret.

Experimental results

Research questions

• RQ1 Can Thompson sampling achieve regret independent of the slope parameter $\beta$ in logistic bandits, despite prior bounds showing exponential dependence?
• RQ2 What is the role of the fragility dimension $\eta$ in determining the fundamental limits of regret in logistic bandits?
• RQ3 Is it possible to achieve $T^{1-\epsilon}$ regret for any $\epsilon > 0$ without dependence on $\eta$ or $\lambda$?
• RQ4 How does the structure of the action and parameter sets affect the difficulty of learning in logistic bandits, especially when $\mathcal{A} \neq \Theta$?
• RQ5 Can the information ratio framework be extended to non-linear bandits like logistic bandits, and what new technical challenges arise?

Key findings

• Thompson sampling achieves a Bayesian regret bound of $\tilde{O}(d\sqrt{T})$ when the action set $\mathcal{A}$ equals the parameter set $\Theta$, with no dependence on the slope parameter $\beta$.
• For the general case where $\mathcal{A} \neq \Theta$, the regret is bounded by $\tilde{O}(\sqrt{d\eta T}/\lambda)$, where $\eta$ is the fragility dimension and $\lambda$ is the worst-case optimal log-odds.
• The paper proves that no algorithm can achieve $\mathrm{poly}(d,1/\lambda) \cdot T^{1-\epsilon}$ regret for any $\epsilon > 0$, establishing $\eta$ as an essential parameter in the regret bound.
• The fragility dimension $\eta$ can grow exponentially with $d$ in certain configurations, particularly when the worst-case optimal log-odds $\iota = 0$, but is expected to scale linearly with $d$ in most practical settings.
• The analysis reveals that removing actions from the action set can increase the problem's difficulty, as demonstrated by a counterexample where $\eta$ increases when actions are removed, making learning harder despite a smaller action space.
• The information ratio bound technique developed in this work provides a new pathway for analyzing non-linear bandit problems, particularly those with exponential family rewards.