[Paper Review] Feedback graph regret bounds for Thompson Sampling and UCB

Thodoris Lykouris, Éva Tardos+1 more|arXiv (Cornell University)|May 23, 2019

Advanced Bandit Algorithms Research4 citations

TL;DR

This paper establishes that Thompson Sampling and Upper Confidence Bound (UCB) achieve regret bounds in stochastic multi-armed bandits with graph-based feedback by combining the graph structure and arm gap parameters, despite not explicitly using the graph for exploration. The key result is an expected regret bound of $\mathcal{O}\left(\max_{I\in\mathcal{I}(G)}\sum_{a\in I}\frac{\ln T \ln(kT)}{\Delta(a)}\right)$, where $I$ is an independent set of the feedback graph $G$, and $\Delta(a)$ is the gap of arm $a$. The analysis reveals that these classical algorithms naturally leverage partial feedback without modification.

ABSTRACT

We study the stochastic multi-armed bandit problem with the graph-based feedback structure introduced by Mannor and Shamir. We analyze the performance of the two most prominent stochastic bandit algorithms, Thompson Sampling and Upper Confidence Bound (UCB), in the graph-based feedback setting. We show that these algorithms achieve regret guarantees that combine the graph structure and the gaps between the means of the arm distributions. Surprisingly this holds despite the fact that these algorithms do not explicitly use the graph structure to select arms; they observe the additional feedback but do not explore based on it. Towards this result we introduce a "layering technique" highlighting the commonalities in the two algorithms.

Motivation & Objective

• To analyze the performance of Thompson Sampling and UCB in the graph-based feedback bandit setting introduced by Mannor and Shamir.
• To determine whether these classical algorithms can achieve improved regret by leveraging partial feedback from neighboring arms.
• To establish regret bounds that depend on both the graph structure (via independent sets) and the gaps between arm means.
• To show that these algorithms achieve enhanced performance without modifying their selection rules, solely by incorporating observed feedback.

Proposed method

• Introduce a layering technique to unify the analysis of UCB and Thompson Sampling, highlighting shared structural properties.
• Define a partitioning of arms based on gap levels $\phi$, where $\mathcal{V}_\phi$ contains arms with gaps in $[2^{-\phi-1}, 2^{-\phi})$, to analyze regret in phases.
• Use concentration inequalities and tail bounds (e.g., from Agrawal and Goyal [AG13]) to control the probability that sampled means deviate from true means.
• Bound the number of times suboptimal arms are selected while their empirical means are far from their true means, using independence and gap-based arguments.
• Apply a saturation threshold $L_a$ to distinguish between early (unsaturated) and late (saturated) phases of arm selection.
• Leverage the independence number $\alpha$ of the feedback graph $G$ to derive worst-case bounds over all independent sets $I \in \mathcal{I}(G)$.

Experimental results

Research questions

• RQ1 Can Thompson Sampling and UCB achieve regret bounds that combine the graph structure and arm gap parameters in the feedback bandit model?
• RQ2 Does the absence of explicit graph-based exploration in UCB and Thompson Sampling prevent them from benefiting from partial feedback?
• RQ3 How does the regret of these algorithms scale with the independence number of the feedback graph and the minimum gap $\Delta(a)$?
• RQ4 Is the extra logarithmic factor in the regret bound inherent to the algorithms or a limitation of the analysis?

Key findings

• The expected regret of Thompson Sampling is bounded by $\mathcal{O}\left(\max_{I\in\mathcal{I}(G)}\sum_{a\in I}\frac{\ln T \ln(kT)}{\Delta(a)}\right)$, where $\mathcal{I}(G)$ is the set of all independent sets of the feedback graph $G$.
• The regret bound scales with the maximum independent set size $\alpha$, yielding a gap-independent bound of $\mathcal{O}(\sqrt{\alpha T \ln T \ln(kT)})$.
• The analysis shows that UCB and Thompson Sampling achieve improved performance without modifying their selection rules, solely by absorbing feedback from neighboring arms.
• The regret bound is derived via a novel layering technique that partitions arms by gap size and bounds selection probabilities in each layer.
• The extra logarithmic factor in the regret compared to prior work (e.g., Buccapatnam et al.) is shown to potentially be inherent to the analysis method, not the algorithms themselves.
• The results demonstrate that classical bandit algorithms can implicitly exploit graph feedback structures, even when they do not use the graph for exploration.