[Paper Review] A Bit Better? Quantifying Information for Bandit Learning

Adithya M. Devraj, Benjamin Van Roy+1 more|arXiv (Cornell University)|Feb 18, 2021

Advanced Bandit Algorithms Research23 references4 citations

TL;DR

This paper investigates whether using Tsallis entropy instead of Shannon entropy improves the performance of information-directed sampling (IDS) in multi-armed bandit problems. Despite tighter theoretical regret bounds with Tsallis entropy, the authors find through computational experiments that IDS with Shannon entropy achieves realized performance comparable to Tsallis-IDS, suggesting the analytical advantage may not translate to practical gains. They further propose a modified analysis template enabling order-optimal regret bounds for Thompson sampling using Shannon entropy.

ABSTRACT

The information ratio offers an approach to assessing the efficacy with which an agent balances between exploration and exploitation. Originally, this was defined to be the ratio between squared expected regret and the mutual information between the environment and action-observation pair, which represents a measure of information gain. Recent work has inspired consideration of alternative information measures, particularly for use in analysis of bandit learning algorithms to arrive at tighter regret bounds. We investigate whether quantification of information via such alternatives can improve the realized performance of information-directed sampling, which aims to minimize the information ratio.

Motivation & Objective

• To evaluate whether Tsallis entropy-based information quantification improves the realized performance of information-directed sampling (IDS) compared to Shannon entropy.
• To investigate whether the tighter regret bounds observed with Tsallis entropy in theory translate into practical performance gains.
• To develop a modified analysis framework that enables order-optimal regret bounds for Thompson sampling using Shannon entropy.
• To assess the empirical behavior of information ratios under different information measures across various bandit settings.

Proposed method

• The authors compare two variants of IDS: one using Shannon entropy and another using Tsallis entropy to quantify information gain.
• They compute the information ratio as the squared expected regret divided by the respective information measure (Shannon or Tsallis entropy) to guide action selection.
• A computational study evaluates IDS performance using both information measures on synthetic bandit problems, including a counterexample and Beta-Bernoulli bandits.
• The study uses empirical averages of the scaled information ratio over sample paths to estimate the information ratio dynamics over time.
• The authors propose a modified analysis template that allows deriving order-optimal regret bounds for Thompson sampling using Shannon entropy, which was previously unattainable with the standard information ratio framework.
• Discretization is used to approximate continuous integrals in algorithm implementation, enabling numerical evaluation of the information ratio.

Experimental results

Research questions

• RQ1 Does using Tsallis entropy instead of Shannon entropy lead to better empirical performance in information-directed sampling for bandit problems?
• RQ2 Can the tighter theoretical regret bounds for Tsallis-IDS be realized in practice, or is the improvement an artifact of current analysis techniques?
• RQ3 Is it possible to achieve order-optimal regret bounds for Thompson sampling using Shannon entropy via a modified analysis framework?
• RQ4 How do the dynamics of the information ratio evolve over time under different information measures and bandit settings?

Key findings

• Despite tighter theoretical regret bounds for Tsallis-IDS, the realized performance of IDS with Shannon entropy is empirically comparable to that of Tsallis-IDS across multiple bandit environments.
• The information ratio for both Shannon- and Tsallis-based IDS stabilizes over time and converges quickly, especially in high-dimensional settings (K=40).
• The authors demonstrate that a modified analysis template enables order-optimal regret bounds for Thompson sampling using Shannon entropy, which was not possible with the original information ratio framework.
• In the Beta-Bernoulli bandit setting, the scaled information ratio for Shannon entropy remains consistently lower than for Tsallis entropy, even though the worst-case bound is larger for Shannon.
• The information ratio dynamics show chattering near zero when the optimal arm is identified, suggesting both regret and information gain are minimal in the convergence phase.