[Paper Review] Meta-Thompson Sampling

Branislav Kveton, Mikhail Konobeev+5 more|arXiv (Cornell University)|Feb 11, 2021

Advanced Bandit Algorithms Research32 references4 citations

TL;DR

This paper introduces Meta-Thompson Sampling (MetaTS), a meta-learning algorithm that adapts Thompson Sampling to unknown priors by maintaining a meta-posterior over instance priors through interactions with multiple bandit tasks. It achieves improved Bayes regret by learning the true prior over time, with theoretical guarantees and empirical validation showing performance comparable to Thompson Sampling with a known prior.

ABSTRACT

Efficient exploration in bandits is a fundamental online learning problem. We propose a variant of Thompson sampling that learns to explore better as it interacts with bandit instances drawn from an unknown prior. The algorithm meta-learns the prior and thus we call it MetaTS. We propose several efficient implementations of MetaTS and analyze it in Gaussian bandits. Our analysis shows the benefit of meta-learning and is of a broader interest, because we derive a novel prior-dependent Bayes regret bound for Thompson sampling. Our theory is complemented by empirical evaluation, which shows that MetaTS quickly adapts to the unknown prior.

Motivation & Objective

• To address the challenge of designing Thompson Sampling when the true prior over arm means is unknown but drawn from a known meta-prior.
• To develop a computationally efficient meta-learning algorithm that adapts exploration strategies to the realized prior across multiple bandit tasks.
• To provide a novel prior-dependent Bayes regret bound for Thompson Sampling, extending its theoretical analysis.
• To empirically demonstrate that MetaTS adapts quickly to unknown priors and matches the performance of TS with known priors.

Proposed method

• MetaTS maintains a meta-posterior Q_s over the unknown instance prior P_* in each task s, updating it based on observed histories from previous tasks.
• In each task s, MetaTS samples a candidate prior P_s ∼ Q_s and runs standard Thompson Sampling with P_s for n rounds.
• The algorithm uses conjugate priors (e.g., Gaussian) to enable efficient posterior updates, leveraging closed-form updates for meta-posteriors in Gaussian bandits.
• MetaTS is implemented efficiently in both Bernoulli and Gaussian bandits, with updates based on observed rewards and arm pulls.
• Theoretical analysis bounds the Bayes regret using only a single arm pull per task, showing improvement due to adaptation to P_*.
• The method generalizes to linear bandits, with closed-form meta-posterior updates shown in Appendix D.

Experimental results

Research questions

• RQ1 Can Thompson Sampling be improved by meta-learning an unknown prior through interactions with multiple bandit instances?
• RQ2 Does MetaTS achieve Bayes regret comparable to Thompson Sampling with a known, true prior?
• RQ3 What is the theoretical benefit of meta-learning in terms of prior-dependent regret bounds for Thompson Sampling?
• RQ4 How robust is MetaTS to misspecification of the meta-prior?
• RQ5 Can MetaTS scale to higher-dimensional or larger-arm problems while maintaining performance?

Key findings

• MetaTS achieves Bayes regret that improves with meta-learning, even under a conservative analysis relying on only one arm pull per task.
• Empirical results show MetaTS adapts quickly to the unknown prior P_* and matches the performance of Thompson Sampling with a known prior.
• The regret improvement is preserved as the number of arms K or dimension d increases, though it diminishes when the prior width σ₀ approaches the meta-prior width σ_q.
• MetaTS is robust to meta-prior misspecification, with only minor performance degradation observed in experiments.
• A novel prior-dependent upper bound on the Bayes regret of Thompson Sampling is derived, which is of broader theoretical interest.
• Meta-posterior updates in linear bandits with Gaussian noise have a closed-form solution, enabling potential extension to contextual bandits.