[Paper Review] Racing Thompson: an Efficient Algorithm for Thompson Sampling with Non-conjugate Priors

Yichi Zhou, Jun Zhu+1 more|arXiv (Cornell University)|Aug 16, 2017

Advanced Bandit Algorithms Research16 references4 citations

TL;DR

This paper proposes Racing Thompson, an efficient algorithm for Thompson sampling with non-conjugate priors by reformulating it as a best arm identification problem using the Gumbel-Max trick. The method avoids intractable posterior inference by sampling from a tractable proposal distribution, achieving superior performance over SMC and conjugate-prior baselines in both non-conjugate and structured prior settings.

ABSTRACT

Thompson sampling has impressive empirical performance for many multi-armed bandit problems. But current algorithms for Thompson sampling only work for the case of conjugate priors since these algorithms require to infer the posterior, which is often computationally intractable when the prior is not conjugate. In this paper, we propose a novel algorithm for Thompson sampling which only requires to draw samples from a tractable distribution, so our algorithm is efficient even when the prior is non-conjugate. To do this, we reformulate Thompson sampling as an optimization problem via the Gumbel-Max trick. After that we construct a set of random variables and our goal is to identify the one with highest mean. Finally, we solve it with techniques in best arm identification.

Motivation & Objective

• To address the computational intractability of posterior inference in Thompson sampling under non-conjugate priors.
• To enable efficient and scalable Thompson sampling in complex models where conjugate priors are infeasible or suboptimal.
• To provide a theoretically grounded method that avoids full posterior computation while maintaining strong empirical performance.
• To develop a practical algorithm with a principled stopping rule and sample size determination for non-conjugate settings.

Proposed method

• Reformulate Thompson sampling as a best arm identification (BAI) problem using the Gumbel-Max trick to convert sampling from a posterior into maximizing a random variable with Gumbel-distributed perturbations.
• Construct a tractable proposal distribution to sample from, replacing the need for exact posterior inference under non-conjugate priors.
• Use a racing-style algorithm to identify the arm with the highest expected value among the perturbed variables, leveraging theoretical guarantees from BAI.
• Determine the number of samples adaptively using a stopping rule based on subgaussian concentration, ensuring theoretical performance bounds.
• Integrate the method into a sequential decision-making framework where samples are drawn from the proposal distribution and used to select actions via the Gumbel-max criterion.
• Ensure theoretical validity by assuming subgaussianity of the random variables involved in the BAI formulation.

Experimental results

Research questions

• RQ1 Can Thompson sampling be efficiently implemented in non-conjugate prior settings without requiring posterior inference?
• RQ2 How can the Gumbel-Max trick be leveraged to reformulate Thompson sampling as a best arm identification problem?
• RQ3 What is a principled way to determine the number of samples needed in non-conjugate Thompson sampling with theoretical guarantees?
• RQ4 How does the performance of the proposed method compare to SMC and conjugate-prior baselines in non-conjugate and structured prior scenarios?

Key findings

• Racing Thompson significantly outperforms Thompson sampling with an improper Beta prior in Bernoulli bandits, especially as time steps increase.
• The algorithm achieves comparable performance to SMC with a small number of particles in early time steps, but surpasses SMC as the number of observations grows.
• In non-conjugate settings with truncated Gaussian priors, Racing Thompson outperforms both SMC and conjugate-prior Thompson sampling in terms of cumulative regret.
• For structured priors involving dependent arms (e.g., topic-based correlations), the method maintains strong performance, outperforming both SMC and conjugate-prior baselines.
• Theoretical analysis shows that the method's stopping rule is valid under subgaussian assumptions, providing a principled sample size selection mechanism.