[Paper Review] Accelerating Approximate Thompson Sampling with Underdamped Langevin Monte Carlo

Haoyang Zheng, Wei Deng+2 more|arXiv (Cornell University)|Jan 22, 2024

Advanced Bandit Algorithms ResearchDecision Sciences3 citations

TL;DR

This paper proposes an accelerated approximate Thompson sampling algorithm using underdamped Langevin Monte Carlo (ULMC) to improve sample complexity and regret performance in high-dimensional bandit problems. By leveraging momentum in the sampling process and analyzing posterior concentration via stochastic differential equations, the method reduces sample complexity from 𝒪̃(d) to 𝒪̃(√d), achieving improved regret bounds while maintaining scalability and robustness in synthetic experiments.

ABSTRACT

Approximate Thompson sampling with Langevin Monte Carlo broadens its reach from Gaussian posterior sampling to encompass more general smooth posteriors. However, it still encounters scalability issues in high-dimensional problems when demanding high accuracy. To address this, we propose an approximate Thompson sampling strategy, utilizing underdamped Langevin Monte Carlo, where the latter is the go-to workhorse for simulations of high-dimensional posteriors. Based on the standard smoothness and log-concavity conditions, we study the accelerated posterior concentration and sampling using a specific potential function. This design improves the sample complexity for realizing logarithmic regrets from $\mathcal{ ilde O}(d)$ to $\mathcal{ ilde O}(\sqrt{d})$. The scalability and robustness of our algorithm are also empirically validated through synthetic experiments in high-dimensional bandit problems.

Motivation & Objective

• To address scalability issues in approximate Thompson sampling for high-dimensional posterior distributions.
• To improve sample complexity and regret performance in multi-armed bandit problems with non-Gaussian posteriors.
• To integrate underdamped Langevin Monte Carlo (ULMC) into Thompson sampling for more efficient posterior sampling.
• To theoretically analyze posterior concentration rates using a specific potential function in the context of SDE trajectories.
• To empirically validate the robustness and effectiveness of the proposed method across high-dimensional synthetic bandit settings.

Proposed method

• Utilizes underdamped Langevin Monte Carlo (ULMC) as the sampling engine to simulate high-dimensional posterior distributions with momentum.
• Models the posterior as the invariant distribution of a stochastic differential equation (SDE) with a specific potential function.
• Analyzes SDE trajectories under standard smoothness and log-concavity assumptions to derive posterior concentration rates.
• Introduces a momentum-based dynamics that accelerates exploration and improves mixing in high-dimensional spaces.
• Employs a sub-Gaussian concentration bound on gradient estimation errors to control approximation error in the sampling process.
• Derives theoretical bounds on the moment generating function of the distance between sampled points and the true posterior mode.

Experimental results

Research questions

• RQ1 Can underdamped Langevin Monte Carlo reduce the sample complexity of approximate Thompson sampling in high-dimensional settings?
• RQ2 How does the inclusion of momentum in Langevin dynamics affect posterior concentration and regret performance?
• RQ3 What is the theoretical sample complexity required to achieve logarithmic regret using ULMC-based Thompson sampling?
• RQ4 Does improved sample complexity from 𝒪̃(d) to 𝒪̃(√d) translate into better regret performance in practice?
• RQ5 How robust is the proposed method across varying dimensions and hyperparameter configurations in synthetic bandit experiments?

Key findings

• The proposed ULMC-based approximate Thompson sampling reduces sample complexity for achieving logarithmic regret from 𝒪̃(d) to 𝒪̃(√d).
• Theoretical analysis shows that the momentum in ULMC improves posterior concentration rates under standard smoothness and log-concavity conditions.
• Empirical results demonstrate improved regret performance under the same sample complexity constraints, validating the theoretical gains.
• The method maintains robustness and scalability across high-dimensional synthetic bandit problems with up to 1000 dimensions.
• The moment generating function of the sampling error is bounded, supporting the stability and convergence of the algorithm.
• The work provides the first empirical validation that reduced sample complexity directly leads to improved regret in Thompson sampling.