[Paper Review] Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling

Carlos Riquelme, George Tucker+1 more|arXiv (Cornell University)|Feb 25, 2018

Advanced Bandit Algorithms ResearchDecision Sciences46 citations

TL;DR

The paper benchmarked a wide range of approximate Bayesian neural network methods for Thompson Sampling in contextual bandits, highlighting how different posterior approximations affect online decision-making. It finds that some methods underperform in sequential settings while simpler Bayesian-linear-on-top approaches can be robust and easy to tune.

ABSTRACT

Recent advances in deep reinforcement learning have made significant strides in performance on applications such as Go and Atari games. However, developing practical methods to balance exploration and exploitation in complex domains remains largely unsolved. Thompson Sampling and its extension to reinforcement learning provide an elegant approach to exploration that only requires access to posterior samples of the model. At the same time, advances in approximate Bayesian methods have made posterior approximation for flexible neural network models practical. Thus, it is attractive to consider approximate Bayesian neural networks in a Thompson Sampling framework. To understand the impact of using an approximate posterior on Thompson Sampling, we benchmark well-established and recently developed methods for approximate posterior sampling combined with Thompson Sampling over a series of contextual bandit problems. We found that many approaches that have been successful in the supervised learning setting underperformed in the sequential decision-making scenario. In particular, we highlight the challenge of adapting slowly converging uncertainty estimates to the online setting.

Motivation & Objective

• Motivate exploration-exploitation in deep sequential decision making and quantify how approximate posteriors impact Thompson Sampling in contextual bandits.
• Provide a comprehensive benchmark of Bayesian deep learning methods (variational, EP, dropout, bootstrap, noise injection, GPs) for Thompson Sampling.
• Identify practical algorithms that balance accuracy of uncertainty estimation with online decision efficiency.
• Offer guidance on which posterior approximations scale and perform under online constraints.

Proposed method

• Implement Thompson Sampling with multiple posterior approximation families ( Bayesian linear regression, diagonal vs full covariance, Neural Linear, Neural Greedy, Variational Inference, Expectation-Propagation, Dropout, Monte Carlo methods, Bootstrap, Direct Noise Injection, Gaussian Processes ).
• Use neural networks with a shared two-hidden-layer architecture to model context-to-action rewards, with per-action outputs.
• Compare online updates and training schedules (frequency of updates, mini-batch sizes) to reflect online decision-making constraints.
• Evaluate on synthetic and real-world contextual bandit problems with multiple datasets (e.g., Mushroom, Statlog, Covertype, Financial, Jester, Adult, Census, Song).
• Report cumulative regret and simple regret across reshuffled trials to assess robustness of uncertainty estimates.

Experimental results

Research questions

• RQ1 How do different approximate posterior methods for Bayesian neural networks affect Thompson Sampling performance in contextual bandits?
• RQ2 Which posterior approximation strategies scale online and provide robust uncertainty estimates for sequential decision making?
• RQ3 Do simpler approaches (e.g., Bayesian linear regression on top of learned representations) outperform more complex Bayesian neural network approximations in online bandit settings?
• RQ4 How does the online data-feedback loop influence the quality of posterior approximations and regret outcomes?

Key findings

• Dropout, simple noise injection, and bootstrapping can boost performance on some tasks but may not solve challenging exploration problems.
• Many complex posterior methods (variational inference, black-box alpha-divergence, minibatch MCMC) tightly couple representation learning with uncertainty, which can hinder online decision making.
• Bayesian linear regression on the last-layer representation (Neural Linear) offers robust, easy-to-tune uncertainty and strong baseline performance in several settings.
• Diagonal covariance approximations can lead to substantial regret in high-dimensional problems due to over-exploration across many dimensions, while more expressive approximations may mitigate this.
• The paper highlights that posterior quality does not always predict online performance due to the feedback loop in sequential data collection.
• Empirical results across real-world datasets show varying performance across algorithms, with no single method dominating all problems.