[Paper Review] Bayesian Online Model Selection

Aida Afshar, Yuke Zhang+1 more|arXiv (Cornell University)|Feb 20, 2026

Advanced Bandit Algorithms Research0 citations

TL;DR

Introduces a Bayesian algorithm for online model selection in stochastic bandits and proves an oracle-best Bayesian regret bound of Õ(d* M √T + √(M T)); shows data sharing and prior mis-specification effects experimentally.

ABSTRACT

Online model selection in Bayesian bandits raises a fundamental exploration challenge: When an environment instance is sampled from a prior distribution, how can we design an adaptive strategy that explores multiple bandit learners and competes with the best one in hindsight? We address this problem by introducing a new Bayesian algorithm for online model selection in stochastic bandits. We prove an oracle-style guarantee of $O\left( d^* M \sqrt{T} + \sqrt{(MT)} ight)$ on the Bayesian regret, where $M$ is the number of base learners, $d^*$ is the regret coefficient of the optimal base learner, and $T$ is the time horizon. We also validate our method empirically across a range of stochastic bandit settings, demonstrating performance that is competitive with the best base learner. Additionally, we study the effect of sharing data among base learners and its role in mitigating prior mis-specification.

Motivation & Objective

• Motivate online model selection in Bayesian bandits where environment instances are drawn from a prior.
• Design a meta-learner that chooses among multiple base bandit learners with oracle-best guarantees.
• Provide a data-driven approach that leverages posterior sampling to compare base learners without requiring known regret bounds.
• Demonstrate empirical performance and robustness to prior mis-specification and data sharing among base learners.

Proposed method

• Propose a Bayesian online model selection (B-MS) algorithm that maintains a global posterior over base learners and samples mean rewards from it.
• Define a balancing potential φt(i) = nt^i * μ̃t* − ∑l∈It^i μ̃t(al) to compare base learners and select the one with minimum potential.
• Show that when base learners are stationary arms, the method recovers a TS-like Bayesian regret bound.
• Prove an oracle-best Bayesian regret bound of ḂayesRegret_T ≤ ṫilde{O}(d⋆ M √T + √(M T)).
• Demonstrate that sharing data across base learners improves performance and helps mitigate prior mis-specification.

(a) Well-specified meta learner, one well-specified base learner

Experimental results

Research questions

• RQ1 Can a Bayesian meta-learner compete with the best base learner chosen in hindsight for realized environments drawn from a prior?
• RQ2 How does data sharing among base learners affect learning efficiency and robustness to prior mis-specification?
• RQ3 What is the Bayesian regret of the proposed online model selection algorithm, and how does it scale with horizon T, number of base learners M, and the optimal base learner’s regret coefficient d⋆?
• RQ4 How does the Bayesian online model selection framework relate to and generalize Thompson Sampling in stationary arms?
• RQ5 Under what conditions does the proposed method recover classical TS guarantees in special cases?

Key findings

• The proposed B-MS algorithm achieves a Bayesian regret bound of Õ(d⋆ M √T + √(M T)).
• The method generalizes Thompson Sampling; with K base learners each fixing an arm, it recovers the Õ(√(K T)) Bayesian regret.
• Data sharing among base learners improves meta-learner performance across experiments.
• A mis-specified meta-learner can recover when at least one base learner is well-specified, illustrating robustness to mis-specification.
• Experimental results show B-MS competitive with the best base learner across UCB and LinTS settings and various priors.

(b) Mis-specified meta learner, one well-specified base learner