[Paper Review] Garbage In, Reward Out: Bootstrapping Exploration in Multi-Armed Bandits

Branislav Kveton, Csaba Szepesvári+4 more|arXiv (Cornell University)|Nov 13, 2018

Advanced Bandit Algorithms Research30 references13 citations

TL;DR

This paper introduces Giro, a multi-armed bandit algorithm that uses non-parametric bootstrapping with specially designed pseudo rewards to enable effective exploration. It proves a regret bound of $O(K\Delta^{-1}\log n)$ in Bernoulli bandits, establishing the first formal justification for using non-parametric bootstrapping as a principled exploration strategy beyond heuristic applications.

ABSTRACT

We propose a bandit algorithm that explores by randomizing its history of rewards. Specifically, it pulls the arm with the highest mean reward in a non-parametric bootstrap sample of its history with pseudo rewards. We design the pseudo rewards such that the bootstrap mean is optimistic with a sufficiently high probability. We call our algorithm Giro, which stands for garbage in, reward out. We analyze Giro in a Bernoulli bandit and derive a $O(K Δ^{-1} \log n)$ bound on its $n$-round regret, where $Δ$ is the difference in the expected rewards of the optimal and the best suboptimal arms, and $K$ is the number of arms. The main advantage of our exploration design is that it easily generalizes to structured problems. To show this, we propose contextual Giro with an arbitrary reward generalization model. We evaluate Giro and its contextual variant on multiple synthetic and real-world problems, and observe that it performs well.

Motivation & Objective

• To develop a data-driven, tunable exploration strategy for multi-armed and contextual bandits that avoids reliance on confidence sets or posteriors.
• To formally justify non-parametric bootstrapping as a valid and effective method for exploration in bandit problems.
• To design a general framework that easily extends to structured problems like linear and contextual bandits.
• To provide theoretical regret bounds for the proposed algorithm, ensuring statistical efficiency.
• To empirically validate Giro and its contextual variant on synthetic and real-world datasets, demonstrating strong performance.

Proposed method

• Giro constructs a non-parametric bootstrap sample from the agent's historical rewards, including pseudo rewards to ensure optimism in the bootstrap mean.
• Pseudo rewards are assigned extreme values (e.g., 1 or 0) after each arm pull to bias the bootstrap mean toward optimism with high probability.
• The algorithm selects the arm with the highest mean reward in the bootstrap sample, effectively balancing exploration and exploitation.
• Theoretical analysis shows that the bootstrap mean is optimistic with high probability, enabling a tight regret bound.
• The method generalizes naturally to contextual bandits by incorporating context-dependent reward models.
• An efficient implementation is derived for Bernoulli bandits using binomial sampling, avoiding full resampling in each round.

Experimental results

Research questions

• RQ1 Can non-parametric bootstrapping with carefully designed pseudo rewards provide a theoretically sound and practical exploration strategy in multi-armed bandits?
• RQ2 What is the regret performance of such a bootstrapping-based algorithm, and can it achieve a near-optimal bound?
• RQ3 How does the algorithm generalize to structured problems like contextual and linear bandits?
• RQ4 Does the proposed method outperform existing exploration strategies like epsilon-greedy, OFU, and Thompson sampling in practice?
• RQ5 Can randomization via bootstrapping alone induce effective exploration without relying on posterior sampling or confidence sets?

Key findings

• Giro achieves an $O(K\Delta^{-1}\log n)$ regret bound in $K$-armed Bernoulli bandits, matching the optimal rate up to logarithmic factors.
• The design of pseudo rewards ensures that the bootstrap mean is optimistic with high probability, which is key to the regret analysis.
• Theoretical analysis reveals that randomization in the sampling distribution, not just posterior sampling, can induce effective exploration.
• Empirical evaluation shows Giro performs competitively on both synthetic and real-world datasets, including the Adult and Covertype datasets.
• In real-world settings, Giro outperforms neural network-based models, indicating that poor performance in complex models is due to generalization issues, not lack of exploration.
• The algorithm generalizes naturally to contextual bandits, and its performance remains strong even when reward functions are non-linear.