[Paper Review] Infinite Arms Bandit: Optimality via Confidence Bounds

Hock Peng Chan, Shouri Hu|arXiv (Cornell University)|May 30, 2018

Advanced Bandit Algorithms Research27 references3 citations

TL;DR

This paper introduces the Confidence Bound Target (CBT) algorithm for the infinite-armed bandit problem, where arms are selected based on confidence bounds relative to a target value derived from priors or empirically. CBT achieves optimal regret bounds across bounded rewards, matching the theoretical lower bound, and its empirical variant works without prior knowledge, outperforming competitors in simulations on Bernoulli and real-world URL datasets.

ABSTRACT

Berry et al. (1997) initiated the development of the infinite arms bandit problem. They derived a regret lower bound of all allocation strategies for Bernoulli rewards with uniform priors, and proposed strategies based on success runs. Bonald and Proutière (2013) proposed a two-target algorithm that achieves the regret lower bound, and extended optimality to Bernoulli rewards with general priors. We present here a confidence bound target (CBT) algorithm that achieves optimality for rewards that are bounded above. For each arm we construct a confidence bound and compare it against each other and a target value to determine if the arm should be sampled further. The target value depends on the assumed priors of the arm means. In the absence of information on the prior, the target value is determined empirically. Numerical studies here show that CBT is versatile and outperforms its competitors.

Motivation & Objective

• Address the challenge of optimal allocation in the infinite-armed bandit problem, where the number of arms is unbounded and rewards are bounded above.
• Develop a method that achieves the theoretical regret lower bound for bounded rewards, extending prior work on Bernoulli and exponential family distributions.
• Provide a practical variant of the algorithm that does not require prior knowledge of arm mean distributions by estimating the target value empirically.
• Demonstrate the robustness and superiority of the proposed method across diverse reward distributions and real-world data.

Proposed method

• Construct an upper confidence bound (UCB) for each arm's expected reward using observed samples.
• Define a target value based on the assumed prior distribution of arm means, which determines whether an arm should be explored further.
• Compare each arm’s confidence bound against the target: if the bound exceeds the target, the arm is selected; otherwise, it is discarded.
• For unknown priors, estimate the target value empirically from initial samples, enabling the empirical CBT variant.
• Use a universal regret lower bound as a benchmark to validate optimality of the chosen target value.
• Apply the algorithm sequentially, updating confidence bounds and targets dynamically over time to balance exploration and exploitation.

Experimental results

Research questions

• RQ1 Can a confidence bound-based method achieve the theoretical regret lower bound in the infinite-armed bandit problem with bounded rewards?
• RQ2 How can the target value in the confidence bound comparison be chosen to ensure optimality without relying on parametric assumptions?
• RQ3 What is the performance of an empirical version of the algorithm when the true prior distribution is unknown?
• RQ4 How does the proposed CBT algorithm compare to existing methods like two-target, UCB-F, and POKER in terms of regret and scalability?
• RQ5 Can the CBT framework be applied effectively to real-world datasets with unknown reward structures?

Key findings

• The CBT algorithm achieves the universal regret lower bound for bounded rewards, matching the theoretical minimum regret across all possible strategies.
• For Bernoulli rewards with known priors, CBT consistently outperforms the two-target algorithm and UCB-F, achieving lower regret across all tested sample sizes.
• Empirical CBT, which estimates the target value from data, performs competitively with the two-target algorithm and significantly outperforms standard baselines like ε-greedy and ε-first.
• On the URL dataset, empirical CBT surpasses POKER in regret at larger sample sizes (n=1300), indicating strong scalability and adaptability.
• The optimal CBT requires prior knowledge to set the threshold ζ, but empirical CBT achieves near-optimal performance without such knowledge, making it more practical.
• Numerical results show that CBT’s regret scales as Cn^(2/3) for β=2 and Cn^(3/4) for β=3, closely approaching the theoretical lower bound across all settings.