[Paper Review] Bandit algorithms: Letting go of logarithmic regret for statistical robustness

Kumar Ashutosh, Jayakrishnan Nair+2 more|arXiv (Cornell University)|Jun 22, 2020

Advanced Bandit Algorithms Research4 citations

TL;DR

This paper establishes a fundamental trade-off in stochastic multi-armed bandits: algorithms with logarithmic regret are statistically fragile, while statistically robust algorithms must incur slightly super-logarithmic regret. The authors propose three classes of distribution-oblivious algorithms that achieve asymptotically near-logarithmic regret by using slowly growing scaling functions in confidence bounds, ensuring consistency across diverse distributions without requiring prior knowledge of distribution parameters.

ABSTRACT

We study regret minimization in a stochastic multi-armed bandit setting and establish a fundamental trade-off between the regret suffered under an algorithm, and its statistical robustness. Considering broad classes of underlying arms' distributions, we show that bandit learning algorithms with logarithmic regret are always inconsistent and that consistent learning algorithms always suffer a super-logarithmic regret. This result highlights the inevitable statistical fragility of all `logarithmic regret' bandit algorithms available in the literature---for instance, if a UCB algorithm designed for $\\sigma$-subGaussian distributions is used in a subGaussian setting with a mismatched variance parameter, the learning performance could be inconsistent. Next, we show a positive result: statistically robust and consistent learning performance is attainable if we allow the regret to be slightly worse than logarithmic. Specifically, we propose three classes of distribution oblivious algorithms that achieve an asymptotic regret that is arbitrarily close to logarithmic.

Motivation & Objective

• To investigate the statistical robustness of bandit algorithms that achieve logarithmic regret.
• To demonstrate that logarithmic regret algorithms are inherently inconsistent when applied to mismatched distributions, such as using a σ-subGaussian algorithm on σ′-subGaussian instances with σ′ > σ.
• To design algorithms that are both statistically robust and achieve regret asymptotically close to logarithmic, without requiring prior knowledge of distribution parameters.
• To explore how noisy prior information about reward distributions can be incorporated into the regret scaling function to improve short-horizon performance.

Proposed method

• Propose three classes of distribution-oblivious bandit algorithms that use a slowly growing scaling function f(t) in confidence bounds to control regret growth.
• Design R-UCB and R-UCB-G algorithms using f(t) = c + h(t), where h(t) grows slowly (e.g., log^α(t)), to balance asymptotic regret and robustness.
• Incorporate noisy prior information about subGaussian parameters into f(t) by setting f(t) = c + log(t), with c chosen to reflect prior estimates (e.g., σ ≈ 8), improving short-horizon performance.
• Use theoretical analysis to show that regret scales asymptotically as O(f(t)) with f(t) growing slower than any power of t, ensuring consistency across broad classes of distributions.
• Establish that any algorithm with logarithmic regret is inconsistent under distributional mismatch, while consistent algorithms must incur super-logarithmic regret.
• Prove that regret bounds are loose in practice, and empirical results show actual regret is significantly below theoretical upper bounds.

Experimental results

Research questions

• RQ1 Can logarithmic regret bandit algorithms maintain consistency when applied to distributions outside their assumed parametric class?
• RQ2 Is it possible to design bandit algorithms that are both statistically robust and achieve regret asymptotically close to logarithmic without prior knowledge of distribution parameters?
• RQ3 How does incorporating noisy prior information about reward distributions affect short-horizon regret in robust bandit algorithms?
• RQ4 What is the trade-off between the growth rate of the scaling function f(t) and the resulting regret and robustness in confidence-bound-based bandit algorithms?

Key findings

• Logarithmic regret algorithms are statistically fragile: when applied to a mismatched distribution (e.g., σ′-subGaussian with σ′ > σ), they can suffer power-law regret scaling, indicating inconsistency.
• Statistical robustness and logarithmic regret are mutually exclusive: consistent algorithms necessarily incur super-logarithmic regret.
• The proposed R-UCB and R-UCB-G algorithms achieve asymptotic regret that is arbitrarily close to logarithmic by using slowly growing scaling functions f(t), such as log^1.6(t) or log^2(t).
• Empirical results show that faster-growing f(t) leads to higher cumulative regret asymptotically, confirming theoretical expectations.
• Incorporating noisy prior information into f(t), such as setting f(t) = 512 + log(t) based on estimated σ ≈ 8, reduces short-horizon regret compared to f(t) = log(t), without sacrificing robustness.
• Observed regret in simulations is significantly lower than theoretical upper bounds, indicating that existing regret bounds are loose in practice.