[Paper Review] Nonstochastic Multiarmed Bandits with Unrestricted Delays

Tobias Sommer Thune, Nicolò Cesa‐Bianchi+1 more|arXiv (Cornell University)|Jun 3, 2019

Advanced Bandit Algorithms Research22 references3 citations

TL;DR

This paper proposes a delayed-Exp3 algorithm with a novel skipping wrapper to handle unbounded delays in nonstochastic multiarmed bandits, achieving the conjectured regret bound of $\mathcal{O}(\sqrt{(KT+D)\ln K})$ without requiring bounded delays. It further introduces a doubling scheme that eliminates prior knowledge of $T$ and $D$ when delays are available at action time, yielding a refined oracle bound with polynomially better dependence on problem parameters in certain cases.

ABSTRACT

We investigate multiarmed bandits with delayed feedback, where the delays need neither be identical nor bounded. We first prove that "delayed" Exp3 achieves the $O(\sqrt{(KT + D)\ln K} )$ regret bound conjectured by Cesa-Bianchi et al. [2019] in the case of variable, but bounded delays. Here, $K$ is the number of actions and $D$ is the total delay over $T$ rounds. We then introduce a new algorithm that lifts the requirement of bounded delays by using a wrapper that skips rounds with excessively large delays. The new algorithm maintains the same regret bound, but similar to its predecessor requires prior knowledge of $D$ and $T$. For this algorithm we then construct a novel doubling scheme that forgoes the prior knowledge requirement under the assumption that the delays are available at action time (rather than at loss observation time). This assumption is satisfied in a broad range of applications, including interaction with servers and service providers. The resulting oracle regret bound is of order $\min_β(|S_β|+β\ln K + (KT + D_β)/β)$, where $|S_β|$ is the number of observations with delay exceeding $β$, and $D_β$ is the total delay of observations with delay below $β$. The bound relaxes to $O (\sqrt{(KT + D)\ln K} )$, but we also provide examples where $D_β\ll D$ and the oracle bound has a polynomially better dependence on the problem parameters.

Motivation & Objective

• To close the gap between the conjectured and proven regret bounds for nonstochastic multiarmed bandits under variable, unbounded delays.
• To address the limitation of prior algorithms that require bounded delays or prior knowledge of $T$ and $D$.
• To design a practical algorithm that adapts to unknown delays by using a threshold-based skipping mechanism.
• To develop a doubling scheme that eliminates the need for prior knowledge of $T$ and $D$ under the assumption that delays are known at action time.
• To derive a refined oracle regret bound that improves upon the standard bound when $D_\beta \ll D$.

Proposed method

• Introduces a modified 'delayed-Exp3' algorithm that performs updates as soon as losses become available, even with variable delays.
• Proposes a wrapper algorithm called 'Skipper' that skips rounds with delays exceeding a threshold $\beta$, thereby limiting the effective delay seen by the base algorithm.
• Uses a refined analysis of exponential weights with delayed feedback, introducing probability drift lemmas to bound the impact of delayed updates.
• Employs a doubling scheme that adaptively tunes the threshold $\beta$ without prior knowledge of $T$ and $D$, assuming delays are known at action time.
• Derives a novel oracle regret bound: $\min_\beta \left(|S_\beta| + \beta\ln K + \frac{KT + D_\beta}{\beta}\right)$, where $|S_\beta|$ is the number of observations with delay $> \beta$ and $D_\beta$ is the total delay below $\beta$.
• Applies the analysis to show that the bound reduces to $\mathcal{O}(\sqrt{(KT+D)\ln K})$ under standard assumptions, but can be significantly tighter when $D_\beta \ll D$.

Experimental results

Research questions

• RQ1 Can the conjectured regret bound of $\mathcal{O}(\sqrt{(KT+D)\ln K})$ be achieved in the nonstochastic bandit setting with unbounded delays?
• RQ2 Is it possible to remove the requirement for prior knowledge of $T$ and $D$ while maintaining the same regret bound under delayed feedback?
• RQ3 Can a refined regret bound be derived that depends on the distribution of delays, particularly when most delays are small?
• RQ4 Does the assumption that delays are known at action time enable better regret guarantees than when delays are only known at observation time?
• RQ5 Can a doubling scheme be designed to adaptively tune the delay threshold without prior knowledge of $T$ and $D$?

Key findings

• The delayed-Exp3 algorithm achieves the conjectured $\mathcal{O}(\sqrt{(KT+D)\ln K})$ regret bound when delays are bounded and $T$, $D$, and $d_{\text{max}}$ are known.
• The Skipper wrapper algorithm enables the use of delayed-Exp3 under unbounded delays by skipping observations with delays exceeding a threshold $\beta$, maintaining the same regret bound.
• When delays are known at action time, a doubling scheme eliminates the need for prior knowledge of $T$ and $D$, achieving a refined oracle regret bound.
• The refined oracle bound $\min_\beta \left(|S_\beta| + \beta\ln K + \frac{KT + D_\beta}{\beta}\right)$ can be polynomially better than $\mathcal{O}(\sqrt{(KT+D)\ln K})$ when $D_\beta \ll D$.
• The analysis establishes that the regret bound is tight in the worst case, but can be significantly improved in favorable delay distributions.
• The paper provides a concrete example where $D_\beta \ll D$, demonstrating the potential for substantial performance gains with the refined bound.