[Paper Review] No Weighted-Regret Learning in Adversarial Bandits with Delays

Ilai Bistritz, Zhengyuan Zhou+3 more|arXiv (Cornell University)|Mar 8, 2021

Advanced Bandit Algorithms Research4 citations

TL;DR

This paper introduces no weighted-regret learning in adversarial bandits with delays, showing that even with linear regret due to large delays, FKM and EXP3 algorithms still ensure convergence of the weighted ergodic distribution to coarse correlated equilibria (CCE) and the weighted average to Nash equilibria in two-player zero-sum games. The key contribution is a novel doubling trick that maintains optimal regret scaling under unknown delays, enabling robust equilibrium approximation in delayed, bandit-feedback simulations of unknown games.

ABSTRACT

Consider a scenario where a player chooses an action in each round $t$ out of $T$ rounds and observes the incurred cost after a delay of $d_{t}$ rounds. The cost functions and the delay sequence are chosen by an adversary. We show that in a non-cooperative game, the expected weighted ergodic distribution of play converges to the set of coarse correlated equilibria if players use algorithms that have "no weighted-regret" in the above scenario, even if they have linear regret due to too large delays. For a two-player zero-sum game, we show that no weighted-regret is sufficient for the weighted ergodic average of play to converge to the set of Nash equilibria. We prove that the FKM algorithm with $n$ dimensions achieves an expected regret of $O\left(nT^{\frac{3}{4}}+\sqrt{n}T^{\frac{1}{3}}D^{\frac{1}{3}} ight)$ and the EXP3 algorithm with $K$ arms achieves an expected regret of $O\left(\sqrt{\log K\left(KT+D ight)} ight)$ even when $D=\sum_{t=1}^{T}d_{t}$ and $T$ are unknown. These bounds use a novel doubling trick that, under mild assumptions, provably retains the regret bound for when $D$ and $T$ are known. Using these bounds, we show that FKM and EXP3 have no weighted-regret even for $d_{t}=O\left(t\log t ight)$. Therefore, algorithms with no weighted-regret can be used to approximate a CCE of a finite or convex unknown game that can only be simulated with bandit feedback, even if the simulation involves significant delays.

Motivation & Objective

• To address the challenge of delayed feedback in multi-agent online learning, where standard no-regret algorithms fail due to linear regret under large delays.
• To establish that no weighted-regret learning still enables convergence to coarse correlated equilibria (CCE) and Nash equilibria (NE) in non-cooperative and zero-sum games, respectively, even when classical regret is linear.
• To develop a general-purpose doubling trick that preserves the order of magnitude of regret bounds under unknown horizon $T$ and total delay $D$, applicable to any online learning algorithm with delayed feedback.
• To demonstrate that widely used algorithms like FKM and EXP3 achieve no weighted-regret under unbounded delays (e.g., $d_t = O(t\log t)$), enabling their use in simulating unknown games with delayed feedback.
• To extend the theoretical foundation of online learning under delays by introducing weighted-regret as a more robust performance metric than classical regret in non-stationary, adversarial environments.

Proposed method

• Introduces the concept of weighted-regret in adversarial bandits with delays, where the regret is weighted by a sequence that accounts for delayed feedback.
• Proposes a novel doubling trick that adaptively tunes algorithm parameters (e.g., step size) without prior knowledge of $T$ and $D$, preserving the asymptotic order of regret bounds.
• Analyzes the FKM algorithm in $n$-dimensional space, deriving an expected regret bound of $O\left(nT^{\frac{3}{4}} + \sqrt{n}T^{\frac{1}{3}}D^{\frac{1}{3}}\right)$ under delayed feedback.
• Analyzes the EXP3 algorithm for $K$ arms, deriving an expected regret bound of $O\left(\sqrt{\log K\left(KT + D\right)}\right)$, where $D = \sum_{t=1}^T \min\{d_t, T-t+1\}$.
• Establishes that FKM and EXP3 achieve no weighted-regret when the weight sequence matches their step-size sequence, even under unbounded delays such as $d_t = O(t\log t)$.
• Proves that no weighted-regret implies convergence of the expected weighted ergodic distribution to the set of CCE in general non-cooperative games, and to the set of NE in two-player zero-sum games, via $L^1$ convergence of the weighted average of play.

Experimental results

Research questions

• RQ1 Can no weighted-regret learning still lead to convergence to coarse correlated equilibria (CCE) in non-cooperative games when classical regret is linear due to delays?
• RQ2 Does no weighted-regret imply convergence to Nash equilibria (NE) in two-player zero-sum games under delayed feedback?
• RQ3 Can a doubling trick be designed to maintain optimal regret scaling under unknown $T$ and $D$ in delayed online learning?
• RQ4 Do FKM and EXP3 achieve no weighted-regret under unbounded delay sequences like $d_t = O(t\log t)$, despite linear classical regret?
• RQ5 Can the framework of weighted-regret be extended to internal regret to approximate correlated equilibria in delayed feedback settings?

Key findings

• The FKM algorithm achieves an expected regret of $O\left(nT^{\frac{3}{4}} + \sqrt{n}T^{\frac{1}{3}}D^{\frac{1}{3}}\right)$ under delayed feedback, even when $T$ and $D$ are unknown, due to the proposed doubling trick.
• The EXP3 algorithm achieves an expected regret of $O\left(\sqrt{\log K\left(KT + D\right)}\right)$ under unknown $T$ and $D$, with the doubling trick preserving the order of magnitude of the regret bound.
• Despite linear classical regret when $d_t = O(t\log t)$, FKM and EXP3 achieve no weighted-regret with their step-size sequences as weights, enabling equilibrium convergence.
• The expected weighted ergodic distribution of play converges to the set of coarse correlated equilibria (CCE) when all players use no weighted-regret algorithms, even under adversarial delays.
• For two-player zero-sum games, the weighted ergodic average of play converges in $L^1$ to the set of Nash equilibria when players use no weighted-regret algorithms.
• The proposed doubling trick is plug-and-play and provably retains the asymptotic dependence on $T$ and $D$ of the regret bound, even when these parameters are unknown.