[Paper Review] Individual Regret in Cooperative Nonstochastic Multi-Armed Bandits

Yogev Bar-On, Yishay Mansour|arXiv (Cornell University)|Jul 7, 2019

Advanced Bandit Algorithms Research10 citations

TL;DR

This paper proposes distributed regret minimization algorithms for cooperative nonstochastic multi-armed bandits where agents communicate over a network to reduce individual regret. It achieves $ tilde{O}ig( extstyleig(1 + \frac{K}{|\mathcal{N}(v)|}\big)^{1/2} \sqrt{T}\big)$ individual expected regret per agent $v$, resolving an open problem from Cesa-Bianchi et al. (2019b) for both known and unknown communication graphs with only logarithmic difference in performance.

ABSTRACT

We study agents communicating over an underlying network by exchanging messages, in order to optimize their individual regret in a common nonstochastic multi-armed bandit problem. We derive regret minimization algorithms that guarantee for each agent $v$ an individual expected regret of $\widetilde{O}\left(\sqrt{\left(1+\frac{K}{\left|\mathcal{N}\left(v ight) ight|} ight)T} ight)$, where $T$ is the number of time steps, $K$ is the number of actions and $\mathcal{N}\left(v ight)$ is the set of neighbors of agent $v$ in the communication graph. We present algorithms both for the case that the communication graph is known to all the agents, and for the case that the graph is unknown. When the graph is unknown, each agent knows only the set of its neighbors and an upper bound on the total number of agents. The individual regret between the models differs only by a logarithmic factor. Our work resolves an open problem from [Cesa-Bianchi et al., 2019b].

Motivation & Objective

• To address individual regret minimization in cooperative nonstochastic multi-armed bandit problems where agents communicate over a network.
• To design algorithms that ensure each agent achieves low individual regret despite limited global information.
• To resolve an open problem in Cesa-Bianchi et al. (2019b) concerning regret bounds under both known and unknown communication graph settings.
• To analyze the impact of network structure—specifically local neighborhood size—on individual regret performance.

Proposed method

• Agents communicate via a fixed communication graph, exchanging messages to share information about action rewards and decisions.
• The algorithm uses a decentralized exploration-exploitation strategy based on confidence bounds and local averaging of rewards across neighbors.
• For unknown graphs, agents use a distributed estimation procedure to infer network structure and maintain consistent regret bounds.
• The regret analysis leverages concentration inequalities and network connectivity properties to bound individual regret in terms of local neighborhood size $|\mathcal{N}(v)|$.
• A key component is the use of a weighted average of rewards across neighbors to reduce variance and improve individual performance.
• The algorithm adapts to both known and unknown graph settings, with only a logarithmic factor difference in regret performance.

Experimental results

Research questions

• RQ1 Can individual regret in cooperative nonstochastic multi-armed bandits be minimized when agents only communicate locally over a network?
• RQ2 How does the size of an agent’s neighborhood $|\mathcal{N}(v)|$ affect its individual regret bound?
• RQ3 What is the optimal regret performance achievable when the communication graph is unknown to the agents?
• RQ4 How does the proposed algorithm compare to centralized or fully informed settings in terms of regret scaling?
• RQ5 Can the open problem from Cesa-Bianchi et al. (2019b) on individual regret in cooperative bandits be resolved under general network topologies?

Key findings

• The proposed algorithm guarantees individual expected regret of $\widetilde{O}\big(\big(1 + \frac{K}{|\mathcal{N}(v)|}\big)^{1/2} \sqrt{T}\big)$ for each agent $v$, where $T$ is the time horizon and $K$ is the number of actions.
• The regret bound scales inversely with the square root of the number of neighbors, indicating that agents with more neighbors achieve lower regret.
• The algorithm performs nearly optimally even when the communication graph is unknown, with only a logarithmic factor difference in regret compared to the known-graph case.
• The result resolves an open problem posed in Cesa-Bianchi et al. (2019b) regarding individual regret in cooperative nonstochastic bandits.
• The analysis shows that local information and communication suffice to achieve near-optimal regret, without requiring global knowledge of the network.