[Paper Review] When to Call Your Neighbor? Strategic Communication in Cooperative Stochastic Bandits

Udari Madhushani, Naomi Ehrich Leonard|arXiv (Cornell University)|Oct 8, 2021

Advanced Bandit Algorithms Research23 references4 citations

TL;DR

This paper proposes ComEx, a cost-effective communication protocol for cooperative stochastic bandits that reduces communication from Θ(T) to O(log T) while maintaining the same optimal regret performance as full communication. ComEx strategically transmits only rewards from exploratory actions—those likely to involve suboptimal arms—thereby minimizing redundant messages without sacrificing group performance.

ABSTRACT

In cooperative bandits, a framework that captures essential features of collective sequential decision making, agents can minimize group regret, and thereby improve performance, by leveraging shared information. However, sharing information can be costly, which motivates developing policies that minimize group regret while also reducing the number of messages communicated by agents. Existing cooperative bandit algorithms obtain optimal performance when agents share information with their neighbors at extit{every time step}, i.e., full communication. This requires $Θ(T)$ number of messages, where $T$ is the time horizon of the decision making process. We propose extit{ComEx}, a novel cost-effective communication protocol in which the group achieves the same order of performance as full communication while communicating only $O(\log T)$ number of messages. Our key step is developing a method to identify and only communicate the information crucial to achieving optimal performance. Further we propose novel algorithms for several benchmark cooperative bandit frameworks and show that our algorithms obtain extit{state-of-the-art} performance while consistently incurring a significantly smaller communication cost than existing algorithms.

Motivation & Objective

• To reduce communication cost in cooperative stochastic bandits without degrading group performance.
• To identify which information is most valuable for minimizing regret, focusing on rewards from suboptimal arm pulls.
• To develop a protocol that enables agents to communicate only when necessary, based on exploration status.
• To integrate the protocol into multiple benchmark cooperative bandit frameworks with theoretical and empirical validation.
• To achieve state-of-the-art performance with significantly lower communication overhead than existing methods.

Proposed method

• ComEx transmits messages only when an agent performs an exploratory action, defined as pulling an arm that is not the current best estimate.
• The protocol uses a delayed message-passing mechanism with a finite memory window γ to discard outdated information.
• Agents maintain local estimates of arm means and confidence bounds, updating them using received messages from neighbors.
• The algorithm integrates with UCB, Estimation-based UCB, and Thompson Sampling variants to ensure compatibility across frameworks.
• Messages contain agent ID, time, action, and reward, enabling consensus-based estimation while minimizing redundancy.
• A message discard mechanism ensures only recent, relevant data is used, reducing storage and communication overhead.

Experimental results

Research questions

• RQ1 Can we reduce communication cost in cooperative bandits while maintaining optimal regret performance?
• RQ2 Which types of information are most critical for minimizing group regret in cooperative bandit settings?
• RQ3 Can we achieve the same regret order as full communication with only O(log T) messages instead of Θ(T)?
• RQ4 How can we design a communication protocol that selectively transmits only valuable information from exploratory actions?
• RQ5 Can ComEx be effectively integrated into diverse cooperative bandit frameworks with provable performance guarantees?

Key findings

• ComEx achieves the same order of group regret as full communication, proving that O(log T) messages suffice for optimal performance.
• The communication cost of ComEx is O(log T), a significant reduction from the Θ(T) cost of full communication.
• In all benchmark frameworks—decentralized and centralized with reward sharing, message passing, and estimate sharing—ComEx-based algorithms outperform existing methods in communication efficiency.
• Experiments confirm that ComEx-based algorithms consistently incur significantly lower communication costs while maintaining or improving performance over state-of-the-art baselines.
• The protocol is compatible with multiple bandit algorithms, including UCB, Estimation-UCB, and Thompson Sampling, with theoretical guarantees across all variants.
• By transmitting only rewards from exploratory pulls, ComEx avoids redundant communication while preserving the ability to identify and avoid suboptimal arms.