[Paper Review] Near-Optimal Collaborative Learning in Bandits

Clémence Réda, Sattar Vakili+1 more|arXiv (Cornell University)|May 31, 2022

Advanced Bandit Algorithms Research4 citations

TL;DR

This paper proposes a near-optimal algorithm for collaborative best arm identification in a weighted multi-agent bandit setting, where each agent seeks the arm with the highest expected mixed reward—defined as a weighted average of local rewards across all agents. The method uses phased elimination with a data-dependent sampling scheme to minimize sample complexity while maintaining low communication cost, achieving near-optimal performance up to logarithmic factors.

ABSTRACT

This paper introduces a general multi-agent bandit model in which each agent is facing a finite set of arms and may communicate with other agents through a central controller in order to identify, in pure exploration, or play, in regret minimization, its optimal arm. The twist is that the optimal arm for each agent is the arm with largest expected mixed reward, where the mixed reward of an arm is a weighted sum of the rewards of this arm for all agents. This makes communication between agents often necessary. This general setting allows to recover and extend several recent models for collaborative bandit learning, including the recently proposed federated learning with personalization (Shi et al., 2021). In this paper, we provide new lower bounds on the sample complexity of pure exploration and on the regret. We then propose a near-optimal algorithm for pure exploration. This algorithm is based on phased elimination with two novel ingredients: a data-dependent sampling scheme within each phase, aimed at matching a relaxation of the lower bound.

Motivation & Objective

• To address collaborative best arm identification in a multi-agent bandit framework where agents optimize a shared, weighted mixed reward function.
• To minimize sample complexity in pure exploration while maintaining low communication cost among agents.
• To generalize and extend recent models such as federated multi-armed bandits with personalization and collaborative bandit learning.
• To derive new theoretical lower bounds on sample complexity and regret for the collaborative setting.
• To design and evaluate a near-optimal algorithm that matches these theoretical bounds asymptotically.

Proposed method

• Introduces a weighted collaborative bandit model where each agent’s reward is a convex combination of local rewards across all agents, defined by a known weight matrix W.
• Proposes a phased elimination algorithm (W-CPE-BAI) that dynamically allocates samples based on estimated gaps in expected mixed rewards.
• Incorporates a data-dependent sampling scheme within each phase to match a relaxed version of the theoretical lower bound on sample complexity.
• Uses an optimization-based oracle to compute ideal sampling allocations, serving as a benchmark for algorithmic performance.
• Employs a central controller to coordinate communication, enabling agents to share empirical means and improve estimation of mixed rewards.
• Applies CVXPY and MOSEK for solving the underlying optimization problem in the oracle computation, with numerical stability checks.

Experimental results

Research questions

• RQ1 What is the fundamental sample complexity limit for collaborative best arm identification under a weighted mixed reward model?
• RQ2 How can agents minimize exploration cost while maintaining high-confidence identification of the optimal arm?
• RQ3 Can a communication-efficient algorithm achieve near-optimality in sample complexity, up to logarithmic factors?
• RQ4 How does personalization (via weight matrix W) affect the trade-off between exploration and communication?
• RQ5 What is the scaling behavior of the algorithm’s performance as the confidence level δ approaches zero?

Key findings

• The proposed W-CPE-BAI algorithm significantly reduces communication cost compared to the PF-UCB-BAI baseline, especially as personalization increases (α → 1).
• For α = 0.7, W-CPE-BAI uses 10 communication rounds versus 10 for PF-UCB-BAI but achieves a 15% lower exploration cost (45,765 vs. 55,812 samples).
• The ratio of actual exploration cost to the theoretical lower bound (c*/T*) decreases from 68.0 at δ=0.1 to 11.9 at δ=0.00001, indicating strong asymptotic scaling.
• The algorithm achieves near-optimality in sample complexity, with the ratio c*/T* decreasing as δ → 0, consistent with theoretical upper bounds.
• Numerical instability in oracle computation occurs when Δ′_min is too small, suggesting a need for robust online optimization in hard instances.
• All algorithms achieve zero error rate (δ̂ = 0) up to five decimal places, indicating high reliability in identifying the optimal arm.