[Paper Review] Multi-Player Bandits -- a Musical Chairs Approach

Jonathan Rosenski, Ohad Shamir+1 more|arXiv (Cornell University)|Dec 9, 2015

Advanced Bandit Algorithms Research5 references3 citations

TL;DR

This paper introduces two communication-free algorithms—Musical Chairs (MC) and Dynamic Musical Chairs (DMC)—for multi-player multi-armed bandits with collisions, achieving constant regret (MC) and Õ(√(xT)) regret (DMC) under a fixed reward gap, without prior knowledge of the number of players. The methods rely on decentralized exploration and collision resolution via randomization and persistence heuristics.

ABSTRACT

We consider a variant of the stochastic multi-armed bandit problem, where multiple players simultaneously choose from the same set of arms and may collide, receiving no reward. This setting has been motivated by problems arising in cognitive radio networks, and is especially challenging under the realistic assumption that communication between players is limited. We provide a communication-free algorithm (Musical Chairs) which attains constant regret with high probability, as well as a sublinear-regret, communication-free algorithm (Dynamic Musical Chairs) for the more difficult setting of players dynamically entering and leaving throughout the game. Moreover, both algorithms do not require prior knowledge of the number of players. To the best of our knowledge, these are the first communication-free algorithms with these types of formal guarantees. We also rigorously compare our algorithms to previous works, and complement our theoretical findings with experiments.

Motivation & Objective

• To address the challenge of decentralized multi-player multi-armed bandits where players cannot communicate and may collide when selecting the same arm.
• To design a communication-free algorithm that achieves constant regret in the static setting (all players present from start).
• To extend the solution to the dynamic setting where players can enter and leave during the game, ensuring sublinear regret.
• To remove the need for prior knowledge of the number of players, a common assumption in prior work.
• To provide theoretical guarantees and empirical validation for performance under realistic cognitive radio network constraints.

Proposed method

• Proposes the Musical Chairs (MC) algorithm for the static setting, where players explore arms randomly and persist on a chosen arm if no collision occurs.
• Introduces a randomization mechanism in MC that ensures players eventually occupy distinct arms with high probability, minimizing collisions.
• Designs Dynamic Musical Chairs (DMC) for the dynamic setting, where players use a time-decaying persistence probability to adapt to changing player counts.
• Employs a collision detection mechanism that allows players to infer whether they have collided without communication.
• Uses a bounded regret analysis based on the gap between the best arm’s mean reward and the next best, assuming a fixed positive gap.
• Applies concentration inequalities and probabilistic bounds to prove high-probability regret guarantees under adversarial player behavior.

Experimental results

Research questions

• RQ1 Can a communication-free algorithm achieve constant regret in the static multi-player multi-armed bandit setting with unknown player count?
• RQ2 How can players achieve sublinear regret in the dynamic setting where players enter and exit over time, without coordination?
• RQ3 What is the impact of not knowing the number of players on regret performance in collision-based bandit problems?
• RQ4 How do existing algorithms perform in dynamic settings, and in what scenarios do they suffer linear regret?
• RQ5 Can a decentralized algorithm with no coordination still achieve optimal regret scaling under realistic assumptions?

Key findings

• The Musical Chairs (MC) algorithm achieves constant regret with high probability in the static setting, independent of the time horizon T, under a fixed reward gap.
• The Dynamic Musical Chairs (DMC) algorithm achieves Õ(√(xT)) regret with high probability in the dynamic setting, where x is the total number of player entries and exits.
• The regret of prior algorithms such as MEGA can be Ω(T) in dynamic settings, demonstrating that they fail to scale under realistic player dynamics.
• The proposed algorithms do not require knowledge of the number of players, a significant improvement over prior methods that require this information.
• Experiments validate the theoretical findings, showing that MC and DMC outperform existing algorithms in both static and dynamic regimes.
• The analysis shows that the DMC regret bound O(√(xT)) is tight and significantly improves over linear regret in scenarios where prior methods fail.