[Paper Review] Cooperation Speeds Surfing: Use Co-Bandit!

Anuja Meetoo Appavoo, Seth Gilbert+1 more|arXiv (Cornell University)|Jan 23, 2019

Advanced Bandit Algorithms Research19 references3 citations

TL;DR

This paper proposes Co-Bandit, a cooperative multi-armed bandit algorithm that enables mobile devices to share network performance observations and delayed feedback, significantly accelerating convergence to optimal network selection. By leveraging minimal cooperation—such as periodic broadcast of observed bit rates—devices achieve stabilization up to 630x faster than EXP3 while maintaining regret-minimizing and convergence properties.

ABSTRACT

In this paper, we explore the benefit of cooperation in adversarial bandit settings. As a motivating example, we consider the problem of wireless network selection. Mobile devices are often required to choose the right network to associate with for optimal performance, which is non-trivial. The excellent theoretical properties of EXP3, a leading multi-armed bandit algorithm, suggest that it should work well for this type of problem. Yet, it performs poorly in practice. A major limitation is its slow rate of stabilization. Bandit-style algorithms perform better when global knowledge is available, i.e., when devices receive feedback about all networks after each selection. But, unfortunately, communicating full information to all devices is expensive. Therefore, we address the question of how much information is adequate to achieve better performance. We propose Co-Bandit, a novel cooperative bandit approach, that allows devices to occasionally share their observations and forward feedback received from neighbors; hence, feedback may be received with a delay. Devices perform network selection based on their own observation and feedback from neighbors. As such, they speed up each other's rate of learning. We prove that Co-Bandit is regret-minimizing and retains the convergence property of multiplicative weight update algorithms with full information. Through simulation, we show that a very small amount of information, even with a delay, is adequate to nudge each other to select the right network and yield significantly faster stabilization at the optimal state (about 630x faster than EXP3).

Motivation & Objective

• Address the poor practical performance of EXP3 in wireless network selection due to slow stabilization despite strong theoretical guarantees.
• Investigate whether minimal cooperation among devices can drastically improve convergence speed in decentralized, adversarial bandit settings.
• Design a scalable, decentralized algorithm that enables devices to learn faster by sharing observations and forwarding delayed feedback.
• Ensure the proposed method retains the regret-minimizing and convergence properties of full-information multiplicative weight algorithms.
• Evaluate the trade-off between communication overhead and performance gain in dynamic, real-world wireless environments.

Proposed method

• Devices periodically broadcast their observed network performance (e.g., bit rates) using low-overhead channels like Bluetooth.
• Each device forwards feedback received from neighbors, enabling delayed but shared learning across the network.
• Devices update their network selection strategy using a modified EXP3-like algorithm that incorporates both local observations and delayed neighbor feedback.
• The algorithm uses a weighted probability update rule based on estimated losses, with a learning rate that balances exploration and exploitation.
• Feedback propagation is modeled over a time-varying random directed graph representing dynamic device-to-device communication links.
• Theoretical analysis shows that Co-Bandit maintains regret bounds comparable to full-information algorithms, with an upper bound that accounts for cooperation and delay.

Experimental results

Research questions

• RQ1 Can minimal cooperation among devices significantly reduce the time to stabilize at the optimal network in adversarial bandit settings?
• RQ2 How does delayed feedback from neighbors affect the convergence rate and regret performance of decentralized network selection algorithms?
• RQ3 What is the minimal amount of shared information required to achieve substantial performance gains over non-cooperative bandit algorithms like EXP3?
• RQ4 Does cooperation preserve the regret-minimizing and convergence properties of full-information multiplicative weight algorithms in a decentralized, dynamic environment?
• RQ5 How does Co-Bandit scale with increasing numbers of devices and networks under realistic wireless communication patterns?

Key findings

• Co-Bandit achieves stabilization at the optimal network state approximately 630 times faster than EXP3 in simulation, despite using only a small amount of shared information.
• The algorithm maintains regret-minimizing behavior, with an upper bound on regret that depends on the level of cooperation and feedback delay.
• Even with delayed feedback, Co-Bandit preserves the convergence properties of multiplicative weight update algorithms that rely on full information.
• The system gracefully handles transient network dynamics and maintains performance under varying device and network counts.
• Minimal cooperation—such as periodic broadcast of observed bit rates—yields massive performance improvements, demonstrating that low-overhead coordination is highly effective.
• Theoretical analysis confirms that Co-Bandit converges to a stable state under adversarial conditions, with dynamics equivalent to replicator dynamics modified by a cooperation factor q.