[论文解读] Cooperation Speeds Surfing: Use Co-Bandit!

Anuja Meetoo Appavoo, Seth Gilbert+1 more|arXiv (Cornell University)|Jan 23, 2019

Advanced Bandit Algorithms Research参考文献 19被引用 3

一句话总结

本文提出 Co-Bandit，一种协作式多臂赌博机算法，使移动设备能够共享网络性能观测结果与延迟反馈，显著加速最优网络选择的收敛过程。通过采用最小程度的协作（如周期性广播观测到的比特率），设备实现的稳定速度比 EXP3 快达 630 倍，同时保持最小遗憾与收敛特性。

ABSTRACT

In this paper, we explore the benefit of cooperation in adversarial bandit settings. As a motivating example, we consider the problem of wireless network selection. Mobile devices are often required to choose the right network to associate with for optimal performance, which is non-trivial. The excellent theoretical properties of EXP3, a leading multi-armed bandit algorithm, suggest that it should work well for this type of problem. Yet, it performs poorly in practice. A major limitation is its slow rate of stabilization. Bandit-style algorithms perform better when global knowledge is available, i.e., when devices receive feedback about all networks after each selection. But, unfortunately, communicating full information to all devices is expensive. Therefore, we address the question of how much information is adequate to achieve better performance. We propose Co-Bandit, a novel cooperative bandit approach, that allows devices to occasionally share their observations and forward feedback received from neighbors; hence, feedback may be received with a delay. Devices perform network selection based on their own observation and feedback from neighbors. As such, they speed up each other's rate of learning. We prove that Co-Bandit is regret-minimizing and retains the convergence property of multiplicative weight update algorithms with full information. Through simulation, we show that a very small amount of information, even with a delay, is adequate to nudge each other to select the right network and yield significantly faster stabilization at the optimal state (about 630x faster than EXP3).

研究动机与目标

• 解决 EXP3 在无线网络选择中实际性能较差的问题，尽管其理论保证强大，但收敛速度缓慢。
• 探究设备间最小协作是否能在去中心化、对抗性赌博机环境中显著提升收敛速度。
• 设计一种可扩展的去中心化算法，使设备通过共享观测结果与转发延迟反馈，实现更快学习。
• 确保所提方法保留全信息乘法权重算法的最小遗憾与收敛特性。
• 评估在动态、真实无线环境中，通信开销与性能增益之间的权衡。

提出的方法

• 设备使用低开销信道（如蓝牙）周期性广播其观测到的网络性能（如比特率）。
• 每个设备转发从邻居接收到的反馈，实现在网络中延迟但共享的学习。
• 设备使用一种改进的类似 EXP3 的算法更新其网络选择策略，该算法结合了本地观测与延迟的邻居反馈。
• 该算法基于估计损失采用加权概率更新规则，学习率用于平衡探索与利用。
• 反馈传播建模为一个时变的随机有向图，表示动态的设备间通信链路。
• 理论分析表明，Co-Bandit 保持与全信息算法相当的遗憾边界，其上界考虑了协作与延迟的影响。

实验结果

研究问题

• RQ1 设备间最小协作是否能显著缩短在对抗性赌博机环境中达到最优网络状态的稳定时间？
• RQ2 来自邻居的延迟反馈如何影响去中心化网络选择算法的收敛速度与遗憾性能？
• RQ3 实现相对于非协作赌博机算法（如 EXP3）显著性能提升所需的最少共享信息量是多少？
• RQ4 协作是否在去中心化、动态环境中保留了全信息乘法权重算法的最小遗憾与收敛特性？
• RQ5 在真实无线通信模式下，Co-Bandit 在设备与网络数量增加时的可扩展性如何？

主要发现

• 在仿真中，尽管仅共享少量信息，Co-Bandit 实现最优网络状态的稳定速度约为 EXP3 的 630 倍。
• 该算法保持最小遗憾行为，遗憾的上界取决于协作程度与反馈延迟。
• 即使存在延迟反馈，Co-Bandit 仍保留了依赖全信息的乘法权重更新算法的收敛特性。
• 系统能优雅应对瞬态网络动态，在设备与网络数量变化时仍保持性能稳定。
• 最小协作（如周期性广播观测比特率）可带来巨大的性能提升，表明低开销协调极为有效。
• 理论分析证实，Co-Bandit 在对抗性条件下收敛至稳定状态，其动态特性等价于受协作因子 q 修正的复制者动态。