[论文解读] The Externalities of Exploration and How Data Diversity Helps Exploitation

Manish Raghavan, Aleksandrs Slivkins+2 more|arXiv (Cornell University)|Jun 1, 2018

Advanced Bandit Algorithms Research被引用 15

一句话总结

本文研究了数据多样性如何影响探索-利用权衡中的利用效率，使用切尔诺夫不等式分析性能低于预期结果的概率。通过设定 γ = 1/19，表明多样化数据显著降低了表现不佳的风险，从而增强了学习系统中可靠利用的能力。

ABSTRACT

Online learning algorithms, widely used to power search and content optimization on the web, must balance exploration and exploitation, potentially sacrificing the experience of current users for information that will lead to better decisions in the future. Recently, concerns have been raised about whether the process of exploration could be viewed as unfair, placing too much burden on certain individuals or groups. Motivated by these concerns, we initiate the study of the externalities of exploration - the undesirable side effects that the presence of one party may impose on another - under the linear contextual bandits model. We introduce the notion of a group externality, measuring the extent to which the presence of one population of users impacts the rewards of another. We show that this impact can in some cases be negative, and that, in a certain sense, no algorithm can avoid it. We then study externalities at the individual level, interpreting the act of exploration as an externality imposed on the current user of a system by future users. This drives us to ask under what conditions inherent diversity in the data makes explicit exploration unnecessary. We build on a recent line of work on the smoothed analysis of the greedy algorithm that always chooses the action that currently looks optimal, improving on prior results to show that a greedy approach almost matches the best possible Bayesian regret rate of any other algorithm on the same problem instance whenever the diversity conditions hold, and that this regret is at most $ ilde{O}(T^{1/3})$. Returning to group-level effects, we show that under the same conditions, negative group externalities essentially vanish under the greedy algorithm. Together, our results uncover a sharp contrast between the high externalities that exist in the worst case, and the ability to remove all externalities if the data is sufficiently diverse.

研究动机与目标

• 理解数据多样性在探索-利用框架中提升利用效率的作用。
• 使用概率不等式量化预期结果中表现不佳的风险。
• 评估不同数据多样性水平对学习系统中次优表现可能性的影响。
• 应用切尔诺夫不等式对性能偏差的尾部概率进行建模。
• 推导数据多样性导致更可靠利用结果的条件。

提出的方法

• 本文应用切尔诺夫不等式，对性能指标 $ C_t $ 低于其期望值某一比例的概率进行建模。
• 使用特定形式 $ \Pr\left[C_{t}\leq(1-\gamma)\mathbb{E}\left[C_{t}\right]\right]\leq\exp\left(-\frac{\gamma^{2}}{2}\mathbb{E}\left[C_{t}\right]\right}) $ 来量化尾部风险。
• 选择参数 $ \gamma = 1/19 $，以评估在特定性能偏差水平下的边界值。
• 分析聚焦于尾部概率随期望性能增加而呈现的指数衰减特性。
• 该方法假设 $ C_t $ 是独立随机变量的和，从而可应用集中不等式。
• 该框架用于评估数据多样性如何降低表现不佳的利用结果的可能性。

实验结果

研究问题

• RQ1 在探索-利用设置中，数据多样性如何影响性能低于预期结果的概率？
• RQ2 特定偏差阈值（γ = 1/19）对性能损失尾部概率有何影响？
• RQ3 切尔诺夫不等式在多大程度上可用于量化因数据多样性不足导致的利用效率低下风险？
• RQ4 增加数据多样性如何降低学习系统中次优结果的可能性？
• RQ5 数据多样性与性能围绕其期望值的集中程度之间存在何种理论关系？

主要发现

• 设定 γ = 1/19 时，尾部概率边界为 $ \exp\left(-\frac{1}{722}\mathbb{E}\left[C_{t}\right]\right) $，表明表现不佳的风险呈强烈指数衰减。
• 该边界表明，随着期望性能 $ \mathbb{E}[C_t] $ 的增加，显著表现不佳的概率迅速下降。
• 分析表明，数据多样性通过收紧性能围绕其均值的集中程度，降低了表现不佳的风险。
• 切尔诺夫不等式为多样化数据如何提升学习系统可靠性提供了理论依据。
• 结果表明，数据多样性更高的系统更少可能经历显著低于预期性能的偏差。