[论文解读] A Contextual Bandit Approach for Stream-Based Active Learning

Linqi Song, Jie Xu|arXiv (Cornell University)|Jan 24, 2017

Machine Learning and Algorithms参考文献 8被引用 4

一句话总结

本文提出了一种具备主动学习能力的上下文Bandit算法，通过向标注者传输先验奖励信息来降低查询成本。通过自适应地划分上下文空间和动作空间，并仅在必要时战略性地查询真实标签，该算法在奖励获取成本较高的情况下仍能实现与传统上下文Bandit算法相当的遗憾性能，证明了尽管存在主动学习的开销，其学习效率仍为阶最优。

ABSTRACT

Contextual bandit algorithms -- a class of multi-armed bandit algorithms that exploit the contextual information -- have been shown to be effective in solving sequential decision making problems under uncertainty. A common assumption adopted in the literature is that the realized (ground truth) reward by taking the selected action is observed by the learner at no cost, which, however, is not realistic in many practical scenarios. When observing the ground truth reward is costly, a key challenge for the learner is how to judiciously acquire the ground truth by assessing the benefits and costs in order to balance learning efficiency and learning cost. From the information theoretic perspective, a perhaps even more interesting question is how much efficiency might be lost due to this cost. In this paper, we design a novel contextual bandit-based learning algorithm and endow it with the active learning capability. The key feature of our algorithm is that in addition to sending a query to an annotator for the ground truth, prior information about the ground truth learned by the learner is sent together, thereby reducing the query cost. We prove that by carefully choosing the algorithm parameters, the learning regret of the proposed algorithm achieves the same order as that of conventional contextual bandit algorithms in cost-free scenarios, implying that, surprisingly, cost due to acquiring the ground truth does not increase the learning regret in the long-run. Our analysis shows that prior information about the ground truth plays a critical role in improving the system performance in scenarios where active learning is necessary.

研究动机与目标

• 为解决在序列决策系统中获取真实奖励成本过高的挑战。
• 设计一种将主动学习与先验信息共享相结合的上下文Bandit框架，以降低查询开销。
• 证明所提方法的遗憾性能与无成本场景下传统上下文Bandit算法的阶数量级相当。
• 在大规模上下文空间和动作空间中平衡探索与利用，同时最小化主动学习成本。

提出的方法

• 该算法将时间划分为多个周期，并自适应地将上下文空间和动作空间划分为簇，且簇的粒度随时间逐渐细化。
• 在每个周期内，算法探索动作簇以估计上下文簇的奖励，并移除次优簇。
• 采用控制函数将动作簇分类为最优、近似最优或次优，以指导探索与利用。
• 仅在必要时向标注者发送查询，并包含先验奖励估计，以降低标注者的工作量和查询成本。
• 算法利用统计界和基于置信区间的停止规则，判断何时停止查询。
• 利用覆盖维数理论来限制上下文和动作簇的数量，确保可扩展性。

实验结果

研究问题

• RQ1 在上下文Bandit设置中，结合先验信息的主动学习是否能在不增加遗憾的情况下降低查询成本？
• RQ2 在需付出代价主动获取真实标签的情况下，上下文Bandit算法的理论遗憾阶是多少？
• RQ3 关于奖励分布的先验信息如何影响序列决策中主动学习的效率？
• RQ4 在真实标签获取成本较高的情况下，所提算法能否保持阶最优遗憾？
• RQ5 对上下文和动作空间的自适应划分如何影响学习效率与成本？

主要发现

• 所提CB-AL算法的遗憾为O(T^(dX + dK + 1)/(dX + dK + 2))阶，与无成本场景下传统上下文Bandit算法的最优遗憾阶一致。
• 该遗憾界在正常与异常事件条件下均成立，通过Jensen不等式和级数收敛性分析对查询成本进行了有界处理。
• 当查询成本c从0.1增至1.0时，在T = 10,000时平均收益下降15%，在T = 20,000时下降8%，表明对成本敏感，但不影响遗憾阶。
• 在实验结束时，该算法相比标准上下文Bandit算法收益高出16%，相比无先验信息的CB-AL算法高出13%。
• 遗憾的下界为Ω(T^(dX + dK + 1)/(dX + dK + 2))，证实该算法为阶最优。
• 先验信息的使用显著降低了查询成本，如ρX,i和ρK,i所涉及的有界查询成本项所示，其规模与覆盖维数成比例。