[论文解读] Online Multi-Armed Bandits with Adaptive Inference

Maria Dimakopoulou, Zhimei Ren+1 more|arXiv (Cornell University)|Feb 25, 2021

Advanced Bandit Algorithms Research被引用 14

一句话总结

本文提出了一种新型在线多臂老虎机算法——双重自适应汤普森采样（DATS），通过使用双重稳健估计器自适应重加权奖励，校正自适应数据收集带来的偏差，从而提升推理准确性。DATS在遗憾度和样本复杂度方面均优于UCB和标准汤普森采样，同时在对数因子范围内保持了最小最大遗憾最优边界。

ABSTRACT

During online decision making in Multi-Armed Bandits (MAB), one needs to conduct inference on the true mean reward of each arm based on data collected so far at each step. However, since the arms are adaptively selected--thereby yielding non-iid data--conducting inference accurately is not straightforward. In particular, sample averaging, which is used in the family of UCB and Thompson sampling (TS) algorithms, does not provide a good choice as it suffers from bias and a lack of good statistical properties (e.g. asymptotic normality). Our thesis in this paper is that more sophisticated inference schemes that take into account the adaptive nature of the sequentially collected data can unlock further performance gains, even though both UCB and TS type algorithms are optimal in the worst case. In particular, we propose a variant of TS-style algorithms--which we call doubly adaptive TS--that leverages recent advances in causal inference and adaptively reweights the terms of a doubly robust estimator on the true mean reward of each arm. Through 20 synthetic domain experiments and a semi-synthetic experiment based on data from an A/B test of a web service, we demonstrate that using an adaptive inferential scheme (while still retaining the exploration efficacy of TS) provides clear benefits in online decision making: the proposed DATS algorithm has superior empirical performance to existing baselines (UCB and TS) in terms of regret and sample complexity in identifying the best arm. In addition, we also provide a finite-time regret bound of doubly adaptive TS that matches (up to log factors) those of UCB and TS algorithms, thereby establishing that its improved practical benefits do not come at the expense of worst-case suboptimality.

研究动机与目标

• 解决在线多臂老虎机中自适应数据收集导致的均值奖励估计偏差问题。
• 在不牺牲最坏情况理论保证的前提下，提升遗憾度和样本复杂度的实证性能。
• 将因果推断领域的进展整合到在线决策算法中，以提升统计推断能力。
• 开发一种在保持探索有效性的同时，提升自适应采样下估计准确性的方法。

提出的方法

• 提出双重自适应汤普森采样（DATS），作为汤普森采样的一种变体，引入对观测奖励的自适应重加权。
• 采用双重稳健估计器，结合结果回归与逆概率加权，以减少自适应采样带来的偏差。
• 将离线因果推断估计器（如Luedtke & van der Laan, 2016；Hadad et al., 2019）适配至在线老虎机设置。
• 采用一种重加权机制，根据选择概率和结果模型动态调整每条臂奖励的影响。
• 通过汤普森采样维持探索，同时在奖励估计中校正选择偏差。
• 推导出与UCB和TS在对数因子范围内一致的有限时间遗憾边界，确保最坏情况最优性。

实验结果

研究问题

• RQ1 自适应推断方法是否能在在线多臂老虎机中实现超越标准UCB和汤普森采样的实证性能提升？
• RQ2 如何在实时奖励估计中校正自适应收集数据带来的偏差？
• RQ3 双重稳健估计能否适配至在线老虎机设置以提升推断准确性？
• RQ4 改进的推断是否能带来更低的遗憾度和更快的最优臂识别速度？
• RQ5 是否可能在不牺牲最小最大遗憾最优性的情况下实现更优的实证性能？

主要发现

• 在20组合成实验中，DATS在识别最优臂方面的遗憾度显著更低，样本复杂度更快，优于UCB和标准汤普森采样。
• 在使用真实网络服务数据的半合成A/B测试中，DATS在遗憾度和样本效率方面均优于基线方法。
• DATS的有限时间遗憾边界与UCB和汤普森采样的最优边界在对数因子范围内一致。
• 算法实现了问题依赖的遗憾边界 $O\left(\frac{\log T}{\Delta}\right)$，高概率成立。
• 建立了高概率遗憾边界 $O\left(\sqrt{T \log T}\right)$，证实了在一般条件下的鲁棒性。
• 理论分析确认，实证性能的提升并未以最坏情况次优性为代价。