[论文解读] A General Framework to Analyze Stochastic Linear Bandit

Nima Hamidi, Mohsen Bayati|arXiv (Cornell University)|Feb 12, 2020

Advanced Bandit Algorithms Research参考文献 29被引用 15

一句话总结

本文提出了一种用于随机线性Bandit的通用框架，统一并证明了OFUL、Thompson Sampling和OLS Bandit等关键算法的速率最优性。该文提出了一种新颖的速率最优算法Sieved-Greedy（SG），其结合了不确定性复杂度与期望乐观性，实验结果显著优于现有方法。

ABSTRACT

In this paper, we study the well-known stochastic linear bandit problem where a decision-maker sequentially chooses among a set of given actions in $\mathbb{R}^d$, observes their noisy linear reward, and aims to maximize her cumulative expected reward over a horizon of length $T$. We first introduce a general family of algorithms for the problem and prove that they achieve the best-known performance (aka, are rate optimal). Our second contribution is to show that several well-known algorithms for the problem such as optimism in the face of uncertainty linear bandit (OFUL), Thompson sampling (TS), and OLS Bandit (a variant $\epsilon$-greedy) are special cases of our family of algorithms. Therefore, we obtain a unified proof of rate optimality for all of these algorithms, for both Bayesian and frequentist settings. Our new unified technique also yields a number of new results such as obtaining poly-logarithmic (in $T$) regret bounds for OFUL and TS, under a generalized gap assumption and a margin condition as in Goldenshluger and Zeevi (2013). A key component of our analysis technique is the introduction of a new notion of uncertainty complexity that directly captures the complexity of uncertainty in the action sets that we show is connected to regret analysis of any policy. Our third and most important contribution, from both theoretical and practical points of view, is the introduction of a new rate-optimal algorithm called Sieved-Greedy (SG) by combining insights from uncertainty complexity and a new (and general) notion of optimism in expectation. Specifically, SG works by filtering out the actions with relatively low uncertainty and then chooses one among the remaining actions greedily. Our empirical simulations show that SG significantly outperforms existing benchmarks by combining the best attributes of both greedy and OFUL algorithms.

研究动机与目标

• 开发一种用于随机线性Bandit的一般算法族，以实现速率最优的遗憾性能。
• 在单一理论框架下统一现有主流算法——OFUL、Thompson Sampling和OLS Bandit。
• 在广义间隙和边界条件下建立多项式对数遗憾边界。
• 提出一种新算法Sieved-Greedy（SG），通过结合低不确定性过滤与贪婪选择，提升实际性能。
• 将不确定性复杂度与遗憾分析直接关联，为策略评估提供新的理论视角。

提出的方法

• 基于一种新颖的不确定性复杂度概念，提出一种用于随机线性Bandit的一般算法族。
• 引入一种新理论框架，将不确定性复杂度与任意策略的遗憾分析相联系。
• 证明OFUL、Thompson Sampling和OLS Bandit均为所提算法族的特例。
• 通过将不确定性复杂度与期望乐观性的一般概念相结合，开发Sieved-Greedy（SG）。
• 通过移除高不确定性动作进行过滤，然后在剩余集合上应用贪婪选择。
• 利用广义间隙假设和边界条件（如Goldenshluger和Zeevi, 2013所述），推导出多项式对数遗憾边界。

实验结果

研究问题

• RQ1 能否通过单一理论框架统一并证明OFUL和Thompson Sampling等主要随机线性Bandit算法的速率最优性？
• RQ2 不确定性复杂度在刻画线性Bandit策略遗憾中的作用是什么？
• RQ3 在何种条件下可为OFUL和Thompson Sampling实现多项式对数遗憾？
• RQ4 如何形式化并结合期望乐观性与不确定性过滤，以设计一种新且更优的算法？
• RQ5 能否构造一种新算法，结合贪婪与乐观方法在线性Bandit中的优势？

主要发现

• 所提出的通用算法族实现了随机线性Bandit的速率最优遗憾性能。
• 正式证明了OFUL、Thompson Sampling和OLS Bandit均为所提框架的特例，统一了其速率最优性证明。
• 在广义间隙假设和边界条件下，为OFUL和Thompson Sampling建立了多项式对数遗憾边界。
• 提出了Sieved-Greedy（SG）作为新速率最优算法，在实验评估中显著优于现有基准。
• 新提出的不确定性复杂度概念与遗憾分析直接关联，为策略评估提供了理论基础。
• 实验结果表明，SG通过结合低不确定性过滤与贪婪选择，显著优于贪婪和OFUL-based方法。