[论文解读] Efficient Contextual Bandits in Non-stationary Worlds

Haipeng Luo, Chen-Yu Wei+2 more|arXiv (Cornell University)|Aug 5, 2017

Advanced Bandit Algorithms Research参考文献 21被引用 77

一句话总结

The paper develops efficient contextual bandit algorithms that adapt to non-stationary environments using statistical tests, achieving near-optimal regret under various non-stationarity measures and providing a parameter-free option.

ABSTRACT

Most contextual bandit algorithms minimize regret against the best fixed policy, a questionable benchmark for non-stationary environments that are ubiquitous in applications. In this work, we develop several efficient contextual bandit algorithms for non-stationary environments by equipping existing methods for i.i.d. problems with sophisticated statistical tests so as to dynamically adapt to a change in distribution. We analyze various standard notions of regret suited to non-stationary environments for these algorithms, including interval regret, switching regret, and dynamic regret. When competing with the best policy at each time, one of our algorithms achieves regret $\mathcal{O}(\sqrt{ST})$ if there are $T$ rounds with $S$ stationary periods, or more generally $\mathcal{O}(Δ^{1/3}T^{2/3})$ where $Δ$ is some non-stationarity measure. These results almost match the optimal guarantees achieved by an inefficient baseline that is a variant of the classic Exp4 algorithm. The dynamic regret result is also the first one for efficient and fully adversarial contextual bandit. Furthermore, while the results above require tuning a parameter based on the unknown quantity $S$ or $Δ$, we also develop a parameter free algorithm achieving regret $\min\{S^{1/4}T^{3/4}, Δ^{1/5}T^{4/5}\}$. This improves and generalizes the best existing result $Δ^{0.18}T^{0.82}$ by Karnin and Anava (2016) which only holds for the two-armed bandit problem.

研究动机与目标

• Motivate the study of contextual bandits under non-stationary distributions rather than fixed-policy benchmarks.
• Develop efficient algorithms that adapt to distribution changes using statistical tests.
• Provide regret guarantees under multiple non-stationarity notions (interval, switching, dynamic).
• Compare efficiency to baseline Exp4-like methods and enable parameter-free operation.

提出的方法

• Extend i.i.d.-oriented contextual bandit methods with statistical tests to detect distribution changes.
• Derive regret guarantees for interval, switching, and dynamic non-stationary settings.
• Analyze algorithms that compete with the best policy at each time and provide near-optimal bounds.
• Develop a parameter-free variant achieving regret that adapts to unknown non-stationarity levels.

实验结果

研究问题

• RQ1 How can contextual bandit algorithms be adapted to non-stationary environments while remaining efficient?
• RQ2 What regret bounds are achievable under interval, switching, and dynamic non-stationarity?
• RQ3 Can we design a parameter-free algorithm that matches or improves upon known non-stationary regrets?
• RQ4 How does performance compare to inefficient baseline methods related to Exp4 in non-stationary contexts?

主要发现

• An algorithm competing with the best policy at each time achieves O(√(ST)) regret with S stationary periods and T rounds.
• A more general bound of O(Δ^{1/3} T^{2/3}) holds where Δ measures non-stationarity.
• The dynamic regret result yields the first efficient fully adversarial contextual bandit guarantee.
• A parameter-free algorithm attains regret Min{S^{1/4} T^{3/4}, Δ^{1/5} T^{4/5}}.
• These results nearly match the optimal guarantees of inefficient Exp4-like baselines and generalize prior results beyond two-armed bandits.]
• table_headers: []
• table_rows: []