[Paper Review] A Simple Approach for Non-stationary Linear Bandits

Peng Zhao, Lijun Zhang+2 more|arXiv (Cornell University)|Mar 9, 2021

Advanced Bandit Algorithms Research19 references30 citations

TL;DR

The paper identifies a flaw in prior analyses for non-stationary linear bandits, fixes it, and proposes a simple RestartUCB algorithm (with optional bandits-over-bandits meta-learning) achieving a dynamic regret of tetilde{O}(d^{1/2} H^{3/2} P_T + dT/{bd}H) and tetilde{O}(d^{7/8} T^{3/4} P_T^{1/4}) respectively, with or without knowing P_T.

ABSTRACT

This paper investigates the problem of non-stationary linear bandits, where the unknown regression parameter is evolving over time. Existing studies develop various algorithms and show that they enjoy an $\widetilde{\mathcal{O}}(T^{2/3}P_T^{1/3})$ dynamic regret, where $T$ is the time horizon and $P_T$ is the path-length that measures the fluctuation of the evolving unknown parameter. In this paper, we discover that a serious technical flaw makes their results ungrounded, and then present a fix, which gives an $\widetilde{\mathcal{O}}(T^{3/4}P_T^{1/4})$ dynamic regret without modifying original algorithms. Furthermore, we demonstrate that instead of using sophisticated mechanisms, such as sliding window or weighted penalty, a simple restarted strategy is sufficient to attain the same regret guarantee. Specifically, we design an UCB-type algorithm to balance exploitation and exploration, and restart it periodically to handle the drift of unknown parameters. Our approach enjoys an $\widetilde{\mathcal{O}}(T^{3/4}P_T^{1/4})$ dynamic regret. Note that to achieve this bound, the algorithm requires an oracle knowledge of the path-length $P_T$. Combining the bandits-over-bandits mechanism by treating our algorithm as the base learner, we can further achieve the same regret bound in a parameter-free way. Empirical studies also validate the effectiveness of our approach.

Motivation & Objective

• Motivate the study of non-stationary linear bandits where the unknown parameter evolves over time.
• Correct a technical flaw in prior analyses and establish valid dynamic regret guarantees.
• Propose a simple RestartUCB algorithm, with and without meta-learning, that handles drift efficiently.

Proposed method

• Introduce RestartUCB that uses epoch-based regularized least squares within each epoch.
• Derive an upper confidence bound based on a self-normalized concentration inequality and a path-length term.
• Restart the algorithm every H iterations to adapt to parameter drift, and analyze epoch-wise regret.
• Show that RestartUCB with a Bandits-over-Bandits (BOB) mechanism selects the restarting period without knowing P_T.

Experimental results

Research questions

• RQ1 What dynamic regret bound can be achieved for non-stationary linear bandits using a simple restarted UCB approach?
• RQ2 Can a parameter-free, adaptive restarting scheme attain the same regret as oracle-tuned methods?
• RQ3 Does a Bandits-over-Bandits framework remove the need for prior knowledge of P_T while preserving regret guarantees?
• RQ4 How does the proposed method compare to sliding-window and weighted approaches in terms of memory and complexity?
• RQ5 What are the implications of the identified technical flaw in prior analyses and how can it be fixed?

Key findings

• RestartUCB achieves a dynamic regret of tetilde{O}(d^{1/2} H^{3/2} P_T + dT/{bd}H) per Theorem 2.
• With optimally chosen restarting period, RestartUCB attains tetilde{O}(d^{7/8} T^{3/4} P_T^{1/4}) dynamic regret under known P_T.
• RestartUCB-BOB removes the need for prior knowledge of P_T and retains the tetilde{O}(d^{7/8} T^{3/4} P_T^{1/4}) bound as per Theorem 3.
• The paper identifies and fixes a technical gap in prior analyses, ensuring a valid bound through Lemma 1 and related results.
• Compared to WindowUCB, the RestartUCB approach uses constant memory and online updates without storing data.