[Paper Review] Be Aware of Non-Stationarity: Nearly Optimal Algorithms for Piecewise-Stationary Cascading Bandits.

Lingda Wang, Huozhi Zhou+3 more|arXiv (Cornell University)|Sep 12, 2019

Advanced Bandit Algorithms Research36 references3 citations

TL;DR

This paper proposes GLRT-CascadeUCB and GLRT-CascadeKL-UCB, nearly optimal algorithms for piecewise-stationary cascading bandits that use a parameter-free generalized likelihood ratio test (GLRT) to detect changes in user preferences. The algorithms achieve a regret upper bound of $\mathcal{O}(\sqrt{NLT\log T})$, matching the minimax lower bound of $\Omega(\sqrt{NLT})$ up to a logarithmic factor, demonstrating near-optimality with fewer tuning parameters and improved dependence on $L$.

ABSTRACT

Cascading bandit (CB) is a popular model for web search and online advertising, where an agent aims to learn the $K$ most attractive items out of a ground set of size $L$ during the interaction with a user. However, the stationary CB model may be too simple to apply to real-world problems, where user preferences may change over time. Considering piecewise-stationary environments, two efficient algorithms, exttt{GLRT-CascadeUCB} and exttt{GLRT-CascadeKL-UCB}, are developed and shown to ensure regret upper bounds on the order of $\mathcal{O}(\sqrt{NLT\log{T}})$, where $N$ is the number of piecewise-stationary segments, and $T$ is the number of time slots. At the crux of the proposed algorithms is an almost parameter-free change-point detector, the generalized likelihood ratio test (GLRT). Comparing with existing works, the GLRT-based algorithms: i) are free of change-point-dependent information for choosing parameters; ii) have fewer tuning parameters; iii) improve at least the $L$ dependence in regret upper bounds. In addition, we show that the proposed algorithms are optimal (up to a logarithm factor) in terms of regret by deriving a minimax lower bound on the order of $\Omega(\sqrt{NLT})$ for piecewise-stationary CB. The efficiency of the proposed algorithms relative to state-of-the-art approaches is validated through numerical experiments on both synthetic and real-world datasets.

Motivation & Objective

• Address the limitation of stationary cascading bandit models in capturing time-varying user preferences in real-world web search and online advertising.
• Develop efficient algorithms for piecewise-stationary cascading bandits that adapt to changing user preferences without requiring prior knowledge of change points.
• Reduce the number of tuning parameters compared to existing methods while improving the dependence on the item set size $L$ in regret bounds.
• Establish theoretical optimality by deriving a minimax lower bound of $\Omega(\sqrt{NLT})$ and showing the proposed algorithms nearly match this bound.
• Validate the effectiveness of the proposed algorithms through extensive experiments on synthetic and real-world datasets.

Proposed method

• Introduce a generalized likelihood ratio test (GLRT) as a change-point detector that requires no user-specified parameters or knowledge of change-point statistics.
• Design two algorithms—GLRT-CascadeUCB and GLRT-CascadeKL-UCB—by integrating GLRT with UCB and KL-UCB principles for cascading bandits.
• Use the GLRT to dynamically detect shifts in user preference distributions across time segments, triggering policy resets when changes are detected.
• Maintain confidence bounds on item attractiveness using UCB and KL-UCB formulations, adjusted after each detected change-point.
• Ensure the regret analysis accounts for both exploration within segments and detection delay across segments, leading to a tight $\mathcal{O}(\sqrt{NLT\log T})$ upper bound.
• Leverage the structure of cascading bandits, where only the first few items in a ranked list are observed, to design efficient exploration strategies under partial feedback.

Experimental results

Research questions

• RQ1 Can a parameter-free change-point detection mechanism improve the adaptability and reduce tuning burden in piecewise-stationary cascading bandits?
• RQ2 To what extent does the GLRT-based detection improve the regret dependence on the item set size $L$ compared to prior methods?
• RQ3 Are the proposed algorithms nearly optimal in terms of regret, given a minimax lower bound for the problem?
• RQ4 How do the GLRT-based algorithms perform in practice compared to state-of-the-art approaches on both synthetic and real-world data?
• RQ5 Can the GLRT effectively detect changes in user preferences without prior knowledge of the number or timing of change points?

Key findings

• The proposed GLRT-CascadeUCB and GLRT-CascadeKL-UCB algorithms achieve a regret upper bound of $\mathcal{O}(\sqrt{NLT\log T})$, which matches the minimax lower bound of $\Omega(\sqrt{NLT})$ up to a logarithmic factor, proving near-optimality.
• The GLRT-based approach eliminates the need for change-point-dependent parameter tuning, making it more practical and robust than existing methods.
• The algorithms improve the dependence on $L$ in the regret bound compared to prior work, which often scales poorly with the size of the ground set.
• Numerical experiments on synthetic and real-world datasets confirm that the proposed algorithms outperform state-of-the-art approaches in terms of regret and adaptability.
• The GLRT detector effectively identifies changes in user preferences with minimal tuning, enabling timely policy updates without prior knowledge of segment boundaries.
• The theoretical analysis confirms that the proposed algorithms are optimal up to a logarithmic factor, establishing a strong theoretical foundation for their use in non-stationary environments.