[Paper Review] A Linear Bandit for Seasonal Environments

Giuseppe Di Benedetto, Vito Bellini+1 more|arXiv (Cornell University)|Apr 28, 2020

Advanced Bandit Algorithms Research33 references4 citations

TL;DR

This paper proposes All-Season, a contextual linear bandit algorithm that detects abrupt changes in non-stationary, seasonal environments and leverages past stationary states to improve learning efficiency. By maintaining multiple base bandit learners and dynamically weighting them based on recent performance, the method outperforms state-of-the-art algorithms in both synthetic and real-world recommendation tasks, especially under recurring seasonal patterns.

ABSTRACT

Contextual bandit algorithms are extremely popular and widely used in recommendation systems to provide online personalised recommendations. A recurrent assumption is the stationarity of the reward function, which is rather unrealistic in most of the real-world applications. In the music recommendation scenario for instance, people's music taste can abruptly change during certain events, such as Halloween or Christmas, and revert to the previous music taste soon after. We would therefore need an algorithm which can promptly react to these changes. Moreover, we would like to leverage already observed rewards collected during different stationary periods which can potentially reoccur, without the need of restarting the learning process from scratch. A growing literature has addressed the problem of reward's non-stationarity, providing algorithms that could quickly adapt to the changing environment. However, up to our knowledge, there is no algorithm which deals with seasonal changes of the reward function. Here we present a contextual bandit algorithm which detects and adapts to abrupt changes of the reward function and leverages previous estimations whenever the environment falls back to a previously observed state. We show that the proposed method can outperform state-of-the-art algorithms for non-stationary environments. We ran our experiment on both synthetic and real datasets.

Motivation & Objective

• Address the limitation of existing contextual bandit algorithms that assume stationary reward functions, which is unrealistic in real-world applications like music streaming.
• Tackle the challenge of non-stationary, seasonally recurring reward functions where reward patterns abruptly shift and reoccur over time.
• Enable efficient adaptation to changes without restarting learning from scratch, by reusing previously learned stationary states.
• Improve performance in real-world recommendation systems by detecting change points and leveraging historical data from reoccurring environments.
• Develop a robust, scalable algorithm suitable for industrial deployment in large-scale, dynamic recommendation platforms.

Proposed method

• Maintains a collection of base bandit learners, each trained on a unique stationary state of the reward function.
• Assigns dynamic weights to each base learner based on the likelihood that recent observations stem from its corresponding stationary state.
• Employs a short-term memory bandit to detect shifts toward previously unseen reward states, triggering the creation of a new base learner.
• Uses a General Bayes approach for computing posterior predictive weights to improve robustness in model misspecification.
• Integrates with any base algorithm (e.g., Linear Thompson Sampling), enabling modular and extensible design.
• Applies pruning strategies to manage computational cost, with potential improvements via Bayesian Core Sets.

Experimental results

Research questions

• RQ1 Can a contextual bandit algorithm effectively detect abrupt changes in reward functions while reusing knowledge from previously observed stationary states?
• RQ2 How does the performance of a model that leverages seasonal patterns compare to state-of-the-art non-stationary bandit algorithms in real-world and synthetic environments?
• RQ3 To what extent can the proposed method reduce regret and exploration cost by avoiding retraining from scratch when reward functions reoccur?
• RQ4 How robust is the algorithm to model misspecification and change-point detection errors in complex, real-world settings?
• RQ5 Can the integration of a short-term memory mechanism enable detection of novel reward states not previously observed?

Key findings

• All-Season (Disc) outperformed all baselines, including SW-LinTS and D-LinTS, across all settings on both MNIST and Fashion-MNIST datasets, achieving average rewards of 0.77 ± 0.01 (regular), 0.74 ± 0.02 (realistic), and 0.77 ± 0.01 (extreme) on MNIST two-arm tasks.
• On the digit-recognition task, All-Season (Disc) achieved 0.74 ± 0.01 average reward in the regular setting, significantly outperforming SW-LinTS (0.36 ± 0.00) and DenBand (0.33 ± 0.00).
• In the item-recognition task, All-Season (Disc) maintained 0.49 ± 0.02 average reward in the regular setting, surpassing SW-LinTS (0.35 ± 0.01) and BoB (0.32 ± 0.00).
• The All-Season (Disc) variant consistently outperformed All-Season (SW), indicating that the discrete weighting mechanism is more effective than sliding window-based weighting for seasonal adaptation.
• All-Season demonstrated superior robustness and stability in extreme and realistic settings, with minimal performance degradation compared to baselines.
• The algorithm achieved significant performance gains in scenarios with abrupt, recurring changes, proving effective in real-world seasonal environments like music recommendation during holidays.