[Paper Review] Multi-Armed Bandits with Non-Stationary Rewards

Corinna Cortes, Giulia DeSalvo+3 more|arXiv (Cornell University)|Oct 29, 2017

Advanced Bandit Algorithms Research16 references7 citations

TL;DR

This paper introduces the first theoretical analysis of multi-armed bandits with non-stationary rewards, proposing UCB-type algorithms using weighted discrepancy as a measure of non-stationarity. It achieves logarithmic regret bounds for both standard and path-dependent dynamic pseudo-regret under natural conditions, offering a unified framework for non-stationary stochastic processes.

ABSTRACT

The multi-armed bandit problem where the rewards are realizations of general non-stationary stochastic processes is a challenging setting which has not been previously tackled in the bandit literature in its full generality. We present the first theoretical analysis of this problem by deriving guarantees for both the path-dependent dynamic pseudo-regret and the standard pseudo-regret that, remarkably, are both logarithmic in the number of rounds under certain natural conditions. We describe several UCB-type algorithms based on the notion of weighted discrepancy, a key measure of non-stationarity for stochastic processes. We show that discrepancy provides a unified framework for the analysis of non-stationary rewards. Our experiments demonstrate a significant improvement in practice compared to standard benchmarks.

Motivation & Objective

• To address the gap in the bandit literature concerning general non-stationary stochastic processes in multi-armed bandit problems.
• To develop theoretical guarantees for both standard and path-dependent dynamic pseudo-regret in non-stationary settings.
• To introduce weighted discrepancy as a unifying measure of non-stationarity for stochastic processes.
• To design UCB-type algorithms that adapt to non-stationary reward distributions using this measure.
• To empirically validate the proposed algorithms against standard benchmarks, showing practical improvements.

Proposed method

• Proposes a novel measure called weighted discrepancy to quantify the degree of non-stationarity in stochastic processes.
• Designs UCB-type algorithms that incorporate weighted discrepancy to adaptively balance exploration and exploitation.
• Derives theoretical regret bounds by analyzing the growth rate of weighted discrepancy over time.
• Establishes logarithmic bounds for both standard and path-dependent dynamic pseudo-regret under mild regularity conditions on the reward processes.
• Uses time-weighted averaging to estimate the evolving mean rewards, adjusting for non-stationarity.
• Applies a confidence bound mechanism that dynamically adjusts based on the estimated discrepancy of each arm.

Experimental results

Research questions

• RQ1 Can logarithmic regret be achieved in multi-armed bandits with general non-stationary stochastic rewards?
• RQ2 How can non-stationarity in reward processes be formally measured and leveraged in algorithm design?
• RQ3 Can a unified framework be developed to analyze both standard and dynamic regret under non-stationarity?
• RQ4 Do UCB-type algorithms based on weighted discrepancy outperform standard benchmarks in practice?
• RQ5 What theoretical conditions ensure logarithmic regret in non-stationary bandit settings?

Key findings

• The proposed algorithms achieve logarithmic standard pseudo-regret under natural conditions on the reward processes.
• Path-dependent dynamic pseudo-regret is also shown to be logarithmic, marking a significant theoretical advance.
• Weighted discrepancy serves as a unifying measure that enables the analysis of diverse non-stationary reward processes.
• Theoretical guarantees are derived without assuming specific parametric forms of non-stationarity, enhancing generality.
• Experiments demonstrate consistent performance improvements over standard UCB and baseline algorithms in non-stationary environments.
• The framework successfully handles both slowly and abruptly changing reward distributions through adaptive weighting.