[Paper Review] Tracking the Best Expert in Non-stationary Stochastic Environments

Chen-Yu Wei, Yi-Te Hong+1 more|arXiv (Cornell University)|Dec 2, 2017

Advanced Bandit Algorithms ResearchDecision Sciences13 references16 citations

TL;DR

This paper introduces a new parameter, $Λ$, measuring total statistical variance in non-stationary stochastic environments to refine regret analysis in multi-armed bandit and full-information settings. It establishes that even with constant $Λ$, $Γ$, and $V$, bandit regret still grows with $T$, while full-information regret can be constant under constant $Γ$ and $Λ$, or exhibit $T^{1/3}$ dependence under constant $V$ and $Λ$, with matching upper and lower bounds provided.

ABSTRACT

We study the dynamic regret of multi-armed bandit and experts problem in non-stationary stochastic environments. We introduce a new parameter $Λ$, which measures the total statistical variance of the loss distributions over $T$ rounds of the process, and study how this amount affects the regret. We investigate the interaction between $Λ$ and $Γ$, which counts the number of times the distributions change, as well as $Λ$ and $V$, which measures how far the distributions deviates over time. One striking result we find is that even when $Γ$, $V$, and $Λ$ are all restricted to constant, the regret lower bound in the bandit setting still grows with $T$. The other highlight is that in the full-information setting, a constant regret becomes achievable with constant $Γ$ and $Λ$, as it can be made independent of $T$, while with constant $V$ and $Λ$, the regret still has a $T^{1/3}$ dependency. We not only propose algorithms with upper bound guarantee, but prove their matching lower bounds as well.

Motivation & Objective

• To understand how non-stationarity affects regret in stochastic online learning by introducing a new parameter $Λ$ measuring total statistical variance of loss distributions.
• To analyze the interplay between $Λ$, $Γ$ (number of distribution changes), and $V$ (total deviation of means) in determining regret bounds.
• To close the gap between existing upper bounds and lower bounds in non-stationary stochastic settings by proving matching regret lower bounds.

Proposed method

• Introduces $Λ$ as the sum of variances of loss distributions over $T$ rounds, providing a refined measure of non-stationarity beyond $Γ$ and $V$.
• Uses a reduction-based proof strategy to derive regret lower bounds by constructing adversarial loss distribution sequences that force any algorithm to incur high regret.
• Applies a conditional expectation argument to bound the expected number of times an algorithm selects a suboptimal arm in a time interval, leveraging KL divergence and concentration inequalities.
• Employs a recursive construction of loss distributions across intervals of length $B = \sqrt[3]{\Lambda T / (32K V^2)}$, ensuring bounded variance and mean drift.
• Uses Lemma 4.4 to guarantee the existence of distributions $\mathcal{P}$ and $\mathcal{Q}$ with controlled mean gap $\epsilon$, variance $\sigma^2$, and KL divergence $\leq \epsilon^2 / \sigma^2$, enabling tight regret analysis.
• Derives matching upper and lower bounds in both bandit and full-information settings by analyzing the trade-off between $\Lambda$, $V$, $\Gamma$, and $T$.

Experimental results

Research questions

• RQ1 How does the total statistical variance $\Lambda$ of loss distributions affect regret in non-stationary stochastic environments?
• RQ2 Can constant regret be achieved in the full-information setting when $\Gamma$ and $\Lambda$ are bounded, even as $T$ grows?
• RQ3 What is the fundamental limit of regret in the bandit setting when $\Gamma$, $V$, and $\Lambda$ are all constant?
• RQ4 How do the parameters $\Lambda$, $V$, and $\Gamma$ interact to shape the regret lower bound in non-stationary stochastic online learning?
• RQ5 Is the $T^{1/3}$ dependency in regret unavoidable when $V$ and $\Lambda$ are constant, even in the full-information setting?

Key findings

• In the bandit setting, even with constant $\Gamma$, $V$, and $\Lambda$, the regret lower bound grows as $\Omega(\sqrt[3]{\Lambda V T} + \sqrt{V T})$, showing that $T$-dependence persists despite bounded non-stationarity.
• In the full-information setting, constant regret is achievable when both $\Gamma$ and $\Lambda$ are bounded, making the regret independent of $T$.
• When $V$ and $\Lambda$ are constant, the regret lower bound in the full-information setting is $\Omega(\sqrt[3]{\Lambda V T})$, which matches the upper bound and shows a $T^{1/3}$ dependency is unavoidable.
• The paper proves that the $\sqrt[3]{\Lambda V T}$ term is tight in both full-information and bandit settings, with matching upper and lower bounds.
• The existence of distributions $\mathcal{P}$ and $\mathcal{Q}$ with controlled mean gap, variance, and KL divergence is formally established via Lemma 4.4, enabling the construction of hard instances for regret lower bounds.
• The analysis reveals that the bandit setting remains fundamentally more challenging than the full-information setting, as even bounded $\Lambda$, $\Gamma$, and $V$ do not eliminate $T$-dependence in regret.