[Paper Review] Nearly Optimal Regret for Stochastic Linear Bandits with Heavy-Tailed Payoffs

Bo Xue, Guanghui Wang+2 more|arXiv (Cornell University)|Apr 28, 2020

Advanced Bandit Algorithms Research24 references4 citations

TL;DR

This paper proposes two novel algorithms—based on median of means and dynamic truncation—for stochastic linear bandits with heavy-tailed payoffs, requiring only finite $1+\epsilon$ moments. It establishes a nearly optimal regret bound of $\widetilde{O}(d^{1/2}T^{1/(1+\epsilon)})$, matching the lower bound up to logarithmic factors when $\epsilon=1$, and validates performance via experiments on Student's $t$ and Pareto noise.

ABSTRACT

In this paper, we study the problem of stochastic linear bandits with finite action sets. Most of existing work assume the payoffs are bounded or sub-Gaussian, which may be violated in some scenarios such as financial markets. To settle this issue, we analyze the linear bandits with heavy-tailed payoffs, where the payoffs admit finite $1+ε$ moments for some $ε\in(0,1]$. Through median of means and dynamic truncation, we propose two novel algorithms which enjoy a sublinear regret bound of $\widetilde{O}(d^{\frac{1}{2}}T^{\frac{1}{1+ε}})$, where $d$ is the dimension of contextual information and $T$ is the time horizon. Meanwhile, we provide an $Ω(d^{\fracε{1+ε}}T^{\frac{1}{1+ε}})$ lower bound, which implies our upper bound matches the lower bound up to polylogarithmic factors in the order of $d$ and $T$ when $ε=1$. Finally, we conduct numerical experiments to demonstrate the effectiveness of our algorithms and the empirical results strongly support our theoretical guarantees.

Motivation & Objective

• To address the limitation of existing stochastic linear bandit algorithms that assume bounded or sub-Gaussian payoffs, which may not hold in real-world settings like financial markets.
• To develop algorithms robust to heavy-tailed payoff distributions with only finite $1+\epsilon$ moments for $\epsilon \in (0,1]$.
• To close the gap between existing upper bounds and the known lower bound for finite-armed linear bandits under heavy-tailed assumptions.
• To establish a tight regret bound that matches the information-theoretic lower bound up to logarithmic factors in $T$ and $d$.

Proposed method

• Employs the median of means technique to robustly estimate the mean payoff under heavy-tailed distributions with finite $1+\epsilon$ moments.
• Introduces dynamic truncation to adaptively control the influence of extreme payoff values in the learning process.
• Designs two distinct algorithms—one using median-of-means estimation and another using truncation—both tailored for linear bandit settings with finite action sets.
• Derives a high-probability confidence bound on the parameter estimate using robust statistics, ensuring stability under heavy-tailed noise.
• Uses a novel analysis framework combining concentration inequalities under weak moment assumptions to bound regret.
• Establishes a lower bound via a carefully constructed hard instance with $K$ arms and $d$-dimensional contexts, proving the optimality of the regret order.

Experimental results

Research questions

• RQ1 Can we achieve sublinear regret in stochastic linear bandits when payoffs have only finite $1+\epsilon$ moments, rather than assuming sub-Gaussian or bounded payoffs?
• RQ2 What is the optimal dependence of regret on the time horizon $T$ and dimension $d$ under heavy-tailed payoff assumptions?
• RQ3 Can we design algorithms that are robust to both symmetric and asymmetric heavy-tailed noise distributions in linear bandits?
• RQ4 How does the regret bound scale with $d$ and $T$ under the minimal moment assumption of $1+\epsilon$ moments?

Key findings

• The proposed algorithms achieve a regret bound of $\widetilde{O}(d^{1/2}T^{1/(1+\epsilon)})$, which improves upon prior work by an $O(\sqrt{d})$ factor for finite action sets.
• When $\epsilon = 1$, the regret bound becomes $\widetilde{O}(\sqrt{dT})$, matching the minimax lower bound up to logarithmic factors.
• An $\Omega(d^{\epsilon/(1+\epsilon)}T^{1/(1+\epsilon)})$ lower bound is established, confirming the optimality of the regret order in $T$ and near-optimality in $d$.
• Numerical experiments show that the proposed algorithms outperform existing methods—MoM, CRT, MENU, and TOFU—under both symmetric (Student’s $t$) and asymmetric (Pareto) heavy-tailed noise.
• SupBMM, the median-of-means variant, achieves the smallest regret, consistent with its tighter logarithmic factor in the theoretical bound.
• The results demonstrate that robust estimation techniques like median-of-means and dynamic truncation are effective for linear bandits under weak moment assumptions.