[论文解读] Information Directed Sampling and Bandits with Heteroscedastic Noise

Johannes Kirschner, Andreas Krause|arXiv (Cornell University)|Jan 29, 2018

Advanced Bandit Algorithms Research参考文献 16被引用 73

一句话总结

本文介绍 Information Directed Sampling (IDS) 用于具有异方差噪声的随机臂问题，推导出通过遗憾信息比率得到的频率派遗憾界限，并为线性和 RKHS 设置开发了带有在线最小二乘浓缩不等式的 IDS 变体。

ABSTRACT

In the stochastic bandit problem, the goal is to maximize an unknown function via a sequence of noisy evaluations. Typically, the observation noise is assumed to be independent of the evaluation point and to satisfy a tail bound uniformly on the domain; a restrictive assumption for many applications. In this work, we consider bandits with heteroscedastic noise, where we explicitly allow the noise distribution to depend on the evaluation point. We show that this leads to new trade-offs for information and regret, which are not taken into account by existing approaches like upper confidence bound algorithms (UCB) or Thompson Sampling. To address these shortcomings, we introduce a frequentist regret analysis framework, that is similar to the Bayesian framework of Russo and Van Roy (2014), and we prove a new high-probability regret bound for general, possibly randomized policies, which depends on a quantity we refer to as regret-information ratio. From this bound, we define a frequentist version of Information Directed Sampling (IDS) to minimize the regret-information ratio over all possible action sampling distributions. This further relies on concentration inequalities for online least squares regression in separable Hilbert spaces, which we generalize to the case of heteroscedastic noise. We then formulate several variants of IDS for linear and reproducing kernel Hilbert space response functions, yielding novel algorithms for Bayesian optimization. We also prove frequentist regret bounds, which in the homoscedastic case recover known bounds for UCB, but can be much better when the noise is heteroscedastic. Empirically, we demonstrate in a linear setting with heteroscedastic noise, that some of our methods can outperform UCB and Thompson Sampling, while staying competitive when the noise is homoscedastic.

研究动机与目标

• Motivate and formalize stochastic bandits where observation noise depends on the evaluation point (heteroscedasticity).
• Develop a frequentist regret framework analogous to Russo and Van Roy (2014) and define the regret-information ratio.
• Introduce a frequentist Information Directed Sampling (IDS) to minimize the regret-information ratio across action distributions.
• Extend concentration inequalities for online least squares to heteroscedastic noise to enable robust confidence bounds.
• Formulate IDS variants for linear and RKHS response functions and derive corresponding regret bounds and practical algorithms.

提出的方法

• Define a new regret bound for randomized policies that depends on the regret-information ratio and total information gain (gamma_T).
• Introduce a surrogate regret-information ratio Psi_t^+ using confidence bounds Delta_t^+ to enable IDS optimization.
• Prove existence and structural properties (e.g., two-action support) of minimizers for Psi_t^+(mu) and derive deterministic IDS (DIDS) as a cheaper alternative.
• Generalize online least squares concentration inequalities to heteroscedastic noise in linear and RKHS settings, leading to confidence intervals for f.
• Define two information gain functions I_t^F and I_t^UCB to drive IDS in RKHS/linear settings and relate them to mutual information and posterior variance.
• Outline algorithmic variants and provide theoretical regret bounds that recover UCB-like results in homoscedastic cases and improve under heteroscedastic noise.

实验结果

研究问题

• RQ1 How does heteroscedastic noise affect exploration-exploitation trade-offs in bandits?
• RQ2 Can a frequentist regret framework, using a regret-information ratio, bound regret under heteroscedastic noise?
• RQ3 Can Information Directed Sampling (IDS) be adapted to minimize regret while maximizing information gain in heteroscedastic settings?
• RQ4 How can online least squares concentration results be extended to heteroscedastic noise for linear and RKHS models?
• RQ5 Do IDS variants outperform UCB and Thompson Sampling in heteroscedastic linear/RKHS bandits, and how do they compare in the homoscedastic case?

主要发现

• A new high-probability regret bound for randomized policies depending on the regret-information ratio and gamma_T.
• A frequentist version of Information Directed Sampling (IDS) that minimizes a surrogate regret-information ratio using confidence bounds.
• Demonstrated that IDS variants can outperform UCB and Thompson Sampling in linear heteroscedastic settings, with competitive performance when noise is homoscedastic.
• Extended online least squares concentration inequalities to heteroscedastic noise in both finite-dimensional and RKHS settings.
• Proved that the IDS minimizer can be supported on at most two actions, aiding computational tractability in continuous action spaces.
• Recovered known UCB-type regret bounds in the homoscedastic case while offering potential improvements under heteroscedastic noise.