[Paper Review] Online Bandits with (Biased) Offline Data: Adaptive Learning under Distribution Mismatch

Wang Chi Cheung, Lixing Lyu|arXiv (Cornell University)|May 4, 2024

Advanced Bandit Algorithms ResearchDecision Sciences3 citations

TL;DR

This paper proposes MIN-UCB, an adaptive online bandit algorithm that leverages biased offline data when distributions are sufficiently close, while ignoring them when mismatched. It establishes a regret bound that strictly improves upon UCB when offline data are informative, with tight instance-dependent and independent bounds proven via novel linear programming formulations.

ABSTRACT

Traditional online learning models are typically initialized from scratch. By contrast, contemporary real-world applications often have access to historical datasets that can potentially enhanced the online learning processes. We study how offline data can be leveraged to facilitate online learning in stochastic multi-armed bandits and combinatorial bandits. In our study, the probability distributions that govern the offline data and the online rewards can be different. We first show that, without a non-trivial upper bound on their difference, no non-anticipatory policy can outperform the classical Upper Confidence Bound (UCB) policy, even with the access to offline data. In complement, we propose an online policy MIN-UCB for multi-armed bandits. MIN-UCB outperforms the UCB when such an upper bound is available. MIN-UCB adaptively chooses to utilize the offline data when they are deemed informative, and to ignore them otherwise. We establish that MIN-UCB achieves tight regret bounds, in both instance independent and dependent settings. We generalize our approach to the combinatorial bandit setting by introducing MIN-COMB-UCB, and we provide corresponding instance dependent and instance independent regret bounds. We illustrate how various factors, such as the biases and the size of offline datasets, affect the utility of offline data in online learning. We discuss several applications and conduct numerical experiments to validate our findings.

Motivation & Objective

• To design an online bandit policy that effectively uses offline data when distributions are close, but avoids degradation when they are mismatched.
• To address the challenge of distribution shift between offline and online reward distributions in stochastic bandits.
• To provide both instance-dependent and instance-independent regret bounds that are tight under valid bias bounds.
• To establish conditions under which offline data can be safely leveraged without harming performance.

Proposed method

• Proposes MIN-UCB, an adaptive UCB-style policy that dynamically chooses whether to use offline data based on a valid bias bound $V$.
• Uses a confidence bound that incorporates both online exploration and offline data, with a thresholding mechanism to assess data relevance.
• Employs a novel linear program to characterize the optimal regret bound under distributional mismatch.
• Applies the chain rule and KL divergence bounds to derive regret lower bounds for hard instances.
• Introduces a two-instance construction to prove regret lower bounds under different distributional settings.
• Uses Gaussian bandit instances with controlled mean shifts to derive lower bounds in terms of $\sqrt{KT}$ and $T \cdot V_{\text{max}}$.

Experimental results

Research questions

• RQ1 Can an online policy outperform vanilla UCB when offline data are available but possibly biased?
• RQ2 What conditions allow offline data to improve regret, and when should they be ignored?
• RQ3 What is the tightest possible regret bound for bandit learning with biased offline data?
• RQ4 How does the regret depend on the degree of distributional mismatch between offline and online data?
• RQ5 Can both instance-dependent and instance-independent regret bounds be simultaneously tight under a valid bias bound?

Key findings

• No non-anticipatory policy can outperform UCB without additional information on distributional mismatch, establishing an impossibility result.
• MIN-UCB achieves a regret bound strictly better than UCB when the offline and online distributions are sufficiently close, as quantified by the valid bias bound $V$.
• When $P^{\text{(off)}} = P^{\text{(on)}}$, the instance-dependent regret bound of MIN-UCB matches the known optimal bound (2), proving tightness.
• The instance-independent regret bound of MIN-UCB is tight up to a logarithmic factor, matching a novel lower bound derived via a two-instance construction.
• The optimal regret bound involves solving a novel linear program, which characterizes the trade-off between exploration and exploitation under distribution shift.
• Numerical experiments confirm that MIN-UCB outperforms UCB when offline data are informative, and avoids degradation when they are not.