[Paper Review] Heteroscedastic Bandits with Reneging.

Ping-Chun Hsieh, Xi Liu+2 more|arXiv (Cornell University)|Oct 29, 2018

Advanced Bandit Algorithms Research28 references4 citations

TL;DR

This paper proposes a novel heteroscedastic contextual bandit model with reneging, where users may permanently leave if rewards fall below their individual acceptance thresholds and reward variance depends on context. It introduces the HR-UCB algorithm, proving high-probability regret of $\mathcal{O}\Big(\sqrt{{T}}\big(\log({T})\big)^{3/2}\Big)$, addressing real-world limitations in user dropout and heteroscedastic rewards.

ABSTRACT

Although shown to be useful in many areas as models for solving sequential decision problems with side observations (contexts), contextual bandits are subject to two major limitations. First, they neglect user that occurs in real-world applications. That is, users unsatisfied with an interaction quit future interactions forever. Second, they assume that the reward distribution is homoscedastic, which is often invalidated by real-world datasets, e.g., datasets from finance. We propose a novel model of heteroscedastic contextual bandits with reneging to overcome the two limitations. Our model allows each user to have a distinct acceptance level, with any interaction falling short of that level resulting in that user reneging. It also allows the variance to be a function of context. We develop a UCB-type of policy, called HR-UCB, and prove that with high probability it achieves $\mathcal{O}\Big(\sqrt{{T}}\big(\log({T})\big)^{3/2}\Big)$ regret.

Motivation & Objective

• To address the limitation of contextual bandits in modeling user reneging, where users permanently leave after unsatisfactory interactions.
• To overcome the assumption of homoscedastic rewards by allowing variance to depend on context.
• To develop a UCB-type algorithm that handles both user-specific acceptance levels and context-dependent variance.
• To theoretically bound the regret of the proposed policy under these realistic conditions.

Proposed method

• Model users with individual acceptance levels; any reward below this level triggers permanent reneging.
• Allow reward variance to be a context-dependent function, capturing heteroscedasticity in real-world data.
• Design the HR-UCB algorithm using upper confidence bounds tailored to the heteroscedastic and reneging structure.
• Incorporate context-aware confidence intervals that account for varying reward variances.
• Use high-probability concentration bounds to control uncertainty in the presence of reneging and heteroscedasticity.
• Prove regret bound by analyzing the cumulative effect of suboptimal actions under the model's constraints.

Experimental results

Research questions

• RQ1 How can contextual bandit models be extended to handle user reneging due to dissatisfaction?
• RQ2 What is the impact of context-dependent reward variance on regret in sequential decision-making?
• RQ3 Can a UCB-based policy be designed to maintain low regret under both heteroscedastic rewards and user dropout?
• RQ4 What theoretical regret bound can be achieved under these combined constraints?
• RQ5 How do individual user acceptance thresholds affect long-term learning performance?

Key findings

• The HR-UCB algorithm achieves a high-probability regret bound of $\mathcal{O}\Big(\sqrt{{T}}\big(\log({T})\big)^{3/2}\Big)$, which is sublinear and nearly optimal.
• The model successfully captures real-world phenomena: user reneging and heteroscedastic reward variance.
• The regret bound accounts for both context-dependent variance and user-specific acceptance thresholds.
• The theoretical analysis confirms that the algorithm maintains performance even when users drop out after poor interactions.
• The proposed method improves upon standard contextual bandits by incorporating realistic behavioral and statistical constraints.