[Paper Review] Contextual Bandits under Delayed Feedback.

Claire Vernade, Alexandra Carpentier+3 more|arXiv (Cornell University)|Jan 1, 2018

Advanced Bandit Algorithms Research25 references20 citations

TL;DR

This paper proposes a UCB-based contextual bandit algorithm for delayed and censored feedback, where rewards may not arrive promptly or may be lost entirely. It introduces a novel regret analysis and empirically validates the method against a practical baseline, showing improved performance under real-world feedback delays.

ABSTRACT

Delayed feedback is an ubiquitous problem in many industrial systems employing bandit algorithms. Most of those systems seek to optimize binary indicators as clicks. In that case, when the reward is not sent immediately, the learner cannot distinguish a negative signal from a not-yet-sent positive one: she might be waiting for a feedback that will never come. In this paper, we define and address the contextual bandit problem with delayed and censored feedback by providing a new UCB-based algorithm. In order to demonstrate its effectiveness, we provide a finite time regret analysis and an empirical evaluation that compares it against a baseline commonly used in practice.

Motivation & Objective

• To address the challenge of delayed and censored feedback in contextual bandit applications, common in industrial systems like online advertising.
• To enable reliable learning when rewards are not immediately available or may never be received.
• To design a UCB-based algorithm that maintains performance despite feedback delays and censoring.
• To provide a finite-time regret analysis for the proposed algorithm under delayed and censored feedback.
• To empirically evaluate the algorithm against a standard baseline used in practice.

Proposed method

• The proposed algorithm extends the Upper Confidence Bound (UCB) principle to handle delayed feedback by adjusting confidence intervals based on expected feedback arrival times.
• It models the feedback process as censored, where missing feedback is treated as potentially positive but not yet observed.
• The algorithm maintains an estimate of the expected reward and uncertainty, updating only when feedback arrives.
• It incorporates a delay-aware exploration strategy that accounts for the probability of feedback not being received.
• The regret analysis accounts for both the delay distribution and the censoring mechanism in deriving finite-time bounds.
• Empirical evaluation compares the algorithm against a standard UCB baseline in simulated and real-world settings with delayed feedback.

Experimental results

Research questions

• RQ1 How can contextual bandit algorithms be adapted to maintain performance when feedback is delayed or censored?
• RQ2 What is the theoretical impact of feedback delay and censoring on learning regret in contextual bandit settings?
• RQ3 Can a UCB-based algorithm achieve better performance than standard baselines under delayed and censored feedback?
• RQ4 How does the proposed algorithm's regret bound scale with delay and censoring rates?
• RQ5 What empirical improvements does the algorithm demonstrate in realistic feedback delay scenarios?

Key findings

• The proposed algorithm achieves a finite-time regret bound that accounts for both delay and censoring, improving theoretical guarantees under real-world feedback conditions.
• Empirical results show the algorithm outperforms a standard UCB baseline in environments with delayed and censored feedback.
• The algorithm maintains stable performance even when feedback delays are long or unpredictable.
• The regret analysis demonstrates that the algorithm's performance degrades gracefully with increasing delay and censoring rates.
• The method effectively distinguishes between unobserved feedback and negative feedback, reducing the risk of prolonged inaction due to missing signals.