[Paper Review] A Reduction from Delayed to Immediate Feedback for Online Convex Optimization with Improved Guarantees

Alexander Ryabchenko, Idan Attias+1 more|arXiv (Cornell University)|Feb 2, 2026

Advanced Bandit Algorithms Research0 citations

TL;DR

The paper introduces a continuous-time delay model and delay-to-immediate feedback reductions that achieve improved, delay-adaptive regret bounds for both online convex optimization with first-order feedback and bandit convex optimization. It unifies analysis and yields state-of-the-art or improved rates that adapt to delays and convexity.

ABSTRACT

We develop a reduction-based framework for online learning with delayed feedback that recovers and improves upon existing results for both first-order and bandit convex optimization. Our approach introduces a continuous-time model under which regret decomposes into a delay-independent learning term and a delay-induced drift term, yielding a delay-adaptive reduction that converts any algorithm for online linear optimization into one that handles round-dependent delays. For bandit convex optimization, we significantly improve existing regret bounds, with delay-dependent terms matching state-of-the-art first-order rates. For first-order feedback, we recover state-of-the-art regret bounds via a simpler, unified analysis. Quantitatively, for bandit convex optimization we obtain $O(\sqrt{d_{ ext{tot}}} + T^{\frac{3}{4}}\sqrt{k})$ regret, improving the delay-dependent term from $O(\min\{\sqrt{T d_{ ext{max}}},(Td_{ ext{tot}})^{\frac{1}{3}}\})$ in previous work to $O(\sqrt{d_{ ext{tot}}})$. Here, $k$, $T$, $d_{ ext{max}}$, and $d_{ ext{tot}}$ denote the dimension, time horizon, maximum delay, and total delay, respectively. Under strong convexity, we achieve $O(\min\{σ_{ ext{max}} \ln T, \sqrt{d_{ ext{tot}}}\} + (T^2\ln T)^{\frac{1}{3}} {k}^{\frac{2}{3}})$, improving the delay-dependent term from $O(d_{ ext{max}} \ln T)$ in previous work to $O(\min\{σ_{ ext{max}} \ln T, \sqrt{d_{ ext{tot}}}\})$, where $σ_{ ext{max}}$ denotes the maximum number of outstanding observations and may be considerably smaller than $d_{ ext{max}}$.

Motivation & Objective

• Motivate and model online learning with round-dependent delays using a continuous-time framework.
• Develop a delay-to-immediate feedback reduction that works for both first-order and bandit feedback.
• Provide delay-adaptive regret bounds that do not require advance knowledge of delay parameters.
• Achieve improved regret rates in bandit convex optimization and recover state-of-the-art bounds in first-order OCO.
• Offer a unified analysis through a drift-penalized online linear optimization (OLO) reduction.

Proposed method

• Introduce a continuous-time model where predictions and observations are events on a timeline, enabling a natural regret decomposition into non-delayed regret and prediction drift.
• Show equivalences among delay-related quantities (delays, backlogs, and their duals) and prove a regret decomposition for steady algorithms.
• Reduce delayed OCO/BCO to drift-penalized online linear optimization (OLO) via wrappers that translate feedback with delays into non-delayed updates.
• Wrap Proximal Follow-The-Regularized-Leader (P-FTRL) and Online Mirror Descent (OMD) to achieve delay-adaptive regret bounds.
• Apply single-point gradient estimation for bandit feedback within the same reduction framework.
• Incorporate an adaptive skipping scheme to further reduce delay impact when large delays occur.

Experimental results

Research questions

• RQ1 How can delayed feedback in online convex optimization be modeled and analyzed without assuming prior knowledge of delays?
• RQ2 Can delayed OCO and BCO be reduced to non-delayed problems with drift penalties, yielding adaptive regret bounds across first-order and bandit settings?
• RQ3 What are the resulting regret bounds under convex and strongly convex loss functions, and how do they depend on delay quantities like d_tot, d_max, and sigma_max?
• RQ4 Does the proposed reduction recover or improve upon state-of-the-art bounds for delayed OCO and BCO, including two-point bandit feedback?

Key findings

• For bandit convex optimization with delays, the reduction yields O(sqrt(d_tot) + T^{3/4} sqrt(k)) expected regret.
• For strongly convex bandit losses, the bound improves to O(min{sigma_max ln T, sqrt(d_tot)} + (T^2 ln T)^{1/3} k^{2/3}).
• For online convex optimization with first-order feedback, the reduction recovers O(sqrt(d_tot) + sqrt(T)) regret in convex cases.
• Under strong convexity, first-order OCO bounds become O(min{sigma_max ln T, sqrt(d_tot)} + (T^2 ln T)^{1/3} k^{2/3}).
• The framework provides delay-adaptive guarantees that do not require advance knowledge of d_max, d_tot, sigma_max, or T, and supports skipping to further reduce the delay term to O(min_Q {|Q| + (sum_{t not in Q} d_t)^{1/2}}).
• The two-point bandit feedback extension yields regret bounds of O(sqrt(d_tot) + sqrt(Tk)) in the convex case and O(min{sigma_max ln T, sqrt(d_tot)} + k ln T) under strong convexity.