[Paper Review] Nonlinear stochastic multiarmed bandit problems with inexact oracle

Alexander Gasnikov, Ekaterina Krymova+3 more|arXiv (Cornell University)|Sep 5, 2015

Advanced Bandit Algorithms Research14 references3 citations

TL;DR

This paper proposes a zeroth-order stochastic online optimization method for convex and strongly convex problems using noisy function evaluations, generalizing mirror descent to handle inexact oracles. It establishes convergence rates that remain effective even under bounded noise, showing that the method achieves optimal regret bounds under mild conditions, with explicit dependence on noise level and problem dimensionality.

ABSTRACT

In the paper we consider one point and two point multiarmed bamdit problems. In other words we consider the online stochastic convex optimization problems with oracle that return the value (realization) of the function at one point or at two points. We allow these values to be inexact, but the level of noise should be small enough. We generalize well known results for inexact oracle case. And we also generalize classical results to prox-structures differ from euclidian.

Motivation & Objective

• To develop a zeroth-order online optimization method that works with noisy function evaluations instead of gradients.
• To analyze the convergence rate of such a method under bounded noise (inexact oracle) conditions.
• To determine the threshold noise level beyond which performance degrades significantly.
• To extend mirror descent to online settings with only function value feedback, including one- and two-point feedback mechanisms.
• To establish theoretical guarantees matching known lower bounds, indicating optimality of the proposed method.

Proposed method

• Uses a modified mirror descent framework where gradient estimates are replaced by function value differences (zeroth-order approximation).
• Employs a two-point feedback mechanism to improve gradient estimation accuracy compared to one-point feedback.
• Introduces a smoothing technique to handle noisy function evaluations while maintaining convergence guarantees.
• Derives convergence bounds under the assumption that the expected gradient error is bounded by a constant δ (inexact oracle condition).
• Applies the method to both convex and strongly convex settings, deriving different convergence rates for each case.
• Uses a carefully tuned step size schedule to balance exploration and convergence, ensuring optimal regret bounds.

Experimental results

Research questions

• RQ1 What is the optimal convergence rate for zeroth-order online optimization under inexact oracle conditions with bounded noise?
• RQ2 How does the performance of one-point versus two-point feedback compare in terms of regret bounds?
• RQ3 At what noise level does the presence of noise become negligible in terms of convergence rate?
• RQ4 Can mirror descent be effectively generalized to online settings with only function value feedback?
• RQ5 Are the proposed methods optimal in terms of matching known information-theoretic lower bounds?

Key findings

• The proposed method achieves a regret bound of order O(√(ln n / N)) for the one-point feedback case, matching known lower bounds up to logarithmic factors.
• For the two-point feedback case, the regret bound improves to O(ln n / N), demonstrating the benefit of additional function evaluations.
• The method remains effective even when the function evaluations are corrupted by bounded noise δ, with convergence rates depending on δ in a controlled manner.
• In the strongly convex case, the regret bound improves to O(1/N), indicating linear convergence rate.
• The theoretical analysis confirms that the method is optimal in terms of information-theoretic complexity, matching lower bounds from prior work.
• The results show that noise up to a certain level δ does not significantly affect the convergence rate, establishing a threshold beyond which performance degrades.