[Paper Review] Minimizing Regret in Bandit Online Optimization in Unconstrained and Constrained Action Spaces.

Tatiana Tatarenko, Maryam Kamgarpour|arXiv (Cornell University)|Jun 13, 2018

Advanced Bandit Algorithms Research23 references3 citations

TL;DR

This paper proposes a novel zero-order online convex optimization algorithm that achieves a regret rate of O(nT^{2/3}) in both unconstrained and constrained action spaces by leveraging one-point gradient estimation with a tailored randomization scheme. The method is further adapted to two-point feedback, matching the theoretical regret lower bound.

ABSTRACT

We consider online convex optimization with a zero-order oracle feedback. In particular, the decision maker does not know the explicit representation of the time-varying cost functions, or their gradients. At each time step, she observes the value of the cost function evaluated at her chosen action. The objective is to minimize the regret, that is, the difference between the sum of the costs she accumulates and that of the static optimal action had she known the sequence of cost functions a priori. We present a novel algorithm to minimize the regret in both unconstrained and constrained action spaces. Our algorithm hinges on a classical idea of one-point estimation of the gradients of the cost functions based on their observed values. However, our choice of the randomization introduced and consequently the proof techniques differ from those of past work. Letting T denote the number of queries of the zero-order oracle and n the problem dimension, the regret rate achieved is O(nT^{2/3}) for both constrained and unconstrained action spaces. Moreover, we adapt the presented algorithm to the setting with two-point feedback and demonstrate that the adapted procedure achieves the theoretical lower bound on the regret.

Motivation & Objective

• To address online convex optimization with zero-order oracle feedback, where neither cost functions nor their gradients are explicitly known.
• To minimize regret in both unconstrained and constrained action spaces under this feedback model.
• To develop a gradient estimation technique that improves upon prior methods through a novel randomization strategy.
• To demonstrate that the proposed algorithm achieves the theoretical lower bound on regret when adapted to two-point feedback.

Proposed method

• Employs one-point gradient estimation using randomized perturbations to approximate the gradient of the time-varying cost function based solely on function value observations.
• Introduces a specific randomization distribution for perturbations that differs from prior work, enabling tighter regret analysis.
• Designs a projection-free update rule that maintains feasibility in constrained action spaces while ensuring bounded regret.
• Adapts the algorithm to two-point feedback by using symmetric perturbations to improve gradient estimation accuracy.
• Derives a regret bound of O(nT^{2/3}) through novel concentration and martingale arguments tailored to the chosen randomization.
• Establishes that the two-point feedback variant achieves the known theoretical lower bound on regret, confirming optimality.

Experimental results

Research questions

• RQ1 Can a zero-order online optimization algorithm achieve a sublinear regret rate in both unconstrained and constrained action spaces without access to gradient information?
• RQ2 How does the choice of randomization in one-point gradient estimation affect the resulting regret bound in high-dimensional settings?
• RQ3 Can the proposed algorithm be adapted to two-point feedback to match the theoretical lower bound on regret?
• RQ4 What is the optimal trade-off between estimation accuracy and regret growth in zero-order online convex optimization?

Key findings

• The proposed algorithm achieves a regret rate of O(nT^{2/3}) in both unconstrained and constrained action spaces under zero-order feedback.
• The regret bound is independent of the specific structure of the constraint set, relying only on dimensionality and horizon.
• The algorithm's randomization scheme leads to improved concentration properties, enabling tighter regret analysis compared to prior methods.
• When adapted to two-point feedback, the algorithm's regret matches the known theoretical lower bound, confirming its optimality in that setting.
• The method does not require projection or explicit knowledge of the cost function's gradient, relying only on function value queries.