[Paper Review] An Accelerated DFO Algorithm for Finite-sum Convex Functions

Yu-Wen Chen, Antonio Orvieto+1 more|arXiv (Cornell University)|Jul 7, 2020

Stochastic Gradient Optimization Techniques4 citations

TL;DR

This paper proposes ZO-Varag, a novel derivative-free optimization algorithm for finite-sum convex functions that achieves an accelerated convergence rate by combining variance reduction with Gaussian smoothing and momentum. It proves accelerated convergence for both smooth convex and strongly convex settings, validated empirically on multiple datasets with stable performance under varying hyperparameters.

ABSTRACT

Derivative-free optimization (DFO) has recently gained a lot of momentum in machine learning, spawning interest in the community to design faster methods for problems where gradients are not accessible. While some attention has been given to the concept of acceleration in the DFO literature, existing stochastic algorithms for objective functions with a finite-sum structure have not been shown theoretically to achieve an accelerated rate of convergence. Algorithms that use acceleration in such a setting are prone to instabilities, making it difficult to reach convergence. In this work, we exploit the finite-sum structure of the objective in order to design a variance-reduced DFO algorithm that provably yields acceleration. We prove rates of convergence for both smooth convex and strongly-convex finite-sum objective functions. Finally, we validate our theoretical results empirically on several tasks and datasets.

Motivation & Objective

• To design a provably accelerated derivative-free optimization algorithm for finite-sum convex functions.
• To address the instability of momentum in stochastic DFO settings through variance reduction.
• To leverage the finite-sum structure to achieve faster convergence rates without gradient access.
• To validate theoretical convergence rates empirically on real-world datasets.

Proposed method

• Uses Gaussian smoothing to estimate gradients via function evaluations, avoiding explicit derivative computation.
• Applies variance reduction through the Varag framework, reducing gradient estimation noise.
• Incorporates momentum via a Katyusha-style update with a momentum parameter $ p = 0.5 $, ensuring theoretical stability.
• Employs a two-point stochastic estimator for gradient approximation, enabling efficient function query usage.
• Introduces a coordinate-wise variant to improve per-coordinate convergence and reduce stalling effects.
• Uses averaging strategies (Option I and II) to improve convergence, with empirical evidence favoring Option II (no averaging) in practice.

Experimental results

Research questions

• RQ1 Can a derivative-free algorithm achieve an accelerated convergence rate for finite-sum convex functions?
• RQ2 How can momentum be stabilized in stochastic DFO settings without access to gradients?
• RQ3 What role does variance reduction play in improving convergence speed and stability in DFO?
• RQ4 How do hyperparameters like smoothing parameter $ \nu $, step size, and momentum $ p $ affect convergence and final suboptimality?
• RQ5 Does the coordinate-wise variant of the algorithm outperform the standard version in practice?

Key findings

• ZO-Varag achieves an accelerated convergence rate for both smooth convex and strongly convex finite-sum functions, matching the theoretical optimal rate.
• The algorithm's convergence is stable and accelerated when using $ p = 0.5 $, as proven in the theoretical analysis.
• Empirical results show that Option II (no averaging) converges faster than Option I (averaging) on both diabetes and ijcnn1 datasets.
• Smaller smoothing parameters $ \nu $ lead to lower final suboptimality, especially in logistic regression, though the effect is negligible in ridge regression.
• The step size $ \alpha_s \gamma_s $ controls the stalling effect: reducing it decreases final suboptimality but slows convergence.
• The coordinate-wise variant performs nearly identically to ZO-Varag in practice, with only minor differences in stalling error due to a $ d $-times larger step size.