[Paper Review] One Sample Stochastic Frank-Wolfe

Mingrui Zhang, Zebang Shen+3 more|arXiv (Cornell University)|Oct 10, 2019

Stochastic Gradient Optimization Techniques42 references11 citations

TL;DR

This paper introduces 1-SFW, the first one-sample stochastic Frank-Wolfe algorithm that achieves optimal convergence rates—$Ó(1/\epsilon^2)$ for convex problems and $Ó(1/\epsilon^3)$ for non-convex problems—by leveraging a novel unbiased momentum estimator, eliminating the need for batch size or learning rate tuning while maintaining stability and efficiency in stochastic, projection-free optimization.

ABSTRACT

One of the beauties of the projected gradient descent method lies in its rather simple mechanism and yet stable behavior with inexact, stochastic gradients, which has led to its wide-spread use in many machine learning applications. However, once we replace the projection operator with a simpler linear program, as is done in the Frank-Wolfe method, both simplicity and stability take a serious hit. The aim of this paper is to bring them back without sacrificing the efficiency. In this paper, we propose the first one-sample stochastic Frank-Wolfe algorithm, called 1-SFW, that avoids the need to carefully tune the batch size, step size, learning rate, and other complicated hyper parameters. In particular, 1-SFW achieves the optimal convergence rate of $\mathcal{O}(1/ε^2)$ for reaching an $ε$-suboptimal solution in the stochastic convex setting, and a $(1-1/e)-ε$ approximate solution for a stochastic monotone DR-submodular maximization problem. Moreover, in a general non-convex setting, 1-SFW finds an $ε$-first-order stationary point after at most $\mathcal{O}(1/ε^3)$ iterations, achieving the current best known convergence rate. All of this is possible by designing a novel unbiased momentum estimator that governs the stability of the optimization process while using a single sample at each iteration.

Motivation & Objective

• Address the instability and poor convergence of existing stochastic Frank-Wolfe methods under single-sample gradients.
• Overcome the trade-off between sample efficiency and convergence rate in projection-free stochastic optimization.
• Develop a method that achieves optimal convergence complexity with only one stochastic sample per iteration.
• Enable efficient optimization in non-oblivious stochastic settings where the data distribution depends on the decision variable.
• Achieve state-of-the-art convergence rates across convex, non-convex, and monotone DR-submodular settings without tuning hyperparameters.

Proposed method

• Propose a novel unbiased momentum estimator to stabilize gradient approximation with only one sample per iteration.
• Design a one-sample stochastic Frank-Wolfe algorithm (1-SFW) that replaces projection with linear minimization over the constraint set.
• Introduce a recursive averaging scheme that ensures convergence despite biased gradient estimators.
• Use a diminishing step size $\eta_t = T^{-1}$ to balance convergence and stability in the non-convex regime.
• Leverage smoothness and boundedness assumptions on the objective and gradient to derive convergence bounds.
• Apply Jensen’s inequality and Cauchy-Schwarz to bound the expected suboptimality and gradient error in the analysis.

Experimental results

Research questions

• RQ1 Can a stochastic Frank-Wolfe method achieve optimal convergence rates with only one sample per iteration?
• RQ2 Can stability be preserved in stochastic Frank-Wolfe without relying on large mini-batches or variance-reduction techniques?
• RQ3 Does the proposed method maintain optimal complexity in non-oblivious stochastic settings where the data distribution depends on the decision variable?
• RQ4 Can the algorithm achieve the best-known convergence rate of $\mathcal{O}(1/\epsilon^3)$ for non-convex problems with a single sample?
• RQ5 Is it possible to eliminate hyperparameter tuning (e.g., batch size, learning rate) while maintaining optimal convergence?

Key findings

• 1-SFW achieves an $\mathcal{O}(1/\epsilon^2)$ convergence rate for stochastic convex minimization, matching the optimal complexity of batch methods.
• For monotone DR-submodular maximization, 1-SFW attains a $(1 - 1/e) - \epsilon$ approximation with $\mathcal{O}(1/\epsilon^2)$ sample complexity.
• In the non-convex setting, 1-SFW finds an $\epsilon$-first-order stationary point in $\mathcal{O}(1/\epsilon^3)$ iterations, matching the best-known rate.
• The algorithm uses only one sample per iteration and avoids tuning of batch size, step size, or learning rate.
• The convergence analysis relies on a new unbiased momentum estimator that controls bias and ensures stability.
• The method is applicable to non-oblivious stochastic optimization, where the data distribution depends on the decision variable, broadening its applicability to reinforcement learning and submodular optimization.