[Paper Review] Variance-Reduced and Projection-Free Stochastic Optimization

Elad Hazan, Haipeng Luo|arXiv (Cornell University)|Feb 5, 2016

Stochastic Gradient Optimization Techniques15 references79 citations

TL;DR

This paper proposes two variance-reduced, projection-free stochastic Frank-Wolfe algorithms that significantly reduce the number of stochastic gradient evaluations needed to achieve $1 - \epsilon$ accuracy. By combining Nesterov's acceleration with variance reduction, the methods achieve $\mathcal{O}(\ln \frac{1}{\epsilon})$ stochastic gradients for smooth and strongly convex objectives and $\mathcal{O}(\frac{1}{\epsilon^{1.5}})$ for smooth Lipschitz objectives—improving over prior work that required $\mathcal{O}(\frac{1}{\epsilon})$ and $\mathcal{O}(\frac{1}{\epsilon^2})$ respectively.

ABSTRACT

The Frank-Wolfe optimization algorithm has recently regained popularity for machine learning applications due to its projection-free property and its ability to handle structured constraints. However, in the stochastic learning setting, it is still relatively understudied compared to the gradient descent counterpart. In this work, leveraging a recent variance reduction technique, we propose two stochastic Frank-Wolfe variants which substantially improve previous results in terms of the number of stochastic gradient evaluations needed to achieve $1-ε$ accuracy. For example, we improve from $O(\frac{1}ε)$ to $O(\ln\frac{1}ε)$ if the objective function is smooth and strongly convex, and from $O(\frac{1}{ε^2})$ to $O(\frac{1}{ε^{1.5}})$ if the objective function is smooth and Lipschitz. The theoretical improvement is also observed in experiments on real-world datasets for a multiclass classification application.

Motivation & Objective

• Address the lack of efficient projection-free stochastic optimization methods for large-scale machine learning problems with structured constraints.
• Improve the convergence rate of stochastic Frank-Wolfe algorithms by reducing the number of stochastic gradient evaluations required to achieve $1 - \epsilon$ accuracy.
• Leverage variance reduction and Nesterov's acceleration to achieve faster convergence while maintaining the projection-free property.
• Demonstrate theoretical and empirical improvements over existing stochastic Frank-Wolfe and projected stochastic gradient descent methods on real-world datasets.

Proposed method

• Propose STOC (Stochastic Frank-Wolfe with Variance Reduction) by integrating variance reduction into the Frank-Wolfe framework to reduce gradient variance.
• Introduce STORC (Stochastic Optimization with Recursive Correction) using Nesterov's acceleration and recursive correction to improve convergence rates.
• Use a recursive update rule that maintains a sequence of auxiliary points to track the optimal solution path, improving convergence speed.
• Apply variance reduction via control variates to stabilize gradient estimates and reduce noise in stochastic updates.
• Maintain projection-free updates by solving linear subproblems over the constraint set $\Omega$, ensuring computational efficiency.
• Bound the expected suboptimality gap using smoothness and strong convexity properties, leading to improved convergence rates.

Experimental results

Research questions

• RQ1 Can variance reduction techniques from gradient descent be effectively adapted to projection-free stochastic Frank-Wolfe optimization?
• RQ2 What is the optimal trade-off between the number of stochastic gradient evaluations and linear optimization calls in stochastic Frank-Wolfe algorithms?
• RQ3 Can Nesterov's acceleration be combined with variance reduction to achieve faster convergence in the stochastic Frank-Wolfe setting?
• RQ4 How do the proposed algorithms compare in practice to projected stochastic gradient descent and prior stochastic Frank-Wolfe methods on real-world machine learning tasks?
• RQ5 Is it possible to achieve $\mathcal{O}(\ln \frac{1}{\epsilon})$ stochastic gradient complexity for strongly convex objectives in a projection-free setting?

Key findings

• For smooth and strongly convex objectives, the proposed STORC algorithm achieves $\mathcal{O}(\ln \frac{1}{\epsilon})$ stochastic gradient evaluations, improving upon the previous $\mathcal{O}(\frac{1}{\epsilon})$ bound.
• For smooth but non-strongly convex objectives, the method reduces the stochastic gradient complexity from $\mathcal{O}(\frac{1}{\epsilon^2})$ to $\mathcal{O}(\frac{1}{\epsilon^{1.5}})$.
• The number of linear optimization calls remains $\mathcal{O}(\frac{1}{\epsilon})$, matching prior work, but the stochastic gradient complexity is significantly reduced.
• The algorithms require at most $\mathcal{O}(\ln \frac{1}{\epsilon})$ exact gradient computations, which is computationally negligible compared to other operations.
• Experiments on three large real-world datasets for multiclass classification show significant performance gains over both previous projection-free algorithms and projected stochastic gradient descent variants.
• The theoretical convergence rates are validated empirically, with the proposed methods outperforming baseline algorithms in terms of convergence speed and final accuracy.