[Paper Review] Faster Rates for the Frank-Wolfe Method over Strongly-Convex Sets

Dan Garber, Elad Hazan|arXiv (Cornell University)|Jun 5, 2014

Sparse and Compressive Sensing Techniques24 references99 citations

TL;DR

This paper establishes that the Frank-Wolfe method achieves a faster convergence rate of $O(1/t^2)$ when optimizing over strongly convex sets, significantly improving upon the standard $O(1/t)$ rate. The method remains projection-free and leverages closed-form linear optimization over norms like $\ell_p$, Schatten, and group norms, which are shown to be strongly convex while preserving computational efficiency.

ABSTRACT

The Frank-Wolfe method (a.k.a. conditional gradient algorithm) for smooth optimization has regained much interest in recent years in the context of large scale optimization and machine learning. A key advantage of the method is that it avoids projections - the computational bottleneck in many applications - replacing it by a linear optimization step. Despite this advantage, the known convergence rates of the FW method fall behind standard first order methods for most settings of interest. It is an active line of research to derive faster linear optimization-based algorithms for various settings of convex optimization. In this paper we consider the special case of optimization over strongly convex sets, for which we prove that the vanila FW method converges at a rate of $\frac{1}{t^2}$. This gives a quadratic improvement in convergence rate compared to the general case, in which convergence is of the order $\frac{1}{t}$, and known to be tight. We show that various balls induced by $\ell_p$ norms, Schatten norms and group norms are strongly convex on one hand and on the other hand, linear optimization over these sets is straightforward and admits a closed-form solution. We further show how several previous fast-rate results for the FW method follow easily from our analysis.

Motivation & Objective

• To close the gap between projection-free methods and optimal first-order methods by achieving faster convergence rates for the Frank-Wolfe algorithm.
• To identify and characterize convex sets induced by common regularization norms (e.g., $\ell_p$, Schatten, group norms) that are strongly convex.
• To demonstrate that linear optimization over these sets remains tractable with closed-form solutions.
• To unify and generalize prior fast-rate results for Frank-Wolfe via a single theoretical framework.
• To explore whether faster convergence than $O(1/t^2)$ is achievable under strong convexity of the feasible set.

Proposed method

• Theoretical analysis proves that the vanilla Frank-Wolfe method converges at $O(1/t^2)$ when the feasible set is strongly convex, under smooth and convex objective conditions.
• The paper introduces and formalizes the notion of strong convexity of a set with respect to a norm, defining it via a curvature-like parameter.
• It derives explicit strong convexity parameters for $\ell_p$ balls ($p \in (1,2]$), Schatten $p$-norm balls, and $\ell_{s,p}$ group norm balls.
• For each class of norms, it proves that linear optimization over the corresponding ball admits a closed-form solution using singular value or row-wise soft-thresholding.
• The analysis relies on duality and norm properties, particularly leveraging Hölder’s inequality and the relationship between conjugate exponents.
• It shows that known fast-rate results for variants of Frank-Wolfe (e.g., with away steps) follow as corollaries from the main convergence theorem.

Experimental results

Research questions

• RQ1 Can the vanilla Frank-Wolfe method achieve $O(1/t^2)$ convergence when the feasible set is strongly convex, even without additional algorithmic modifications?
• RQ2 Which common regularization norms in machine learning give rise to strongly convex feasible sets while preserving efficient linear optimization?
• RQ3 Can the convergence rate of $O(1/t^2)$ be achieved without requiring strong convexity of the objective function?
• RQ4 Do existing fast-rate results for Frank-Wolfe variants (e.g., with away steps) follow from a unified analysis based on set strong convexity?
• RQ5 Is it possible to achieve logarithmic convergence rates ($O(\log(1/\epsilon))$) for Frank-Wolfe over strongly convex sets, matching projected gradient methods?

Key findings

• The Frank-Wolfe method achieves a $O(1/t^2)$ convergence rate when the feasible set is strongly convex, representing a quadratic improvement over the standard $O(1/t)$ rate.
• The $\ell_p$ ball for $p \in (1,2]$ is $\frac{p-1}{r}$-strongly convex with respect to the $\|\cdot\|_{S(p)}$ norm and $\frac{(p-1)\min(m,n)^{1/2 - 1/p}}{r}$-strongly convex with respect to the Frobenius norm.
• The Schatten $p$-norm ball $\mathbb{B}_{S(p)}(r)$ is strongly convex with respect to both the Schatten and Frobenius norms, and linear optimization over it admits a closed-form solution via SVD.
• The $\ell_{s,p}$ group norm ball $\mathbb{B}_{s,p}(r)$ is $\frac{(s-1)(p-1)}{(s+p-2)r}$-strongly convex with respect to the $\|\cdot\|_{s,p}$ norm and $n^{1/s - 1/2}m^{1/p - 1/2}\frac{(s-1)(p-1)}{(s+p-2)r}$-strongly convex with respect to the Frobenius norm.
• Linear optimization over these sets can be computed in closed form: for Schatten norms via SVD and soft-thresholding of singular values, and for group norms via row-wise thresholding.
• Theoretical results unify and generalize prior fast convergence rates for Frank-Wolfe, showing they are special cases of the main convergence theorem under strong convexity of the set.