[Paper Review] On the Global Linear Convergence of Frank-Wolfe Optimization Variants

Simon Lacoste-Julien, Martin Jaggi|arXiv (Cornell University)|Nov 18, 2015

Sparse and Compressive Sensing TechniquesEngineering36 references160 citations

TL;DR

This paper establishes global linear convergence for multiple Frank-Wolfe variants—away-steps FW, pairwise FW, fully-corrective FW, and Wolfe’s minimum norm point—under a weaker condition than strong convexity. It proves that convergence depends on the product of the function’s condition number and a novel geometric condition number of the polytope, enabling faster, globally convergent optimization for structured machine learning problems.

ABSTRACT

The Frank-Wolfe (FW) optimization algorithm has lately re-gained popularity thanks in particular to its ability to nicely handle the structured constraints appearing in machine learning applications. However, its convergence rate is known to be slow (sublinear) when the solution lies at the boundary. A simple less-known fix is to add the possibility to take 'away steps' during optimization, an operation that importantly does not require a feasibility oracle. In this paper, we highlight and clarify several variants of the Frank-Wolfe optimization algorithm that have been successfully applied in practice: away-steps FW, pairwise FW, fully-corrective FW and Wolfe's minimum norm point algorithm, and prove for the first time that they all enjoy global linear convergence, under a weaker condition than strong convexity of the objective. The constant in the convergence rate has an elegant interpretation as the product of the (classical) condition number of the function with a novel geometric quantity that plays the role of a 'condition number' of the constraint set. We provide pointers to where these algorithms have made a difference in practice, in particular with the flow polytope, the marginal polytope and the base polytope for submodular optimization.

Motivation & Objective

• To clarify and unify the convergence behavior of major Frank-Wolfe variants used in practice.
• To establish global linear convergence for away-steps FW, pairwise FW, fully-corrective FW, and Wolfe’s minimum norm point algorithm.
• To identify a weaker sufficient condition than strong convexity for linear convergence, applicable even when the objective is not globally strongly convex.
• To introduce a novel geometric condition number for polytopes that quantifies convergence speed independently of the solution’s location.
• To demonstrate that the convergence rate constant separates cleanly into the product of the function’s condition number and the polytope’s geometric condition number.

Proposed method

• Introduces a generalized geometric strong convexity condition that weakens the requirement of classical strong convexity.
• Defines a new geometric quantity, the polytope condition number, which captures the curvature of the constraint set and influences convergence speed.
• Uses a unified analysis framework to prove linear convergence for four FW variants: AFW, PFW, FCFW, and MNP.
• Employs a gap-based analysis where suboptimality error is bounded by the square of the Frank-Wolfe gap, scaled by the generalized strong convexity constant.
• Replaces the classical strong convexity constant μ with a generalized version ˜μ_f that depends on the geometry of the active set and the solution’s relative position.
• Demonstrates that the convergence rate is affine-invariant and does not deteriorate when the optimal solution lies at the boundary of the polytope.

Experimental results

Research questions

• RQ1 Can away-steps and other FW variants achieve global linear convergence under weaker conditions than strong convexity?
• RQ2 What geometric property of the constraint set governs the linear convergence rate of Frank-Wolfe variants?
• RQ3 How does the convergence rate depend on the location of the optimal solution within the polytope?
• RQ4 Can a single theoretical framework unify the convergence analysis of multiple FW variants including fully-corrective and minimum norm point methods?
• RQ5 Is there a condition number for polytopes that plays a role analogous to the condition number of a function in determining convergence speed?

Key findings

• All four FW variants—away-steps FW, pairwise FW, fully-corrective FW, and Wolfe’s minimum norm point—achieve global linear convergence under a generalized geometric strong convexity condition.
• The linear convergence rate is governed by the product of the function’s condition number and a novel geometric condition number of the polytope, which captures the curvature of the feasible set.
• The convergence constant remains bounded away from zero even when the optimal solution lies at the boundary of the polytope, unlike previous results that deteriorate with proximity to the relative boundary.
• The generalized strong convexity condition ˜μ_f > 0 is strictly weaker than Robinson’s condition and allows linear convergence even for non-strictly convex functions or multiple global minima.
• The convergence rate is affine-invariant and does not require prior knowledge of problem-specific parameters such as the strong convexity constant or Lipschitz constant.
• The analysis provides a tight bound on suboptimality error: h_t ≤ g_t² / (2˜μ_f), which mirrors the classical case but with ˜μ_f replacing the standard μ_f^A.