[Paper Review] Convergence Rate of Frank-Wolfe for Non-Convex Objectives

Simon Lacoste-Julien|arXiv (Cornell University)|Jul 1, 2016

Stochastic Gradient Optimization Techniques5 references103 citations

TL;DR

The paper proves that the Frank-Wolfe algorithm attains a stationary point at a rate of O(1/√t) for non-convex objectives with Lipschitz-continuous gradients, using an affine-invariant analysis.

ABSTRACT

We give a simple proof that the Frank-Wolfe algorithm obtains a stationary point at a rate of $O(1/\sqrt{t})$ on non-convex objectives with a Lipschitz continuous gradient. Our analysis is affine invariant and is the first, to the best of our knowledge, giving a similar rate to what was already proven for projected gradient methods (though on slightly different measures of stationarity).

Motivation & Objective

• Motivate and analyze Frank-Wolfe on non-convex objectives over a convex, compact domain.
• Establish an affine-invariant convergence rate to stationarity under Lipschitz gradient assumptions.
• Provide a bound that parallels rates known for projected gradient methods.
• Clarify how the FW gap serves as a meaningful, affine-invariant stationarity measure.

Proposed method

• Define the FW gap g_t as max_{s in M} <s - x^{(t)}, -∇f(x^{(t)})> and use it as the stationarity measure.
• Show a descent inequality via the curvature constant C_f and a line-search or affine-invariant quadratic upper bound step, leading to a bound on f(x^{(t+1)}).
• Derive the rate min_{0≤k≤t} g_k ≤ max{2h_0, C_f} / √(t+1), with h_0 = f(x^{(0)}) - min_{x in M} f(x).
• Assume f has a finite curvature constant C_f over the compact convex domain M, which follows from Lipschitz gradient assumptions.
• Utilize an affine-invariant descent lemma to relate progress per iteration to the current FW gap.
• Discuss the implications of the bound and its relation to convergence of stationary points.

Experimental results

Research questions

• RQ1 What is the convergence rate of Frank-Wolfe when applied to non-convex objectives with Lipschitz gradient?
• RQ2 Can an affine-invariant analysis yield rates comparable to projected gradient methods for non-convex problems?
• RQ3 How does the Frank-Wolfe gap serve as a measure of stationarity in constrained, non-convex settings?
• RQ4 Under what conditions (curvature constant finite) can we guarantee an O(1/√t) rate for FW?
• RQ5 How do line-search and fixed-step variants compare in achieving the rate?

Key findings

• The minimal FW gap encountered after t iterations is O(1/√t).
• A concrete bound: min_{0≤k≤t} g_k ≤ max{2h_0, C} / √(t+1).
• The result applies to non-convex f with Lipschitz gradient and finite curvature constant C_f over a compact convex domain M.
• If the objective is convex on a subset of M, the FW gap upper-bounds suboptimality over that subset.
• The rate matches the order of O(1/√t) seen for projected gradient methods, within an affine-invariant framework.