[Paper Review] Convergence Analysis of the Frank-Wolfe Algorithm and Its Generalization in Banach Spaces

Hong‐Kun Xu|arXiv (Cornell University)|Oct 19, 2017

Sparse and Compressive Sensing Techniques14 references3 citations

TL;DR

This paper presents a convergence analysis of the Frank-Wolfe algorithm and its generalization in Banach spaces, establishing convergence under uniformly continuous Fréchet derivatives and deriving a rate of convergence $ O(1/k^{ u}) $ for $ \nu $-Hölder continuous gradients via a novel curvature constant of order $ \sigma \in (1,2] $. The analysis covers both line minimization and open loop stepsize rules, extending prior results to broader function classes and general spaces.

ABSTRACT

The Frank-Wolfe algorithm, a very first optimization method and also known as the conditional gradient method, was introduced by Frank and Wolfe in 1956. Due to its simple linear subproblems, the Frank-Wolfe algorithm has recently been received much attention for solving large-scale structured optimization problems arising from many applied areas such as signal processing and machine learning. In this paper we will discuss in detail the convergence analysis of the Frank-Wolfe algorithm in Banach spaces. Two ways of the selections of the stepsizes are discussed: the line minimization search method and the open loop rule. In both cases, we prove the convergence of the Frank-Wolfe algorithm in the case where the objective function $f$ has uniformly continuous (on bounded sets) Fréchet derivative $f'$. We introduce the notion of the curvature constant of order $σ\in (1,2]$ and obtain the rate $O(\frac{1}{k^{σ-1}})$ of convergence of the Frank-Wolfe algorithm. In particular, this rate reduces to $O(\frac{1}{k^ν})$ if $f'$ is $ν$-Hölder continuous for $ν\in (0,1]$, and to $O(\frac{1}{k})$ if $f'$ is Lipschitz continuous. A generalized Frank-Wolfe algorithm is also introduced to address the problem of minimizing a composite objective function. Convergence of iterates of both Frank-Wolfe and generalized Frank-Wolfe algorithms are investigated.

Motivation & Objective

• To extend the convergence analysis of the Frank-Wolfe algorithm to general Banach spaces, where projections and duality maps are problematic.
• To establish convergence under weaker assumptions than Lipschitz continuity of the Fréchet derivative, specifically uniform continuity on bounded sets.
• To derive convergence rates for the Frank-Wolfe and generalized Frank-Wolfe algorithms under Hölder continuous gradients.
• To introduce and utilize the notion of curvature constant of order $ \sigma \in (1,2] $ to quantify convergence rates.
• To investigate the convergence behavior of iterates under various convexity conditions (strict, uniform, compactness).

Proposed method

• Introduces the curvature constant of order $ \sigma \in (1,2] $, generalizing the classical curvature constant to capture higher-order smoothness.
• Analyzes two stepsize strategies: line minimization search and open loop rule, both ensuring convergence under uniform continuity of $ f' $.
• Uses a telescoping inequality approach with the function decrease $ \varphi(x_{k+1}) - \varphi^* \leq (1 - \gamma_k)(\varphi(x_k) - \varphi^*) + \frac{\gamma_k^\sigma}{\sigma} C_f^{(\sigma)} $ to derive rate bounds.
• Applies Lemma 2.4 on recursive sequences to bound the error decay as $ O(1/k^{\sigma-1}) $, which reduces to $ O(1/k^\nu) $ for $ \nu $-Hölder continuous gradients.
• Generalizes the Frank-Wolfe algorithm to composite objectives $ \varphi(x) = f(x) + g(x) $ with $ f, g $ convex and lower semicontinuous.
• Establishes convergence of iterates via weak and strong convergence results under strict convexity, uniform convexity, or compactness.

Experimental results

Research questions

• RQ1 Can the Frank-Wolfe algorithm converge in Banach spaces under weaker smoothness assumptions than Lipschitz continuity of the gradient?
• RQ2 What is the convergence rate of the Frank-Wolfe algorithm when the Fréchet derivative is $ \nu $-Hölder continuous for $ \nu \in (0,1] $?
• RQ3 How does the curvature constant of order $ \sigma \in (1,2] $ relate to the convergence rate, and can it generalize existing results?
• RQ4 Under what conditions do the iterates of the Frank-Wolfe and generalized Frank-Wolfe algorithms converge strongly or weakly?
• RQ5 Can the open loop stepsize rule ensure convergence and rate guarantees in the generalized composite setting?

Key findings

• The Frank-Wolfe algorithm converges in Banach spaces when the Fréchet derivative $ f' $ is uniformly continuous on bounded sets, a weaker condition than Lipschitz continuity.
• For $ \nu $-Hölder continuous gradients, the convergence rate is $ O(1/k^\nu) $, which includes $ O(1/k) $ for Lipschitz gradients.
• The curvature constant of order $ \sigma \in (1,2] $ enables a unified rate analysis, yielding $ O(1/k^{\sigma-1}) $ convergence for $ f' $ with this smoothness.
• Under the open loop rule, the generalized Frank-Wolfe algorithm achieves $ O(1/k^{\sigma-1}) $ rate for composite objectives $ \varphi = f + g $.
• Iterates converge weakly to a solution if $ f $ is strictly convex, and strongly if $ f $ is uniformly convex or $ C $ is compact with finitely many cluster points.
• The convergence rate $ O(1/k^\nu) $ is optimal in the sense that it matches known lower bounds for first-order methods under $ \nu $-Hölder smoothness.