[Paper Review] Revisiting Frank-Wolfe for Polytopes: Strict Complementarity and Sparsity

Dan Garber|arXiv (Cornell University)|May 31, 2020

Advanced Combinatorial Mathematics28 references4 citations

TL;DR

This paper revisits the Frank-Wolfe algorithm for smooth convex minimization over polytopes, showing that standard variants with away-steps still suffer from explicit dependence on ambient dimension in worst-case scenarios. It establishes that under a strict complementarity condition, convergence becomes linear and depends only on the dimension of the optimal face, explaining empirical success in sparse problems and linking the condition to noise-robustness of optimal solutions.

ABSTRACT

In recent years it was proved that simple modifications of the classical Frank-Wolfe algorithm (aka conditional gradient algorithm) for smooth convex minimization over convex and compact polytopes, converge with linear rate, assuming the objective function has the quadratic growth property. However, the rate of these methods depends explicitly on the dimension of the problem which cannot explain their empirical success for large scale problems. In this paper we first demonstrate that already for very simple problems and even when the optimal solution lies on a low-dimensional face of the polytope, such dependence on the dimension cannot be avoided in worst case. We then revisit the addition of a strict complementarity assumption already considered in Wolfe's classical book \cite{Wolfe1970}, and prove that under this condition, the Frank-Wolfe method with away-steps and line-search converges linearly with rate that depends explicitly only on the dimension of the optimal face. We motivate strict complementarity by proving that it implies sparsity-robustness of optimal solutions to noise.

Motivation & Objective

• To explain the empirical success of Frank-Wolfe variants with away-steps in large-scale sparse problems, where convergence rates appear independent of ambient dimension.
• To demonstrate that worst-case convergence rates of standard Frank-Wolfe methods with away-steps still depend explicitly on the ambient problem dimension, even when the optimal solution lies on a low-dimensional face.
• To re-engage the classical strict complementarity condition from Wolfe’s work and show it enables dimension-independent linear convergence rates dependent only on the optimal face dimension.
• To motivate strict complementarity by proving it implies robustness of optimal solutions to deterministic noise perturbations.

Proposed method

• Introduces a worst-case construction showing that Frank-Wolfe with away-steps still requires steps scaling with ambient dimension, even when the optimal solution is a vertex (0-dimensional face).
• Revisits Wolfe’s strict complementarity condition, which ensures that the gradient at the optimal solution is strictly positive on vertices not in the optimal face.
• Proves that under strict complementarity, the Frank-Wolfe method with away-steps and line-search converges linearly with rate depending only on the dimension of the optimal face, not the ambient space.
• Uses a dual gap analysis and bounds on the norm of search directions to show that once iterates enter the optimal face, the convergence rate improves and depends only on the face’s geometry.
• Establishes that strict complementarity implies that the optimal face is robust to deterministic perturbations, meaning small noise does not shift the optimal solution to a higher-dimensional face.
• Leverages existing convergence results (e.g., Theorem 2 in Lacoste-Julien & Jaggi, 2015) to bound the dual gap and control the step size, ensuring iterates eventually enter and remain in the optimal face.

Experimental results

Research questions

• RQ1 Can the explicit dependence of Frank-Wolfe convergence rates on the ambient dimension be avoided when the optimal solution lies on a low-dimensional face of a polytope?
• RQ2 Does the classical strict complementarity condition, previously used by Wolfe and Guélat & Marcotte, lead to improved convergence rates that depend only on the optimal face dimension?
• RQ3 What is the relationship between strict complementarity and the robustness of optimal solutions to deterministic noise in sparse optimization problems?
• RQ4 Why do away-step Frank-Wolfe variants perform well in practice despite theoretical rates that scale with ambient dimension?

Key findings

• A worst-case construction demonstrates that even when the optimal solution is a vertex (0-dimensional face), Frank-Wolfe with away-steps still requires a number of iterations that scales with the ambient dimension, showing that such dependence is unavoidable in general.
• Under the strict complementarity condition, the Frank-Wolfe method with away-steps and line-search converges linearly with a rate that depends only on the dimension of the optimal face, not the ambient dimension.
• The strict complementarity condition ensures that the gradient at the optimal solution has a positive inner product with all vertices outside the optimal face, which enables the algorithm to avoid unnecessary steps and maintain iterates within the optimal face.
• The paper proves that strict complementarity implies that the optimal face is robust to deterministic perturbations: small noise does not shift the optimal solution to a higher-dimensional face, supporting its interpretation as a form of problem well-conditioning.
• The convergence rate bound is improved by replacing the global diameter bound D with the diameter of the optimal face D_F*, leading to a tighter and face-dimension-dependent rate.
• The analysis shows that once iterates enter the optimal face (after a finite number of steps), the algorithm behaves as if optimizing over a lower-dimensional polytope, enabling faster convergence.