[Paper Review] Affine Invariant Analysis of Frank-Wolfe on Strongly Convex Sets

Thomas Kerdreux, Lewis Liu+2 more|arXiv (Cornell University)|Nov 6, 2020

Stochastic Gradient Optimization Techniques38 references4 citations

TL;DR

This paper introduces an affine-invariant analysis of the Frank-Wolfe algorithm on strongly convex sets by proposing a new structural assumption—directional smoothness—that enables norm-independent convergence rates. It develops an affine-invariant backtracking line-search that, surprisingly, recovers optimal affine-invariant step sizes even when using non-invariant norms in smoothness estimation, resolving a long-standing inconsistency between the algorithm's affine-covariant nature and prior norm-dependent convergence bounds.

ABSTRACT

It is known that the Frank-Wolfe (FW) algorithm, which is affine-covariant, enjoys accelerated convergence rates when the constraint set is strongly convex. However, these results rely on norm-dependent assumptions, usually incurring non-affine invariant bounds, in contradiction with FW's affine-covariant property. In this work, we introduce new structural assumptions on the problem (such as the directional smoothness) and derive an affine invariant, norm-independent analysis of Frank-Wolfe. Based on our analysis, we propose an affine invariant backtracking line-search. Interestingly, we show that typical backtracking line-searches using smoothness of the objective function surprisingly converge to an affine invariant step size, despite using affine-dependent norms in the step size's computation. This indicates that we do not necessarily need to know the set's structure in advance to enjoy the affine-invariant accelerated rate.

Motivation & Objective

• To resolve the inconsistency between the affine-covariant nature of the Frank-Wolfe algorithm and existing norm-dependent convergence bounds on strongly convex sets.
• To develop a convergence analysis that is invariant under affine transformations, aligning theoretical guarantees with the algorithm's intrinsic geometric properties.
• To introduce a new structural assumption—directional smoothness—that enables affine-invariant convergence rates without prior knowledge of the set's geometry.
• To design a backtracking line-search method that yields affine-invariant step sizes despite using affine-dependent norms in smoothness estimation.
• To theoretically and empirically demonstrate that standard backtracking line-searches naturally recover affine-invariant step sizes, even when norms are not invariant.

Proposed method

• Introduces directional smoothness as a new structural assumption, defined via local smoothness constants along descent directions, which bounds the function's curvature in a directionally adaptive way.
• Derives an affine-invariant convergence rate for Frank-Wolfe on strongly convex sets using directional smoothness, achieving O(1/K²) rate under mild conditions.
• Proposes a backtracking line-search that estimates the directional smoothness constant L_loc(x_k) using the ℓ₂ norm, yet provably converges to an affine-invariant optimal step size.
• Establishes that the backtracking procedure, despite using non-invariant norms, yields step sizes that match the optimal affine-invariant rate, due to the structure of the smoothness estimation.
• Uses a novel analysis framework that avoids dependence on arbitrary norms by focusing on directional behavior and affine-invariant quantities.
• Employs theoretical bounds and experimental validation on projection and classification problems to confirm the robustness and invariance of the proposed method.

Experimental results

Research questions

• RQ1 Can an affine-invariant convergence analysis be established for Frank-Wolfe on strongly convex sets without relying on norm-dependent assumptions?
• RQ2 Does a backtracking line-search using non-invariant norms still yield affine-invariant step sizes in practice?
• RQ3 What structural assumption enables norm-independent, affine-invariant convergence rates in Frank-Wolfe for strongly convex sets?
• RQ4 Why do standard backtracking line-searches, which depend on norms, still achieve affine-invariant convergence in practice?
• RQ5 Can directional smoothness be used as a general-purpose replacement for traditional smoothness and strong convexity assumptions in Frank-Wolfe analysis?

Key findings

• The proposed directional smoothness assumption provides a norm-independent, affine-invariant characterization of function curvature, enabling improved convergence analysis.
• The backtracking line-search method based on directional smoothness estimation yields step sizes that are provably affine-invariant, even when using ℓ₂ norms in the computation.
• Empirical results show that the affine-invariant backtracking method converges faster than standard backtracking and naive FW with 1/L step size, especially in ill-conditioned settings.
• The directional smoothness constant L_loc(x) is invariant under affine transformations, and its estimate remains stable across different condition numbers of the problem's transformation matrix.
• Theoretical analysis proves that the backtracking line-search finds a step size γ⋆ satisfying min{1, 1/(2L_loc)} ≤ γ⋆, which matches the optimal affine-invariant rate.
• Experiments on projection and classification tasks confirm that the affine-invariant method maintains consistent performance under affine transformations, unlike norm-dependent alternatives.