[Paper Review] Greedy Algorithms for Cone Constrained Optimization with Convergence Guarantees

Francesco Locatello, Michael Tschannen+2 more|arXiv (Cornell University)|May 31, 2017

Sparse and Compressive Sensing Techniques20 references4 citations

TL;DR

This paper introduces novel greedy algorithms for cone-constrained optimization over the conic hull of a general atom set, providing the first convergence guarantees for non-negative Matching Pursuit (MP) methods. It establishes sublinear (O(1/t)) and linear (O(e^−t)) convergence rates for smooth convex and strongly convex objectives, respectively, with explicit constants depending on atom set geometry.

ABSTRACT

Greedy optimization methods such as Matching Pursuit (MP) and Frank-Wolfe (FW) algorithms regained popularity in recent years due to their simplicity, effectiveness and theoretical guarantees. MP and FW address optimization over the linear span and the convex hull of a set of atoms, respectively. In this paper, we consider the intermediate case of optimization over the convex cone, parametrized as the conic hull of a generic atom set, leading to the first principled definitions of non-negative MP algorithms for which we give explicit convergence rates and demonstrate excellent empirical performance. In particular, we derive sublinear ($\mathcal{O}(1/t)$) convergence on general smooth and convex objectives, and linear convergence ($\mathcal{O}(e^{-t})$) on strongly convex objectives, in both cases for general sets of atoms. Furthermore, we establish a clear correspondence of our algorithms to known algorithms from the MP and FW literature. Our novel algorithms and analyses target general atom sets and general objective functions, and hence are directly applicable to a large variety of learning settings.

Motivation & Objective

• Address the lack of convergence guarantees in existing greedy algorithms for optimization over the conic hull of atoms, a critical intermediate case between linear span and convex hull parameterizations.
• Develop principled non-negative MP variants for conic hulls that are provably convergent and applicable to general smooth convex objectives, not just quadratic ones.
• Establish explicit sublinear and linear convergence rates for the proposed algorithms, depending on the geometry of the atom set and objective smoothness/strong convexity.
• Bridge the theoretical gap between Frank-Wolfe (convex hull) and Matching Pursuit (linear span) by introducing algorithms for the conic hull case with unified convergence analysis.
• Enable broader applicability to real-world problems such as non-negative matrix factorization, unmixing, and sparse modeling by removing assumptions on finite atom sets or specific objective forms.

Proposed method

• Propose a non-orthogonal MP algorithm for minimizing smooth convex functions over the conic hull of a general atom set, using a linear minimization oracle (LMO) to select atoms in each iteration.
• Introduce away-step, pairwise, and fully corrective MP variants inspired by Frank-Wolfe and generalized MP, enabling weight corrections for previously selected atoms to improve convergence.
• Define a new curvature-like quantity, $\mu_{f,\rho\mathcal{A}}^{A}$, to characterize the progress per iteration, which depends on the geometry of the atom set and the gradient behavior.
• Derive convergence rates using a novel analysis framework that relates the duality gap to the progress per iteration via the curvature parameter and smoothness constants.
• Use an approximate LMO with a multiplicative error $\delta$ to allow for practical implementation while maintaining convergence guarantees.
• Establish affine-invariant smoothness and strong convexity notions to ensure rate bounds are independent of norm choice and robust to scaling.

Experimental results

Research questions

• RQ1 Can greedy algorithms for optimization over the conic hull of a general atom set be designed with theoretical convergence guarantees, even when the atom set is infinite or not aligned with descent directions?
• RQ2 What convergence rates can be achieved for non-negative MP algorithms on smooth convex and strongly convex objectives over conic hulls, and how do they depend on the atom set geometry?
• RQ3 How do the proposed MP variants (non-orthogonal, away-step, pairwise, fully corrective) compare in convergence speed and stability to existing Frank-Wolfe and Matching Pursuit methods in the conic hull setting?
• RQ4 Can the convergence analysis be extended to general smooth convex functions without restricting to quadratic objectives, as in prior non-negative MP work?
• RQ5 What is the role of the atom set's diameter, radius, and C-width in determining the convergence rate of the proposed algorithms?

Key findings

• The proposed non-orthogonal MP algorithm achieves a sublinear convergence rate of $\mathcal{O}(1/t)$ for smooth convex objectives, with explicit constants depending on the atom set's geometry and the objective's smoothness.
• For strongly convex objectives, the algorithm achieves linear convergence at rate $\mathcal{O}(e^{-t})$, with the rate constant depending on the strong convexity parameter and the C-width of the atom set.
• The away-step and fully corrective MP variants also achieve sublinear and linear convergence rates, respectively, under the same assumptions, with convergence bounds derived via the same curvature parameter $\mu_{f,\rho\mathcal{A}}^{A}$.
• The convergence rates are independent of the norm used, relying on affine-invariant notions of smoothness and strong convexity, ensuring robustness to scaling.
• The analysis recovers known convergence rates from generalized MP and Frank-Wolfe when the conic hull coincides with the linear span or convex hull, demonstrating consistency across frameworks.
• The proposed algorithms maintain the same computational complexity as standard MP, with no additional cost for achieving convergence guarantees.