[Paper Review] Greedy approximation in convex optimization

Vladimir Temlyakov|arXiv (Cornell University)|Jun 2, 2012

Sparse and Compressive Sensing Techniques15 references7 citations

TL;DR

This paper establishes a theoretical framework for using greedy approximation algorithms—originally developed in nonlinear approximation theory—for finding sparse solutions to convex optimization problems in Banach spaces. It demonstrates that the same greedy techniques, such as the Weak Chebyshev Greedy Algorithm (WCGA), can achieve convergence rates independent of dimension by leveraging smoothness properties of the objective function and duality in Banach spaces.

ABSTRACT

We study sparse approximate solutions to convex optimization problems. It is known that in many engineering applications researchers are interested in an approximate solution of an optimization problem as a linear combination of elements from a given system of elements. There is an increasing interest in building such sparse approximate solutions using different greedy-type algorithms. The problem of approximation of a given element of a Banach space by linear combinations of elements from a given system (dictionary) is well studied in nonlinear approximation theory. At a first glance the settings of approximation and optimization problems are very different. In the approximation problem an element is given and our task is to find a sparse approximation of it. In optimization theory an energy function is given and we should find an approximate sparse solution to the minimization problem. It turns out that the same technique can be used for solving both problems. We show how the technique developed in nonlinear approximation theory, in particular, the greedy approximation technique can be adjusted for finding a sparse solution of an optimization problem.

Motivation & Objective

• To bridge the gap between sparse approximation in nonlinear approximation theory and sparse solutions in convex optimization.
• To develop dimension-independent convergence guarantees for greedy algorithms in infinite-dimensional Banach spaces.
• To adapt greedy-type algorithms—originally for function approximation—to solve sparse optimization problems with respect to a redundant dictionary.
• To establish that the same algorithmic framework applies to both approximation and optimization by exploiting duality and smoothness.
• To provide convergence rates for greedy algorithms under general smoothness conditions on the objective function.

Proposed method

• Adapts the Weak Chebyshev Greedy Algorithm (WCGA) to convex optimization by using the negative gradient as a selection criterion in each greedy step.
• Uses a weakness sequence $ t_k \leq 1 $ to control the selection of dictionary elements, generalizing the standard greedy choice.
• Defines the approximant $ G_m $ as the best approximation to the current residual in the span of selected dictionary elements.
• Employs a norming functional $ F_{f^{c}_{m-1}} $ to guide the greedy selection, ensuring maximal descent in the objective function.
• Replaces the standard approximation inequality $ \|x + u y\| \geq F_x(x + u y) $ with a convexity-based inequality in optimization proofs.
• Replaces the smoothness condition $ \rho(u) \leq \gamma u^q $ with a more general inequality $ E(x + u y) - E(x) - u\langle E'(x), y \rangle \leq 2\rho(E, u\|y\|) $ for broader applicability.

Experimental results

Research questions

• RQ1 Can greedy approximation techniques from nonlinear approximation theory be effectively repurposed for sparse convex optimization?
• RQ2 What are the convergence rates of greedy algorithms for sparse optimization in infinite-dimensional Banach spaces?
• RQ3 How does the smoothness of the objective function influence the convergence behavior of greedy algorithms in optimization?
• RQ4 Under what conditions does the greedy algorithm produce a solution that is both sparse and optimal within the $ m $-term approximation class?
• RQ5 Can the same theoretical framework be applied to both approximation and optimization problems by exploiting duality?

Key findings

• Greedy algorithms such as WCGA, WRGA, and WGAFR can be adapted to convex optimization problems with convergence rates independent of the ambient dimension.
• The convergence rate of the WCGA for convex optimization depends on the modulus of smoothness $ \rho(E, u) $ of the objective function $ E $, with faster rates when $ \rho(E, u) \sim u^q $ for $ 1 < q \leq 2 $.
• For $ E(x) = \|f - x\|^q $, the greedy algorithms in optimization coincide with their counterparts in approximation theory, confirming consistency across domains.
• The use of the convex hull $ A_1(\mathcal{D}) $ as a constraint set allows for tighter convergence bounds in algorithms like WRGA(co), especially when $ E $ satisfies the inequality $ E(x + u(y - x)) - E(x) - u\langle E'(x), y - x \rangle \leq 2\rho(E, u\|y - x\|) $.
• Theoretical convergence results for greedy algorithms in optimization are derived using the same recurrence techniques as in approximation theory, with only the underlying inequality (e.g., (5.1) vs. (1.9)) differing.
• The paper shows that the boundedness of the sublevel set $ D = \{x : E(x) \leq E(0)\} $ is not preserved under renorming by $ A_1(\mathcal{D}) $, highlighting the importance of the original Banach space structure.