[Paper Review] Chebushev Greedy Algorithm in convex optimization

Vladimir Temlyakov|arXiv (Cornell University)|Dec 4, 2013

Sparse and Compressive Sensing Techniques10 references6 citations

TL;DR

This paper introduces and analyzes the E-Greedy Chebyshev Algorithm (EGCA) and Weak Chebyshev Greedy Algorithm (WCGA) for convex optimization in Banach spaces, generalizing greedy approximation techniques from Hilbert to Banach spaces. It establishes convergence rates via Lebesgue-type inequalities, showing that the error decays exponentially in the iteration count when the true solution is sparse or approximately sparse with respect to a given dictionary, under smoothness and restricted strong convexity assumptions.

ABSTRACT

Chebyshev Greedy Algorithm is a generalization of the well known Orthogonal Matching Pursuit defined in a Hilbert space to the case of Banach spaces. We apply this algorithm for constructing sparse approximate solutions (with respect to a given dictionary) to convex optimization problems. Rate of convergence results in a style of the Lebesgue-type inequalities are proved.

Motivation & Objective

• To extend greedy approximation methods from Hilbert spaces to Banach spaces for solving convex optimization problems.
• To develop a convergence theory for sparse approximate solutions using greedy algorithms in infinite-dimensional settings.
• To establish rate-of-convergence bounds in terms of the smoothness of the objective function and the sparsity of the true solution.
• To analyze the E-Greedy Chebyshev Algorithm (EGCA) and Weak Chebyshev Greedy Algorithm (WCGA) under uniform smoothness and restricted strong convexity.
• To quantify the trade-off between approximation error and iteration count using Lebesgue-type inequalities in the context of dictionary-based sparsity.

Proposed method

• Proposes the E-Greedy Chebyshev Algorithm (EGCA) and Weak Chebyshev Greedy Algorithm (WCGA) for minimizing a convex, Fréchet differentiable function $ E $ on a Banach space $ X $.
• At each iteration, selects a dictionary element $ \varphi_m \in \mathcal{D} $ that maximally decreases the directional derivative or minimizes the function along a one-dimensional subspace.
• Constructs the approximant $ G_m $ as the minimizer of $ E $ over the span of the first $ m $ selected dictionary elements.
• Employs a modulus of smoothness $ \rho(E,u) \leq \gamma u^q $ to quantify the smoothness of $ E $, with $ 1 < q \leq 2 $, and assumes restricted strong convexity with parameter $ \beta $.
• Uses a sparsity assumption: the true minimizer $ f_0 $ is within $ \epsilon $-neighborhood of a $ K $-sparse element $ f^\epsilon $, with $ f^\epsilon / A(\epsilon) \in A_1(\mathcal{D}) $.
• Derives recurrence inequalities for the error $ a_n = E(G_n) - E(f^\epsilon) $, leveraging smoothness and convexity to bound the decay rate.

Experimental results

Research questions

• RQ1 How can greedy algorithms be generalized from Hilbert spaces to Banach spaces for convex optimization?
• RQ2 What convergence rates can be achieved by the E-Greedy Chebyshev Algorithm (EGCA) and WCGA under smoothness and sparsity assumptions?
• RQ3 How does the error decay in terms of iteration count $ m $, smoothness parameter $ q $, and sparsity of the true solution?
• RQ4 What is the role of the modulus of smoothness $ \rho(E,u) $ in determining the convergence behavior of greedy algorithms?
• RQ5 Can Lebesgue-type inequalities be established for greedy algorithms in Banach spaces when the solution is approximately sparse?

Key findings

• The WCGA(co) achieves a convergence rate of $ E(G_m) - \inf E \leq C(q,\gamma,B) \left( \sum_{k=1}^m t_k^p \right)^{1-q} $ with $ p = q/(q-1) $, under $ \rho(E,u) \leq \gamma u^q $ and $ f_0 \in A_1(\mathcal{D}) $.
• For $ t_k = t \in (0,1] $, the error decays as $ O(m^{1-q}) $, which is exponential in $ m $ when $ q=2 $, under uniform smoothness.
• The EGCA(co) achieves a similar convergence rate: $ E(G_m) - E(f_0) \leq \max\left( (E(0)-E(f_0)) \exp\left(-\frac{c_1 m}{K^{2r}}\right), 8(\gamma^2/\beta)\epsilon^2 \right) + 2\gamma\epsilon^2 $, with $ c_1 = \frac{\beta}{64\gamma V^2} $.
• When the true solution $ f_0 $ is $ K $-sparse and $ \|f_0 - f^\epsilon\| \leq \epsilon $, the error decays exponentially in $ m/K^{2r} $, indicating fast convergence for sparse solutions.
• The convergence rate depends on the smoothness of $ E $, the sparsity of the solution, and the geometry of the dictionary through the constant $ V $ and $ r $ in the sparsity assumption.
• The analysis shows that the error bound is robust to $ \epsilon $-perturbations of the true solution, with the error growing quadratically in $ \epsilon $.