[Paper Review] Convex optimization on Banach Spaces

Ronald DeVore, Vladimir Temlyakov|arXiv (Cornell University)|Jan 1, 2014

Sparse and Compressive Sensing Techniques14 references4 citations

TL;DR

This paper develops greedy optimization algorithms for convex minimization in general Banach spaces using only function evaluations, without requiring derivatives. It establishes a priori convergence rate bounds that depend on the smoothness of the objective function and the sparsity or compressibility of the minimizer with respect to a given dictionary, achieving exponential error decay under convexity assumptions.

ABSTRACT

Greedy algorithms which use only function evaluations are applied to convex optimization in a general Banach space $X$. Along with algorithms that use exact evaluations, algorithms with approximate evaluations are treated. A priori upper bounds for the convergence rate of the proposed algorithms are given. These bounds depend on the smoothness of the objective function and the sparsity or compressibility (with respect to a given dictionary) of a point in $X$ where the minimum is attained.

Motivation & Objective

• To develop derivative-free greedy algorithms for convex optimization in general Banach spaces.
• To establish a priori convergence rate bounds independent of dimension, relying on function evaluations only.
• To analyze how smoothness of the objective function and sparsity/compressibility of the minimizer affect convergence.
• To extend univariate convex optimization results to multivariate settings on [0,1]^d using coordinate-wise recursion.
• To provide theoretical guarantees for algorithms with both exact and approximate function evaluations.

Proposed method

• Greedy algorithms recursively build approximations using a symmetric dictionary D ⊂ X with unit norm and symmetry under negation.
• At each step, the algorithm selects a dictionary element φ_m that most significantly decreases the objective function E(G_m + αφ_m) for some α.
• The selection avoids computing the Fréchet derivative E′, relying instead on function evaluation comparisons.
• A key technique uses convexity to bound the error in function value approximations via interval reduction and error propagation.
• For multivariate problems, the algorithm applies the univariate method recursively along each coordinate, using the minimum over slices to reduce dimensionality.
• Error bounds are derived using the property that for convex f, the slope increases: (f(b)−f(a))/(b−a) ≤ (f(d)−f(c))/(d−c) for 0≤a<b≤c<d≤1.

Experimental results

Research questions

• RQ1 How can convex optimization be performed in Banach spaces using only function evaluations, without access to derivatives?
• RQ2 What convergence rate bounds can be established for such derivative-free greedy algorithms?
• RQ3 How do smoothness of the objective function and sparsity/compressibility of the minimizer influence the convergence rate?
• RQ4 Can the univariate convex optimization algorithm be extended to multivariate settings with provable error bounds?
• RQ5 What is the impact of approximate function evaluations on the convergence behavior of greedy algorithms?

Key findings

• For univariate convex functions on [0,1], the proposed algorithm achieves an error bound of 2^{-m}(4m+2)^d after (3+2m)^d function evaluations in the d-dimensional case.
• The algorithm guarantees that f(x_m) ≤ min_x f(x) + 2^{-m}(4m+2)^d, with exponential decay in m.
• In the univariate case, the algorithm uses at most 3+2m evaluations per iteration and reduces the search interval by half while incurring an error of at most 4δ per step.
• For the Lipschitz class Lip₁¹, the optimal worst-case error is 1/(4m), showing the algorithm is nearly optimal in this setting.
• The multivariate algorithm recursively applies the univariate method, leveraging convexity of marginal minima to maintain error control.
• The convergence rate depends on both the smoothness of E and the sparsity or compressibility of the minimizer x* with respect to the dictionary D.