[Paper Review] On convexification/optimization of functionals including an l2-misfit term

Marcus Carlsson|arXiv (Cornell University)|Sep 29, 2016

Sparse and Compressive Sensing TechniquesEngineering5 references20 citations

TL;DR

This paper introduces a novel transform, $\mathcal{S}^2$, to compute the lower semi-continuous convex envelope of functionals combining a non-convex penalty (e.g., $\ell^0$-norm or rank) and an $\ell^2$-data fidelity term. It establishes that the convex envelope of $f(x) + \frac{1}{2}\|x - d\|^2$ is $\mathcal{S}^2(f)(x) + \frac{1}{2}\|x - d\|^2$, providing a unifying framework that avoids the bias of standard convex relaxations like $\ell^1$-minimization or nuclear norm minimization.

ABSTRACT

We provide theory for computing the lower semi-continuous convex envelope of functionals of the type f(x) plus an l2 misfit, and discuss applications to various non-convex optimization problems. The latter term is a data fit term whereas f provides structural constraints on x. By minimizing the above expression, possibly with additional constraints, we thus find a tradeoff between matching the measured data and enforcing a particular structure on x, such as sparsity or low rank. For these particular cases, the theory provides alternatives to convex relaxation techniques such as l1 -minimization (for vectors) and nuclear norm-minimization (for matrices). For functionals where the l2 misfit includes a singular matrix and where the convex envelope usually is not explicitly computable, we provide theory for how minimizers of (explicitly computable) approximations of the convex envelope relate to minimizers of the original functional. In particular, we give explicit conditions on when the two coincide.

Motivation & Objective

• Address the bias inherent in standard convex relaxation techniques (e.g., $\ell^1$-minimization, nuclear norm) for sparse and low-rank recovery problems.
• Provide a unifying theoretical framework for computing the lower semi-continuous convex envelope of functionals with non-convex sparsity or low-rank penalties and $\ell^2$-data fidelity.
• Overcome the limitation that the convex envelope of $\|x\|_0$ or $\mathrm{rank}(X)$ is identically zero, which renders standard convex relaxation ineffective.
• Establish conditions under which minimizers of approximations of the convex envelope coincide with minimizers of the original non-convex functional.
• Extend the theory to general Hilbert spaces and matrix-valued functionals, enabling application to diverse optimization problems in signal and image processing.

Proposed method

• Introduce the transform $\mathcal{S}^2_\gamma(f)$, where $\mathcal{S}^2_\gamma(f)(x) + \frac{\gamma}{2}\|x\|^2$ equals the l.s.c. convex envelope of $f(x) + \frac{\gamma}{2}\|x\|^2$.
• Show that the convex envelope of $f(x) + \frac{1}{2}\|x - d\|^2$ is $\mathcal{S}^2(f)(x) + \frac{1}{2}\|x - d\|^2$, with the shape of the envelope independent of the data term $d$.
• Relate $\mathcal{S}^2(f)$ to known constructs such as the Moreau envelope and Lasry-Lions approximants, enabling computational tractability.
• Provide explicit formulas for $\mathcal{S}^2(f)$ in key cases, including $f(x) = \|x\|_0$ and $f(X) = \mathrm{rank}(X)$, and generalize to weighted and constrained variants.
• Use singular value decomposition and matrix perturbation arguments to prove that the convex envelope preserves rank and sparsity structure under specific conditions.
• Establish weak lower semi-continuity of the $\ell^0$ and rank functionals in $\ell^2(\mathbb{N})$ and $\mathcal{B}_2(\mathcal{V}_1, \mathcal{V}_2)$, respectively, to support the convex envelope construction.

Experimental results

Research questions

• RQ1 Can the lower semi-continuous convex envelope of a functional combining a non-convex sparsity or low-rank penalty with an $\ell^2$-data fidelity term be explicitly computed?
• RQ2 How does the proposed $\mathcal{S}^2$ transform relate to existing convexification techniques like $\ell^1$-minimization or nuclear norm minimization?
• RQ3 Under what conditions do minimizers of the convexified functional coincide with minimizers of the original non-convex functional?
• RQ4 What is the role of the parameter $\gamma$ in the $\mathcal{S}^2_\gamma$ transform, and how does it affect the convex envelope?
• RQ5 Can the convex envelope of non-convex functionals like $\|x\|_0$ or $\mathrm{rank}(X)$ be computed meaningfully when their standard convex envelopes are trivial (zero)?

Key findings

• The lower semi-continuous convex envelope of $f(x) + \frac{1}{2}\|x - d\|^2$ is given by $\mathcal{S}^2(f)(x) + \frac{1}{2}\|x - d\|^2$, where $\mathcal{S}^2(f)$ is the $\mathcal{S}^2$ transform of $f$.
• For the $\ell^0$-norm and rank functional, $\mathcal{S}^2(f)$ provides a non-trivial convex envelope that avoids the bias of $\ell^1$- or nuclear norm-based relaxations.
• The minimizer of the convexified functional $\mathcal{S}^2(f)(x) + \frac{1}{2}\|x - d\|^2$ coincides with the minimizer of the original non-convex functional $f(x) + \frac{1}{2}\|x - d\|^2$ under explicit conditions, particularly when the data is noise-free and the solution is sparse or low-rank.
• The $\mathcal{S}^2$ transform is closely related to the Moreau envelope and Lasry-Lions approximants, enabling efficient computation via proximal algorithms.
• The $\ell^0$-norm and rank functionals are weakly lower semi-continuous in $\ell^2(\mathbb{N})$ and $\mathcal{B}_2(\mathcal{V}_1, \mathcal{V}_2)$, respectively, which supports the existence and stability of the convex envelope.
• For matrix functionals, the convex envelope preserves singular vector alignment under perturbation, and the proof relies on showing that the inner product $\langle X(s), Y \rangle$ equals the sum of products of singular values when $X(s)$ is extended piecewise affinely.