[Paper Review] Fast Learning with Nonconvex L1-2 Regularization

Quanming Yao, James T. Kwok+1 more|arXiv (Cornell University)|Oct 29, 2016

Sparse and Compressive Sensing Techniques22 references3 citations

TL;DR

This paper proposes a closed-form proximal gradient solution for the nonconvex ℓ₁₋₂ regularizer, enabling fast and efficient optimization in sparse and low-rank learning. By deriving an analytical proximal step, the method achieves superior convergence speed and recovery performance over state-of-the-art algorithms in compressed sensing, matrix completion, and image denoising.

ABSTRACT

Convex regularizers are often used for sparse learning. They are easy to optimize, but can lead to inferior prediction performance. The difference of $\ell_1$ and $\ell_2$ ($\ell_{1-2}$) regularizer has been recently proposed as a nonconvex regularizer. It yields better recovery than both $\ell_0$ and $\ell_1$ regularizers on compressed sensing. However, how to efficiently optimize its learning problem is still challenging. The main difficulty is that both the $\ell_1$ and $\ell_2$ norms in $\ell_{1-2}$ are not differentiable, and existing optimization algorithms cannot be applied. In this paper, we show that a closed-form solution can be derived for the proximal step associated with this regularizer. We further extend the result for low-rank matrix learning and the total variation model. Experiments on both synthetic and real data sets show that the resultant accelerated proximal gradient algorithm is more efficient than other noncovex optimization algorithms.

Motivation & Objective

• Address the challenge of efficiently optimizing the nonconvex ℓ₁₋₂ regularizer, which combines ℓ₁ and ℓ₂ norms as ‖x‖₁ − ‖x‖₂.
• Overcome the inefficiency of existing methods like DCA and SCP, which require expensive subproblem solves and suffer from slow convergence.
• Enable the use of proximal gradient algorithms—known for convergence guarantees—on ℓ₁₋₂ regularization by deriving a cheap, closed-form proximal step.
• Extend the proposed solution to low-rank matrix learning and total variation models for broader applicability in machine learning and signal processing.

Proposed method

• Derive a closed-form solution for the proximal operator of the ℓ₁₋₂ regularizer, solving argminₓ ½‖x − z‖₂² + λ(‖x‖₁ − ‖x‖₂) using thresholding and soft-thresholding principles.
• Leverage the difference-of-convex (DC) structure of ℓ₁₋₂ to apply proximal gradient methods, which are known to converge to critical points even for nonconvex, nonsmooth regularizers.
• Design a specialized algorithm (nmAPG) that exploits the closed-form proximal step and uses low-rank structure in matrix completion to accelerate SVD computation.
• Extend the framework to low-rank matrix learning by applying the proximal step to the nuclear norm and total variation models, maintaining efficiency.
• Implement an alternating minimization (AltMin) algorithm for image denoising, combining total variation with ℓ₁₋₂ regularization.
• Use efficient numerical techniques such as inexact line search and rank-adaptive updates to maintain speed while preserving accuracy.

Experimental results

Research questions

• RQ1 Can a closed-form proximal step be derived for the nonconvex ℓ₁₋₂ regularizer, which is the difference of ℓ₁ and ℓ₂ norms?
• RQ2 Does using this closed-form proximal step significantly improve the convergence speed and performance of proximal gradient algorithms in sparse and low-rank learning?
• RQ3 How does the proposed method compare to state-of-the-art solvers like DCA, FaNCL, and AIS-Impute in terms of recovery accuracy and runtime on real-world datasets?
• RQ4 Can the ℓ₁₋₂ regularizer outperform other nonconvex regularizers (e.g., log-sum-penalty, capped ℓ₁) in matrix completion and image denoising tasks?
• RQ5 Is the proposed method scalable and effective for large-scale problems, especially when combined with low-rank or total variation structures?

Key findings

• The proposed method achieves the lowest RMSE across all tested images in matrix completion, outperforming both nuclear norm and factorization-based approaches.
• nmAPG runs 20–30 times faster than DCA in image denoising while achieving comparable reconstruction quality.
• In matrix completion, nmAPG achieves significantly lower recovery error than AIS-Impute, SketchCG, ARSS-M3F, LMaFit, and ER1MP, despite being faster than most.
• The closed-form proximal step enables efficient optimization without requiring iterative subproblem solves, reducing per-iteration cost and accelerating convergence.
• The ℓ₁₋₂ regularizer consistently outperforms other nonconvex regularizers in terms of recovery accuracy on both synthetic and real-world data.
• The method maintains strong performance across diverse tasks, including compressed sensing, matrix completion, and image denoising, demonstrating broad applicability.