[Paper Review] Sparse Regularization: Convergence Of Iterative Jumping Thresholding Algorithm

Jinshan Zeng, Shao-Bo Lin+1 more|arXiv (Cornell University)|Feb 24, 2014

Sparse and Compressive Sensing Techniques38 references4 citations

TL;DR

This paper proposes the Iterative Jumping Thresholding (IJT) algorithm for sparse regularization using non-convex penalties with discontinuous thresholding functions, such as the $l_q$-norm for $0<q<1$. It establishes finite support and sign convergence, and proves strong convergence to a local minimizer with asymptotically linear rate under the restricted Kurdyka-Łojasiewicz (rKL) property and additional smoothness conditions.

ABSTRACT

In recent studies on sparse modeling, non-convex penalties have received considerable attentions due to their superiorities on sparsity-inducing over the convex counterparts. Compared with the convex optimization approaches, however, the non-convex approaches have more challenging convergence analysis. In this paper, we study the convergence of a non-convex iterative thresholding algorithm for solving sparse recovery problems with a certain class of non-convex penalties, whose corresponding thresholding functions are discontinuous with jump discontinuities. Therefore, we call the algorithm the iterative jumping thresholding (IJT) algorithm. The finite support and sign convergence of IJT algorithm is firstly verified via taking advantage of such jump discontinuity. Together with the assumption of the introduced restricted Kurdyka-Łojasiewicz (rKL) property, then the strong convergence of IJT algorithm can be proved.Furthermore, we can show that IJT algorithm converges to a local minimizer at an asymptotically linear rate under some additional conditions. Moreover, we derive a posteriori computable error estimate, which can be used to design practical terminal rules for the algorithm. It should be pointed out that the $l_q$ quasi-norm ($0

Motivation & Objective

• To address the challenge of convergence analysis in non-convex sparse regularization, particularly for penalties with discontinuous thresholding functions.
• To establish finite support and sign convergence for iterative thresholding algorithms when the thresholding function has jump discontinuities.
• To prove strong convergence of the IJT algorithm to a local minimizer under the restricted Kurdyka-Łojasiewicz (rKL) property.
• To derive an asymptotically linear convergence rate under additional smoothness and concentration conditions.
• To provide a posteriori computable error estimate for practical termination rules in the algorithm.

Proposed method

• Proposes the Iterative Jumping Thresholding (IJT) algorithm for solving sparse recovery problems with non-convex penalties, particularly those with discontinuous thresholding functions.
• Utilizes the jump discontinuity in the thresholding function to establish finite support and sign convergence of the iterates.
• Applies the restricted Kurdyka-Łojasiewicz (rKL) property to prove strong convergence of the IJT iterates to a stationary point.
• Employs Taylor expansion and matrix analysis to derive a linear convergence rate in the neighborhood of the limit point.
• Derives a posteriori error estimate based on the convergence rate and triangle inequality for practical stopping criteria.
• Analyzes the $l_q$-norm ($0<q<1$) as a key subclass, showing linear convergence under concentration conditions.

Experimental results

Research questions

• RQ1 Can the IJT algorithm achieve finite support and sign convergence for non-convex penalties with discontinuous thresholding functions?
• RQ2 Under what conditions does the IJT algorithm converge strongly to a local minimizer?
• RQ3 What is the asymptotic convergence rate of the IJT algorithm, and under what assumptions is it linear?
• RQ4 How can a computable error estimate be derived to guide practical termination of the IJT algorithm?
• RQ5 Does the IJT algorithm maintain linear convergence for the $l_q$-norm regularization with $0<q<1$ under suitable concentration conditions?

Key findings

• The IJT algorithm achieves finite support and sign convergence due to the jump discontinuities in the thresholding function.
• Under the restricted Kurdyka-Łojasiewicz (rKL) property, the IJT algorithm converges strongly to a local minimizer.
• The IJT algorithm converges to a local minimizer with an asymptotically linear rate under additional smoothness and concentration conditions.
• For the $l_q$-norm regularization with $0<q<1$, the IJT algorithm achieves linear convergence under appropriate concentration conditions.
• A posteriori computable error estimate is derived, enabling practical implementation with reliable termination rules.
• Simulations confirm the theoretical convergence behavior and demonstrate time efficiency of IJT for $q=1/2$ and $q=2/3$ compared to IRLS and IRL1.