[논문 리뷰] Iterative Reweighted Minimization Methods for $l_p$ Regularized Unconstrained Nonlinear Programming

Zhaosong Lu|arXiv (Cornell University)|2012. 09. 29.

Sparse and Compressive Sensing Techniques참고 문헌 26인용 수 7

한 줄 요약

이 논문은 $p \in (0,1)$ 인 $l_p$-정규화된 비선형 프로그래밍 문제에 대해 새로운 반복 가중치 최소화 방법을 제안한다. $\|x\|_p^p$에 대한 리프시츠 연속적인 $\epsilon$-근사값을 도입함으로써, 동적 $\epsilon$ 업데이트 없이 수렴 보장을 가능하게 한다. 주요 기여는 $\epsilon$ 가 계산 가능한 임계값 이하일 경우, 신규 IRL1 방법의 임의의 집적점이 1차 정류점임을 증명한 것이다. 이는 기존 방법이 $\epsilon \to 0$ 가 되어야만 수렴 보장이 가능한 것과 비교해 더 강력한 수렴 성질을 제공한다. 계산 결과는 목적 함수 값과 CPU 시간에서 향상된 안정성을 보여준다.

ABSTRACT

In this paper we study general $l_p$ regularized unconstrained minimization problems. In particular, we derive lower bounds for nonzero entries of first- and second-order stationary points, and hence also of local minimizers of the $l_p$ minimization problems. We extend some existing iterative reweighted $l_1$ (IRL1) and $l_2$ (IRL2) minimization methods to solve these problems and proposed new variants for them in which each subproblem has a closed form solution. Also, we provide a unified convergence analysis for these methods. In addition, we propose a novel Lipschitz continuous $ε$-approximation to $\|x\|^p_p$. Using this result, we develop new IRL1 methods for the $l_p$ minimization problems and showed that any accumulation point of the sequence generated by these methods is a first-order stationary point, provided that the approximation parameter $ε$ is below a computable threshold value. This is a remarkable result since all existing iterative reweighted minimization methods require that $ε$ be dynamically updated and approach zero. Our computational results demonstrate that the new IRL1 method is generally more stable than the existing IRL1 methods [21,18] in terms of objective function value and CPU time.

연구 동기 및 목표

• To develop stable and convergent iterative reweighted minimization methods for $l_p$-regularized unconstrained nonlinear programming with $p \in (0,1)$.
• To derive lower bounds for nonzero entries of first- and second-order stationary points and local minimizers of $l_p$ minimization problems.
• To propose a novel Lipschitz continuous $\epsilon$-approximation to $\|x\|_p^p$ that enables convergence analysis without dynamic $\epsilon$ updates.
• To extend existing IRL1 and IRL2 methods to general $l_p$ problems and provide a unified convergence analysis.
• To demonstrate the computational superiority of the new IRL1 method over existing variants in terms of objective value and CPU time.

제안 방법

• Introduce a new $\epsilon$-approximation to $\|x\|_p^p$ as $\sum_{i=1}^n (|x_i| + \epsilon)^p$, which is Lipschitz continuous and enables stable optimization.
• Develop new IRL1 methods by applying iterative reweighting to the $\epsilon$-approximation, solving each subproblem in closed form.
• Establish a unified convergence analysis for the extended IRL1 and IRL2 methods under the new approximation framework.
• Derive a computable threshold for $\epsilon$ such that any accumulation point of the sequence generated by the new IRL1 method is a first-order stationary point.
• Apply the methods to solve $\min_x \{ f(x) + \lambda \|x\|_p^p \}$ with $f$ having $L_f$-Lipschitz gradient and bounded below.
• Use numerical experiments with random $A$, $b$, and $\lambda = 3 \times 10^{-3}$ to compare performance across $p = 0.1$ and $p = 0.5$.

실험 결과

연구 질문

• RQ1 Can a fixed, non-dynamically updated $\epsilon$-approximation to $\|x\|_p^p$ ensure convergence to a first-order stationary point in $l_p$ minimization?
• RQ2 What are the lower bounds on nonzero entries of first- and second-order stationary points in $l_p$-regularized problems?
• RQ3 How do the proposed IRL1 and IRL2 variants compare in stability and convergence speed to existing methods?
• RQ4 Can a Lipschitz continuous $\epsilon$-approximation to $\|x\|_p^p$ enable convergence guarantees without requiring $\epsilon \to 0$?
• RQ5 Does the new IRL1 method achieve better objective function values and lower CPU time than prior IRL1 methods?

주요 결과

• The new IRL1 method achieves better stability than existing IRL1 methods [21, 18] in terms of both objective function value and CPU time across all tested instances.
• For $p = 0.1$, the new IRL1 variant (IRL1-3) achieves the best objective value in 3 out of 10 instances, with significantly lower average CPU time than IRL1-1 and IRL1-2.
• For $p = 0.5$, all three IRL1 variants achieve similar objective values, but IRL1-3 and IRL1-2 have much lower CPU time than IRL1-1.
• The proposed $\epsilon$-approximation allows convergence to a first-order stationary point when $\epsilon$ is below a computable threshold, without requiring $\epsilon$ to be dynamically decreased to zero.
• The unified convergence analysis confirms global convergence properties for the extended IRL1 and IRL2 methods under the new framework.
• Lower bounds for nonzero entries of first- and second-order stationary points are derived, providing theoretical insight into sparsity of solutions.