[Paper Review] Convergence analysis of a proximal-type algorithm for DC programs with applications to variable selection

Wu, Shuang, Bui Van Dinh+3 more|arXiv (Cornell University)|Aug 17, 2015

Optimization and Variational Analysis26 references3 citations

TL;DR

This paper proposes a boosted proximal point algorithm for solving nonconvex DC programs of the form $\min \{ \varphi(x) + g(x) - h(x) \}$, where $\varphi$ is differentiable and $g,h$ are convex. By combining proximal point iterations with a descent direction from Fukushima-Mine, the method ensures global convergence and convergence rate under the Kurdyka-Łojasiewicz property, extending convergence analysis to inertial variants.

ABSTRACT

We consider a minimization problem of the form $P(φ, g, h):$ $$\min\left\{f(x):= φ(x) + g(x) - h(x) \colon x \in \mathbb{R}^n ight\},$$ where $φ$ is a differentiable function and $g,$ $h$ are convex functions, and introduce iterative methods to finding a critical point of $f$ when $f$ is differentiable. We show that the point computed by proximal point algorithm at each iteration can be used to determine a descent direction for the objective function at this point. This algorithm can be considered as a combination of proximal point algorithm together with a linesearch step that uses this descent direction. We also study convergence results of these algorithms and the inertial proximal methods proposed by Maing$\acute{e}$ and Moudafi (SIAM J. Optim. {\bf 19}(2008), 397--413) under the main assumption that the objective function satisfies the Kurdika--Łojasiewicz property. The proposed algorithm is then applied to solve the variable selection problem in linear regression.

Motivation & Objective

• To develop a globally convergent algorithm for minimizing DC functions $f(x) = \varphi(x) + g(x) - h(x)$ with differentiable $\varphi$ and convex $g,h$.
• To improve convergence behavior by integrating a descent direction into the proximal point framework.
• To establish global convergence and convergence rate for the proposed algorithm under the Kurdyka-Łojasiewicz inequality.
• To extend convergence analysis to inertial proximal methods for DC programs.
• To provide theoretical justification for the use of proximal-type algorithms in nonconvex optimization with applications in variable selection.

Proposed method

• Introduces a boosted proximal point algorithm that combines proximal iterations with a descent direction derived from the gradient of $\varphi$ and subgradients of $g$ and $h$.
• Uses a linesearch step along the descent direction to ensure sufficient decrease in the objective function at each iteration.
• Employs the Kurdyka-Łojasiewicz (KŁ) inequality as the main assumption to prove global convergence and convergence rate.
• Adapts techniques from recent work on KŁ-based convergence analysis to handle nonconvex, non-smooth DC programs.
• Analyzes the inertial proximal method of Maingé and Moudafi by leveraging the KŁ property to establish convergence.
• Derives convergence rate estimates based on the KŁ exponent $\kappa$, showing linear or sublinear convergence depending on $\kappa \in (0,1)$.

Experimental results

Research questions

• RQ1 Can a proximal-type algorithm with a descent direction achieve global convergence for DC programs with differentiable $\varphi$ and convex $g,h$?
• RQ2 Under what conditions does the proposed boosted proximal point algorithm converge globally and with a quantifiable rate?
• RQ3 How does the Kurdyka-Łojasiewicz inequality enable convergence analysis for nonconvex DC programs?
• RQ4 Can the convergence results for standard proximal methods be extended to inertial variants in DC programming?
• RQ5 What is the convergence rate of the proposed algorithm in terms of the KŁ exponent $\kappa$?

Key findings

• The proposed boosted proximal point algorithm globally converges to a critical point of the DC program under the Kurdyka-Łojasiewicz property.
• The algorithm ensures a sufficient decrease in the objective function at each iteration via a linesearch step along a descent direction.
• Global convergence of the inertial proximal method is established under the same KŁ assumption, extending prior results.
• Convergence rate is quantified: if the KŁ exponent $\kappa \in (0,1)$, the sequence converges linearly or sublinearly depending on $\kappa$, with explicit bounds in terms of $\|z^{k+1} - z^k\|$.
• The convergence proof relies on showing that the sequence $\{z^k\}$ is Cauchy, using the KŁ inequality and summability of differences.
• The analysis confirms that the algorithm is applicable to variable selection problems due to the structure of the DC decomposition.