[Paper Review] Complexity of a quadratic penalty accelerated inexact proximal point method for solving linearly constrained nonconvex composite programs

Weiwei Kong, Jefferson G. Melo+1 more|arXiv (Cornell University)|Feb 10, 2018

Sparse and Compressive Sensing Techniques28 references8 citations

TL;DR

This paper proposes a quadratic penalty accelerated inexact proximal point (QP-AIPP) method for solving linearly constrained nonconvex composite programs, combining an accelerated inexact proximal point scheme with an ACG method to solve penalized subproblems. It establishes an iteration-complexity bound of $\mathcal{O}(\hat{\rho}^{-2}\hat{\eta}^{-1})$ for generating a triple $(\hat{z}, \hat{v}, \hat{p})$ satisfying $\|\hat{v}\| \leq \hat{\rho}$ and $\|A\hat{z} - b\| \leq \hat{\eta}$, under mild assumptions including finite optimal value but no bounded domain or solution existence requirement.

ABSTRACT

This paper analyzes the iteration-complexity of a quadratic penalty accelerated inexact proximal point method for solving linearly constrained nonconvex composite programs. More specifically, the objective function is of the form $f + h$ where $f$ is a differentiable function whose gradient is Lipschitz continuous and $h$ is a closed convex function with bounded domain. The method, basically, consists of applying an accelerated inexact proximal point method for solving approximately a sequence of quadratic penalized subproblems associated to the linearly constrained problem. Each subproblem of the proximal point method is in turn approximately solved by an accelerated composite gradient (ACG) method. It is shown that the proposed scheme generates a $ρ$-approximate stationary point in at most ${\cal{O}}(ρ^{-3})$ ACG iterations. Finally, numerical results showing the efficiency of the proposed method are also given.

Motivation & Objective

• To develop a first-order method with improved iteration-complexity for solving linearly constrained nonconvex composite programs where the objective is $f + h$, with $f$ differentiable and $L_f$-Lipschitz, and $h$ closed convex.
• To address the challenge of solving nonconvex problems with linear constraints by combining quadratic penalty methods with inexact proximal point schemes.
• To achieve better complexity than standard accelerated gradient methods by exploiting curvature information and using an accelerated inexact proximal point (AIPP) method for subproblem solution.
• To establish iteration-complexity bounds that depend on $\hat{\rho}$ and $\hat{\eta}$, the tolerances for stationarity and feasibility, respectively, without requiring bounded domain or existence of optimal solutions.

Proposed method

• The method applies a quadratic penalty approach to transform the original problem into a sequence of penalized subproblems of the form $\min_z f(z) + h(z) + \frac{c}{2}\|Az - b\|^2$.
• Each penalized subproblem is solved approximately using an accelerated inexact proximal point (AIPP) method, which iteratively minimizes a proximal approximation of the penalized function.
• The AIPP method solves subproblems of the form $\min_z f(z) + h(z) + \frac{c}{2}\|Az - b\|^2 + \frac{1}{2\lambda}\|z - z_{k-1}\|^2$ via an accelerated composite gradient (ACG) method.
• The penalty parameter $c$ is increased adaptively until the feasibility tolerance $\|Az - b\| \leq \hat{\eta}$ is satisfied, ensuring convergence to a $\hat{\rho}$-approximate stationary point.
• The AIPP method leverages the lower curvature $m$ of the objective function to achieve a complexity bound of $\mathcal{O}(\sqrt{M/m} \cdot \min\{\phi(z_0) - \phi_*, m d_0^2\} / \rho^2 + \sqrt{M/m} \log^+(M/m))$, improving upon standard ACG by a factor of $\sqrt{M/m}$.
• The method is designed to work under minimal assumptions: the optimal value of the original problem is finite, but $\text{dom}\,h$ need not be bounded and an optimal solution need not exist.

Experimental results

Research questions

• RQ1 What is the iteration-complexity of a quadratic penalty method combined with an accelerated inexact proximal point scheme for solving linearly constrained nonconvex composite programs?
• RQ2 How does the use of an AIPP method for solving penalized subproblems improve the complexity compared to standard ACG methods?
• RQ3 Can the method achieve a $\hat{\rho}$-approximate stationary point and $\hat{\eta}$-feasible solution with complexity independent of the domain size or existence of optimal solutions?
• RQ4 What is the dependence of the complexity bound on the penalty parameter $c$, the Lipschitz constant $M$, and the lower curvature $m$ of the penalized objective?
• RQ5 How does the method ensure that the final iterate satisfies both stationarity and feasibility tolerances simultaneously?

Key findings

• The QP-AIPP method generates a $\rho$-approximate stationary point and $\eta$-feasible solution in at most $\mathcal{O}(\hat{\rho}^{-2}\hat{\eta}^{-1})$ accelerated composite gradient (ACG) iterations.
• The method achieves this complexity bound under the mild assumption that the optimal value of the original problem is finite, without requiring bounded domain or existence of optimal solutions.
• The AIPP subproblem solver improves upon standard ACG by reducing the dependence on the Lipschitz constant $M$ from $\mathcal{O}(M)$ to $\mathcal{O}(\sqrt{M})$, due to exploitation of the lower curvature $m$.
• The iteration-complexity bound of the AIPP method is $\mathcal{O}\left(\frac{\sqrt{Mm}}{\rho^2}\min\{\phi(z_0) - \phi_*, m d_0^2\} + \sqrt{\frac{M}{m}}\log^+\left(\frac{M}{m}\right)\right)$, which is superior to standard ACG by a factor of $\sqrt{M/m}$.
• Numerical results demonstrate the efficiency of the proposed method, confirming its theoretical complexity bounds in practice.
• The method ensures that the final iterate satisfies both $\|\hat{v}\| \leq \hat{\rho}$ and $\|A\hat{z} - b\| \leq \hat{\eta}$, with the penalty parameter $c$ adaptively increased to meet feasibility.