[Paper Review] A nonsmooth exact penalty method for equality-constrained optimization: complexity and implementation

Youssef Diouane, Maxence Gollier+1 more|arXiv (Cornell University)|Jan 1, 2024

Advanced Optimization Algorithms Research1 citations

TL;DR

This paper proposes a practical, proximal-based implementation of the exact ℓ2-penalty method for equality-constrained optimization, demonstrating O(ϵ⁻²) worst-case complexity under standard assumptions. It shows that nonsmooth exact penalty methods can be efficiently solved using modern proximal solvers, outperforming augmented Lagrangian methods in robustness and efficiency on small-scale problems while remaining competitive with SQP methods.

ABSTRACT

Penalty methods are a well known class of algorithms for constrained optimization. They transform a constrained problem into a sequence of unconstrained \emph{penalized} problems in the hope that approximate solutions of the latter converge to a solution of the former. If Lagrange multipliers exist, exact penalty methods ensure that the penalty parameter only need increase a finite number of times, but are typically scorned in smooth optimization for the penalized problems are not smooth. This led researchers to consider the implementation of exact penalty methods inconvenient. Recent advances in proximal methods have led to increasingly efficient solvers for nonsmooth optimization. We study a general exact penalty algorithm and use it to show that the exact $\ell_2$-penalty method for equality-constrained optimization can, in fact, be implemented efficiently by solving the penalized problem using a proximal-type algorithm. We study the convergence of our algorithm and establish a worst-case complexity bound of $\mathcal{O}(ε^{-2})$ to bring a stationarity measure below $ε> 0$ under the Mangarasian-Fromowitz constraint qualification and Lipschitz continuity of the objective gradient and constraint Jacobian. While the Lipschitz continuity of the objective gradient is not required for convergence in view of recent works, it is used in our analysis to derive the complexity bound. In a degenerate scenario where the penalty parameter grows unbounded, the complexity becomes $\mathcal{O}(ε^{-8})$, which is worse than another bound found in the literature. Finally, we report numerical experience on small-scale problems from a standard collection and compare our solver with an augmented-Lagrangian and an SQP method. Our preliminary implementation is superior to the augmented Lagrangian in terms of robustness and efficiency, and is competitive with the SQP method.

Motivation & Objective

• To address the long-standing perception that exact penalty methods are impractical due to nonsmoothness, by showing they can be efficiently implemented using modern proximal solvers.
• To establish a worst-case complexity bound of O(ϵ⁻²) for the stationarity measure to fall below ϵ > 0 under the Mangasarian-Fromowitz constraint qualification and Lipschitz continuity of gradients.
• To demonstrate that the ℓ2-penalty method can be implemented efficiently via proximal algorithms, offering a viable alternative to augmented Lagrangian and SQP methods.
• To justify the choice of a properly scaled feasibility measure by analyzing its impact on complexity bounds, especially in degenerate cases.
• To provide the first concrete implementation of the exact ℓ2-penalty method using proximal techniques, with numerical validation on standard test problems.

Proposed method

• Uses a proximal-type algorithm to solve the nonsmooth penalized subproblems arising in the exact ℓ2-penalty method, leveraging efficient proximal operator evaluations.
• Employs a modified Quasi-Newton proximal method (R2N) with adaptive regularization and line search to ensure sufficient decrease in the augmented model.
• Derives efficient procedures for computing proximal operators via solution of trust-region subproblems, including handling rank-deficient Jacobians via alternative saddle-point systems.
• Applies a trust-region framework to the penalized problem, ensuring convergence and enabling complexity analysis under standard assumptions.
• Uses a backtracking line search with adaptive regularization parameter updates to control step size and ensure global convergence.
• Provides a framework general enough to extend to other norms (e.g., ℓ1, ℓ∞) as long as the resulting subproblems are efficiently solvable.

Experimental results

Research questions

• RQ1 Can exact penalty methods for equality-constrained optimization be made practical through modern proximal solvers, despite their nonsmoothness?
• RQ2 What is the worst-case complexity of a proximal implementation of the exact ℓ2-penalty method under standard assumptions?
• RQ3 How does the choice of feasibility measure affect the complexity bound, particularly in degenerate scenarios?
• RQ4 Can the proposed method outperform established methods like augmented Lagrangian and SQP in terms of robustness and efficiency on small-scale problems?
• RQ5 Is the ℓ2-penalty method amenable to efficient implementation via proximal algorithms, and what are the key algorithmic components required?

Key findings

• The proposed algorithm achieves a worst-case complexity bound of O(ϵ⁻²) for reducing the stationarity measure below ϵ > 0 under the Mangasarian-Fromowitz constraint qualification and Lipschitz continuity of the objective gradient and constraint Jacobian.
• In degenerate cases where the penalty parameter grows unboundedly, the complexity degrades to O(ϵ⁻⁸), which is worse than the O(ϵ⁻⁵) bound in prior work, but the authors justify this as a consequence of a properly scaled feasibility measure.
• The numerical experiments show that the proposed solver is more robust and efficient than the augmented Lagrangian method on small-scale problems from a standard test set.
• The solver remains competitive with the SQP method in terms of robustness, though SQP still requires fewer function evaluations in general.
• The implementation is the first known practical realization of the exact ℓ2-penalty method using proximal techniques, with efficient evaluation of proximal operators via trust-region subproblem solutions.
• The framework is extensible to other norms such as ℓ1 and ℓ∞, provided the corresponding proximal operators can be computed efficiently.