[Paper Review] An Inexact Augmented Lagrangian Framework for Nonconvex Optimization with Nonlinear Constraints

Mehmet Fatih Sahin, Armin Eftekhari+3 more|arXiv (Cornell University)|Jun 26, 2019

Sparse and Compressive Sensing TechniquesEngineering61 references19 citations

TL;DR

This paper proposes an inexact augmented Lagrangian method (iALM) for nonconvex optimization with nonlinear constraints, using inexact subproblem solves. It establishes a total iteration complexity of $\tilde{\mathcal{O}}(1/\epsilon^4)$ for first-order stationary points and $\tilde{\mathcal{O}}(1/\epsilon^5)$ for second-order points, matching known theoretical bounds under a verifiable geometric condition related to Polyak-Lojasiewicz and Mangasarian-Fromovitz conditions.

ABSTRACT

We propose a practical inexact augmented Lagrangian method (iALM) for nonconvex problems with nonlinear constraints. We characterize the total computational complexity of our method subject to a verifiable geometric condition, which is closely related to the Polyak-Lojasiewicz and Mangasarian-Fromowitz conditions. In particular, when a first-order solver is used for the inner iterates, we prove that iALM finds a first-order stationary point with $ ilde{\mathcal{O}}(1/ε^4)$ calls to the first-order oracle. If, in addition, the problem is smooth and a second-order solver is used for the inner iterates, iALM finds a second-order stationary point with $ ilde{\mathcal{O}}(1/ε^5)$ calls to the second-order oracle, which matches the known theoretical complexity result in the literature. We also provide strong numerical evidence on large-scale machine learning problems, including the Burer-Monteiro factorization of semidefinite programs, and a novel nonconvex relaxation of the standard basis pursuit template. For these examples, we also show how to verify our geometric condition.

Motivation & Objective

• To develop a practical and theoretically grounded inexact augmented Lagrangian method (iALM) for nonconvex optimization with nonlinear constraints.
• To establish total iteration complexity bounds for iALM under a verifiable geometric condition, linking to Polyak-Lojasiewicz and Mangasarian-Fromovitz conditions.
• To demonstrate the method's effectiveness on large-scale machine learning problems, including Burer-Monteiro factorization of SDPs and nonconvex relaxations of basis pursuit.
• To verify the geometric condition on key problems and provide numerical evidence of iALM's performance.

Proposed method

• Proposes an inexact augmented Lagrangian framework where inner subproblems are solved approximately using first- or second-order solvers.
• Introduces a geometric condition that ensures convergence and simplifies analysis, relating to the Polyak-Lojasiewicz and Mangasarian-Fromovitz conditions.
• Uses first-order oracles for inner solves to achieve $\tilde{\mathcal{O}}(1/\epsilon^4)$ complexity for first-order stationary points.
• Employs second-order solvers for inner subproblems to achieve $\tilde{\mathcal{O}}(1/\epsilon^5)$ complexity for second-order stationary points.
• Applies the framework to Burer-Monteiro factorization of semidefinite programs and nonconvex relaxations of basis pursuit.
• Verifies the geometric condition on specific problems via analytical and numerical means in appendices.

Experimental results

Research questions

• RQ1 What is the total iteration complexity of an inexact augmented Lagrangian method for nonconvex problems with nonlinear constraints?
• RQ2 How does the proposed method’s complexity compare to known theoretical bounds?
• RQ3 Can the geometric condition introduced in this work be verified for key optimization problems like Burer-Monteiro factorization or QAP?
• RQ4 Does the iALM framework maintain theoretical guarantees when using inexact subproblem solves?
• RQ5 How does the method perform numerically on large-scale machine learning problems?

Key findings

• The iALM achieves $\tilde{\mathcal{O}}(1/\epsilon^4)$ first-order oracle complexity for finding a first-order stationary point, correcting a prior error that had claimed $\tilde{\mathcal{O}}(1/\epsilon^3)$.
• With second-order solvers, the method attains $\tilde{\mathcal{O}}(1/\epsilon^5)$ complexity for second-order stationary points, matching known theoretical results.
• The geometric condition introduced is verifiable and connects to well-known optimality conditions such as Polyak-Lojasiewicz and Mangasarian-Fromovitz.
• Numerical experiments on Burer-Monteiro factorization and nonconvex basis pursuit relaxations show strong performance and feasibility.
• The method is robust and future-proof, allowing substitution of different inner solvers without altering the outer framework.
• The geometric condition was successfully verified for key problems, including semidefinite programs and quadratic assignment problems, via analytical and numerical analysis.