[Paper Review] Nonconvex Generalization of ADMM for Nonlinear Equality Constrained Problems

Junxiang Wang, Liang Zhao|arXiv (Cornell University)|May 9, 2017

Sparse and Compressive Sensing Techniques41 references16 citations

TL;DR

This paper proposes neADMM, a novel generic ADMM framework for solving nonlinear equality constrained optimization problems, leveraging a variational inequality framework to establish convergence and sublinear rate guarantees. It introduces an adaptive penalty update strategy and demonstrates effectiveness across diverse applications like matrix completion and sparse learning.

ABSTRACT

The growing demand on efficient and distributed optimization algorithms for large-scale data stimulates the popularity of Alternative Direction Methods of Multipliers (ADMM) in numerous areas, such as compressive sensing, matrix completion, and sparse feature learning. While linear equality constrained problems have been extensively explored to be solved by ADMM, there lacks a generic framework for ADMM to solve problems with nonlinear equality constraints, which are common in practical application (e.g., orthogonality constraints). To address this problem, in this paper, we proposed a new generic ADMM framework for handling nonlinear equality constraints, called neADMM. First, we propose the generalized problem formulation and systematically provide the sufficient condition for the convergence of neADMM. Second, we prove a sublinear convergence rate based on variational inequality framework and also provide an novel accelerated strategy on the update of the penalty parameter. In addition, several practical applications under the generic framework of neADMM are provided. Experimental results on several applications demonstrate the usefulness of our neADMM.

Motivation & Objective

• Address the lack of a generic ADMM framework for nonlinear equality constrained problems, which are common in practical applications such as orthogonality constraints.
• Establish a generalized problem formulation that extends ADMM to nonlinear equality constraints.
• Provide sufficient conditions for convergence of the proposed neADMM algorithm.
• Prove a sublinear convergence rate using a variational inequality framework.
• Develop an accelerated strategy for updating the penalty parameter to improve convergence performance.

Proposed method

• Propose a generalized optimization formulation that incorporates nonlinear equality constraints into the ADMM framework.
• Introduce a variational inequality-based analysis to derive sufficient conditions for convergence of neADMM.
• Establish a sublinear convergence rate of O(1/k) for the algorithm using the variational inequality framework.
• Design an adaptive penalty parameter update strategy that accelerates convergence without requiring line search.
• Integrate the penalty update rule into the standard ADMM update steps to maintain convergence while improving practical performance.
• Ensure the method remains applicable to large-scale and distributed optimization by preserving the decomposability of ADMM.

Experimental results

Research questions

• RQ1 Can a generic ADMM framework be developed to handle nonlinear equality constraints beyond the linear case?
• RQ2 What sufficient conditions ensure the convergence of ADMM when applied to nonlinearly constrained problems?
• RQ3 What convergence rate can be theoretically guaranteed for the proposed neADMM under nonlinear constraints?
• RQ4 How can the penalty parameter be adaptively updated to improve convergence speed without sacrificing convergence guarantees?
• RQ5 How does neADMM perform in practical applications involving nonlinear constraints such as orthogonality or low-rank constraints?

Key findings

• The neADMM framework provides a general and systematic approach to solving nonlinear equality constrained optimization problems using ADMM.
• Theoretical analysis confirms convergence under a set of sufficient conditions derived from variational inequality theory.
• A sublinear convergence rate of O(1/k) is established, indicating global convergence with diminishing error over iterations.
• The proposed adaptive penalty update strategy significantly improves convergence speed in practice compared to fixed or heuristic updates.
• Experimental results on applications including matrix completion and sparse feature learning demonstrate the effectiveness and robustness of neADMM.
• The framework is applicable to a wide range of real-world problems involving nonlinear constraints, such as those arising in signal processing and machine learning.