[Paper Review] Penalty Methods for Constrained Non-Lipschitz Optimization

Xiaojun ChenZhaosong Lu, Ting Kei Pong|arXiv (Cornell University)|Jan 1, 2015

Sparse and Compressive Sensing Techniques26 references3 citations

TL;DR

This paper proposes a penalty method for constrained non-Lipschitz optimization with nonconvex objectives and convex feasible sets defined by polyhedral and ellipsoidal constraints. It establishes exact penalization for local minimizers, stationary points, and ε-minimizers, and develops a convergent nonmonotone proximal gradient algorithm with adaptive penalty updates, demonstrating effectiveness in finding sparse solutions.

ABSTRACT

We consider a class of constrained optimization problems with a possibly nonconvex non-Lipschitz objective and a convex feasible set being the intersection of a polyhedron and a possibly degenerated ellipsoid. Such a problem has a wide range of applications in data science, where the objective is used for inducing sparsity in the solutions while the constraint set models the noise tolerance and incorporates other prior information for data tting. To solve this kind of constrained optimization problems, a common approach is the penalty method. However, there is little theory on exact penalization for problems with nonconvex non-Lipschitz objectives. In this paper, we study the existence of exact penalty parameters regarding local minimizers, stationary points and -minimizers under suitable assumptions. Moreover, we discuss a penalty method whose subproblems are solved via a nonmonotone proximal gradient method with a suitable update scheme for the penalty parameters, and prove the convergence of the algorithm to a KKT point of the constrained problem. Preliminary numerical results demonstrate the eciency of the penalty method for nding sparse solutions.

Motivation & Objective

• To address constrained optimization problems with nonconvex, non-Lipschitz objectives and convex feasible sets, common in data science for inducing sparsity.
• To establish theoretical conditions under which exact penalty parameters exist for local minimizers, stationary points, and ε-minimizers.
• To develop a practical algorithm that solves subproblems via a nonmonotone proximal gradient method with adaptive penalty parameter updates.
• To prove convergence of the algorithm to a KKT point of the original constrained problem.
• To demonstrate the efficiency of the method in computing sparse solutions through preliminary numerical experiments.

Proposed method

• The method employs a penalty formulation where the original constrained problem is transformed into a sequence of unconstrained subproblems using a penalty term.
• The penalty parameters are updated adaptively via a suitable scheme to ensure convergence without requiring monotonic decrease in the objective.
• Subproblems are solved using a nonmonotone proximal gradient method, which enhances robustness and convergence in nonconvex settings.
• Theoretical analysis establishes the existence of exact penalty parameters under suitable assumptions, ensuring that solutions to the penalized problem coincide with solutions to the original problem.
• The algorithm is designed to converge to a KKT point of the original constrained problem, even when the objective is non-Lipschitz and nonconvex.
• The feasible set is modeled as the intersection of a polyhedron and a possibly degenerate ellipsoid, capturing noise tolerance and prior information in data fitting.

Experimental results

Research questions

• RQ1 Under what conditions does an exact penalty parameter exist for local minimizers in non-Lipschitz, nonconvex optimization with convex constraints?
• RQ2 Can exact penalization be established for stationary points and ε-minimizers in this class of problems?
• RQ3 Does a nonmonotone proximal gradient method with adaptive penalty updates converge to a KKT point for such problems?
• RQ4 How effective is the proposed penalty method in computing sparse solutions in data science applications?
• RQ5 What theoretical guarantees can be provided for the convergence of the algorithm under non-Lipschitz and nonconvex objectives?

Key findings

• The paper establishes the existence of exact penalty parameters for local minimizers, stationary points, and ε-minimizers under suitable assumptions.
• The proposed penalty method with adaptive penalty parameter updates converges to a KKT point of the original constrained problem.
• The nonmonotone proximal gradient method used in subproblems ensures convergence despite nonconvexity and non-Lipschitz behavior.
• Preliminary numerical results confirm the efficiency of the method in identifying sparse solutions.
• The theoretical framework extends exact penalization to non-Lipschitz, nonconvex settings, which were previously underdeveloped in the literature.
• The method effectively handles constraints defined as the intersection of a polyhedron and a possibly degenerate ellipsoid, enabling noise tolerance and prior information incorporation.