[Paper Review] A Penalty-free Infeasible Approach for a Class of Nonsmooth Optimization Problems over the Stiefel Manifold

Nachuan Xiao, Xin Liu+1 more|arXiv (Cornell University)|Mar 5, 2021

Sparse and Compressive Sensing Techniques4 citations

TL;DR

This paper proposes a penalty-free infeasible optimization method, Sequential Linearized Proximal Gradient (SLPG), for nonsmooth problems on the Stiefel manifold. SLPG alternates between tangential steps (inexact, orthonormalization-free) and normal steps (feasibility-improving via Taylor approximation), avoiding penalty parameters and achieving high efficiency and global convergence for smooth and ℓ₂,₁-regularized problems.

ABSTRACT

Transforming into an exact penalty function model with convex compact constraints yields efficient infeasible approaches for optimization problems with orthogonality constraints. For smooth and $\ell_{2,1}$-norm regularized cases, these infeasible approaches adopt simple and orthonormalization-free updating scheme and show their high efficiency in the test examples. However, to avoid orthonormalization while enforcing the feasibility of the final solution, these infeasible approaches introduce a quadratic penalty term, where an inappropriate penalty parameter can lead to numerical inefficiency. Inspired by penalty-free approaches for smooth optimization problems, we proposed a proximal first-order algorithm for a class of optimization problems with orthogonality constraints and nonsmooth regularization term. The consequent algorithm, named sequential linearized proximal gradient method (SLPG), alternatively takes tangential steps and normal steps to improve the optimality and feasibility respectively. In SLPG, the orthonormalization process is invoked only once at the last step if high precision in feasibility is needed, showing that main iterations in SLPG are orthonormalization-free. Besides, both the tangential steps and normal steps do not involve the penalty parameter, and thus SLPG is penalty-free and avoids the inefficiency by inappropriate penalty parameter. We analyze the global convergence properties of SLPG where the tangential steps are inexactly computed. By inexactly computing tangential steps, for smooth cases and $\ell_{2,1}$-norm regularized cases, SLPG has a closed-form updating scheme, which leads to its cheap tangential steps. Numerical experiments illustrate the numerical advantages of SLPG when compared with existing first-order methods.

Motivation & Objective

• Address the inefficiency of penalty-based infeasible methods in nonsmooth optimization over the Stiefel manifold due to sensitivity to penalty parameter selection.
• Develop a scalable, infeasible framework that avoids orthonormalization during main iterations while ensuring final feasibility.
• Establish global convergence for a class of composite optimization problems with nonsmooth regularization and orthogonality constraints.
• Provide a computationally efficient approach for problems such as sparse PCA and Kohn-Sham energy minimization with high-dimensional data.
• Achieve orthonormalization-free iterations while maintaining feasibility through a normal step inspired by polar decomposition.

Proposed method

• Propose a sequential linearized proximal gradient (SLPG) method that alternates between tangential and normal steps to improve optimality and feasibility separately.
• Use a linearized proximal subproblem in an affine subspace to compute tangential steps, which reduces to a tangent space subproblem when the iterate is feasible.
• Incorporate a fixed-point iteration or direct closed-form solution for tangential steps when the objective is smooth or ℓ₂,₁-regularized, enabling low-cost computation.
• Approximate the orthonormalization process via a normal step derived from the first-order Taylor expansion of the polar decomposition.
• Invoke orthonormalization only once at the final iteration to achieve high-precision feasibility, ensuring main iterations remain orthonormalization-free.
• Eliminate penalty parameters entirely by decoupling optimality and feasibility improvement, avoiding numerical instability from poor penalty parameter choices.

Experimental results

Research questions

• RQ1 Can a penalty-free infeasible method achieve global convergence for nonsmooth optimization over the Stiefel manifold?
• RQ2 How can tangential and normal steps be designed to improve optimality and feasibility independently without relying on penalty terms?
• RQ3 What is the computational efficiency of an orthonormalization-free method when compared to existing first-order methods on large-scale problems?
• RQ4 Can closed-form or inexpensive solutions be derived for the tangential subproblem in smooth and ℓ₂,₁-regularized cases?
• RQ5 Does the proposed method maintain robust performance across diverse applications such as sparse PCA and Kohn-Sham energy minimization?

Key findings

• SLPG achieves global convergence even when tangential steps are inexactly computed, under the blanket assumption on the problem structure.
• For smooth and ℓ₂,₁-regularized cases, SLPG provides a closed-form or computationally cheap tangential update, enabling efficient iterations.
• Numerical experiments show SLPG outperforms existing first-order methods (e.g., ManOptQR, PenCF, PCAL, OptM) in terms of CPU time and iteration count across diverse test cases.
• On the alanine dataset (n=12671, p=18), SLPG required 73 iterations and 21.27 seconds, outperforming PenCF (83 iterations, 24.10 seconds) and PCAL (97 iterations, 30.56 seconds).
• On the large pentacene dataset (n=44791, p=51), SLPG used 113 iterations and 308.88 seconds, significantly faster than ManOptQR (1000 iterations, 425.62 seconds) and comparable to PenCF (109 iterations, 301.13 seconds).
• SLPG maintains feasibility violation below 1e-15 in all test cases, with final iterates achieving high-precision feasibility through a single final orthonormalization.