[Paper Review] Smoothed analysis for low-rank solutions to semidefinite programs in quadratic penalty form

Srinadh Bhojanapalli, Nicolas Boumal+2 more|arXiv (Cornell University)|Mar 1, 2018

Sparse and Compressive Sensing Techniques20 references16 citations

TL;DR

This paper proposes a smoothed analysis framework for low-rank solutions to semidefinite programs (SDPs) using a quadratic penalty formulation within the Burer–Monteiro factorization. It shows that for $ k = \tilde{\Omega}(\sqrt{m}) $, all second-order stationary points (SOSPs) of the penalized, perturbed problem are approximately optimal, avoiding pathological cases via random perturbation of the cost matrix.

ABSTRACT

Semidefinite programs (SDP) are important in learning and combinatorial optimization with numerous applications. In pursuit of low-rank solutions and low complexity algorithms, we consider the Burer--Monteiro factorization approach for solving SDPs. We show that all approximate local optima are global optima for the penalty formulation of appropriately rank-constrained SDPs as long as the number of constraints scales sub-quadratically with the desired rank of the optimal solution. Our result is based on a simple penalty function formulation of the rank-constrained SDP along with a smoothed analysis to avoid worst-case cost matrices. We particularize our results to two applications, namely, Max-Cut and matrix completion.

Motivation & Objective

• To address the challenge of solving large-scale semidefinite programs (SDPs) efficiently by reducing variable dimension via low-rank factorization.
• To overcome non-convexity and constraint complexity in the Burer–Monteiro factorized SDP formulation.
• To establish conditions under which second-order stationary points (SOSPs) of the penalized, factored problem are globally optimal.
• To use smoothed analysis to avoid rare pathological cost matrices that produce suboptimal SOSPs.
• To extend theoretical guarantees to practical, approximately optimal solutions under bounded residues and perturbed objectives.

Proposed method

• Formulate the rank-constrained SDP using Burer–Monteiro factorization: $ X = UU^T $, reducing variables from $ O(n^2) $ to $ O(nk) $.
• Apply a quadratic penalty to constraints, transforming the problem into $ \min_U L_\mu(U) = \langle C, UU^T \rangle + \mu \sum_i (\langle A_i, UU^T \rangle - b_i)^2 $.
• Use smoothed analysis by adding small i.i.d. Gaussian noise to the cost matrix $ C $, ensuring with high probability that all SOSPs are approximately optimal.
• Analyze second-order stationary points (SOSPs) via gradient and Hessian conditions: small gradient norm and Hessian with minimal negative curvature.
• Establish bounds on solution residues (constraint violations) and objective suboptimality using perturbation theory and spectral norms.
• Apply gradient descent with convergence guarantees: $ \widetilde{O}(1/\epsilon^2) $ iterations to reach $ (\epsilon, \rho^2) $-SOSP, improved to $ \widetilde{O}(1/\epsilon^{1.75}) $ with Nesterov acceleration.

Experimental results

Research questions

• RQ1 Under what conditions are all second-order stationary points (SOSPs) of the penalized, factored SDP globally optimal?
• RQ2 Can smoothed analysis eliminate the set of pathological cost matrices that yield suboptimal SOSPs?
• RQ3 What is the minimal rank $ k $ required for SOSPs of the penalized problem to be approximately optimal after random perturbation of the cost matrix?
• RQ4 How does the convergence rate of gradient-based methods depend on problem parameters in the penalized, factored formulation?
• RQ5 Can the theoretical guarantees be extended to the original SDP (1) from the penalized, perturbed problem (4)?

Key findings

• For $ k = \tilde{\Omega}(\sqrt{m}) $, all second-order stationary points (SOSPs) of the penalized, perturbed problem are approximately optimal with high probability.
• The perturbed gradient method converges to an $ (\epsilon, \rho^2) $-SOSP in $ \widetilde{O}(1/\epsilon^2) $ iterations, improving to $ \widetilde{O}(1/\epsilon^{1.75}) $ with Nesterov acceleration.
• When the Hessian is bounded and the cost matrix is perturbed with i.i.d. Gaussian noise, all SOSPs have bounded residues and are approximately optimal.
• For positive definite cost matrices and compact SDPs, SOSPs of the penalized problem have bounded solutions (residues), ensuring stability.
• The method achieves approximate optimality for the original SDP (1) under mild conditions, with theoretical guarantees extending to Max-Cut and matrix completion applications.
• Computational complexity is $ O(mn^2k) $ per iteration for dense problems, but significantly reduced in sparse settings due to sparsity in constraint matrices.