[Paper Review] Sparse Plus Low Rank Matrix Decomposition: A Discrete Optimization Approach

Dimitris Bertsimas, Ryan Cory-Wright+1 more|arXiv (Cornell University)|Sep 26, 2021

Sparse and Compressive Sensing Techniques32 references4 citations

TL;DR

This paper proposes a discrete optimization approach for Sparse Plus Low-Rank (SLR) matrix decomposition that directly models sparsity and low-rank structure via a novel non-convex formulation. It introduces an alternating minimization heuristic scaling to $ n = 10,000 $, a strong semidefinite relaxation for bound certification, and a branch-and-bound algorithm solving small instances ($ n = 25 $) to certifiable near-optimality, outperforming state-of-the-art methods in sparsity, rank, and mean-square error.

ABSTRACT

We study the Sparse Plus Low-Rank decomposition problem (SLR), which is the problem of decomposing a corrupted data matrix into a sparse matrix of perturbations plus a low-rank matrix containing the ground truth. SLR is a fundamental problem in Operations Research and Machine Learning which arises in various applications, including data compression, latent semantic indexing, collaborative filtering, and medical imaging. We introduce a novel formulation for SLR that directly models its underlying discreteness. For this formulation, we develop an alternating minimization heuristic that computes high-quality solutions and a novel semidefinite relaxation that provides meaningful bounds for the solutions returned by our heuristic. We also develop a custom branch-and-bound algorithm that leverages our heuristic and convex relaxations to solve small instances of SLR to certifiable (near) optimality. Given an input $n$-by-$n$ matrix, our heuristic scales to solve instances where $n=10000$ in minutes, our relaxation scales to instances where $n=200$ in hours, and our branch-and-bound algorithm scales to instances where $n=25$ in minutes. Our numerical results demonstrate that our approach outperforms existing state-of-the-art approaches in terms of rank, sparsity, and mean-square error while maintaining a comparable runtime.

Motivation & Objective

• Address the challenge of decomposing a corrupted data matrix into sparse perturbations and a low-rank component, a core problem in machine learning and operations research.
• Overcome the limitations of existing non-convex and convex methods by directly modeling the discrete nature of sparsity and low-rank structure.
• Develop a scalable heuristic that computes high-quality solutions for large-scale SLR problems ($ n = 10,000 $) in minutes.
• Provide tight, computationally tractable bounds via a novel semidefinite relaxation for solution quality assessment.
• Enable certifiable near-optimality for small instances ($ n = 25 $) using a branch-and-bound framework integrating the heuristic and relaxation.

Proposed method

• Propose a new formulation (1) that enforces exact sparsity ($ orm{m{Y}}_0 riangleq ext{number of nonzeros} riangleq k_1 $) and rank constraints ($ ext{Rank}(m{X}) riangleq k_0 $) directly, rather than via regularization.
• Design Algorithm 1: an alternating minimization heuristic that iteratively optimizes over low-rank and sparse components, leveraging SVD and hard thresholding for scalability.
• Develop a semidefinite relaxation (20) that provides strong lower bounds on the optimal objective value, interpreted as a reverse Huber penalty on sparse and low-rank components.
• Integrate the heuristic and relaxation into a branch-and-bound framework (Algorithm 2), using the relaxation to compute bounds at each node and the heuristic to generate incumbents.
• Use a custom branching rule and warm-starting strategies to accelerate convergence in the branch-and-bound tree.
• Leverage the discrete structure of the problem to tighten bounds and improve solution quality without sacrificing scalability.

Experimental results

Research questions

• RQ1 Can a direct discrete optimization formulation for SLR decomposition outperform existing non-convex and convex methods in terms of sparsity, low-rank approximation quality, and reconstruction error?
• RQ2 To what extent can a convex semidefinite relaxation provide tight bounds on the optimal solution of the non-convex SLR problem, enabling near-certifiable optimality?
• RQ3 How does the proposed alternating minimization heuristic scale in practice, and can it achieve high-quality solutions for large-scale instances ($ n = 10,000 $)?
• RQ4 Can a branch-and-bound algorithm based on the heuristic and relaxation solve small SLR instances ($ n = 25 $) to certifiable near-optimality within reasonable time?
• RQ5 What is the trade-off between solution quality and computational cost when using the exact versus accelerated implementation of the heuristic?

Key findings

• Algorithm 1 (the alternating minimization heuristic) achieves lower mean-square error and lower rank in low-rank matrix reconstruction than ScaledGD, AccAltProj, and fRPCA across all trials.
• The heuristic produces sparser solutions than ScaledGD and AccAltProj, with consistently lower sparse matrix reconstruction error.
• For $ n = 10,000 $, Algorithm 1 computes high-quality solutions in under 10 minutes, demonstrating strong scalability.
• The semidefinite relaxation provides tight bounds and scales to $ n = 200 $, with solution gaps remaining small across test instances.
• Algorithm 2 (branch-and-bound) solves instances up to $ n = 25 $ to certifiable near-optimality, with the final solution only marginally better than the heuristic’s.
• The accelerated implementation of Algorithm 1 runs faster than the exact version while maintaining lower reconstruction error than ScaledGD, especially in high signal-to-noise regimes.