[Paper Review] Factoring nonnegative matrices with linear programs

Victor Bittorf, Benjamin Recht+2 more|arXiv (Cornell University)|Jun 6, 2012

Advanced Optimization Algorithms Research22 references113 citations

TL;DR

This paper introduces Hottopixx, a scalable linear programming-based algorithm for nonnegative matrix factorization (NMF) that identifies salient data rows to reconstruct the full matrix. It achieves provable recovery under similar conditions to Arora et al. (2012) but with improved error bounds, faster convergence, and robustness to noise, enabling efficient factorization of large-scale matrices in minutes using optimized C++ and parallel computing.

ABSTRACT

This paper describes a new approach, based on linear programming, for computing nonnegative matrix factorizations (NMFs). The key idea is a data-driven model for the factorization where the most salient features in the data are used to express the remaining features. More precisely, given a data matrix X, the algorithm identifies a matrix C such that X approximately equals CX and some linear constraints. The constraints are chosen to ensure that the matrix C selects features; these features can then be used to find a low-rank NMF of X. A theoretical analysis demonstrates that this approach has guarantees similar to those of the recent NMF algorithm of Arora et al. (2012). In contrast with this earlier work, the proposed method extends to more general noise models and leads to efficient, scalable algorithms. Experiments with synthetic and real datasets provide evidence that the new approach is also superior in practice. An optimized C++ implementation can factor a multigigabyte matrix in a matter of minutes.

Motivation & Objective

• Address the lack of theoretical guarantees and scalability in existing NMF heuristics, which are often computationally expensive and sensitive to noise.
• Develop a data-driven NMF method that identifies a minimal set of representative rows (features) from the data matrix to reconstruct the rest.
• Provide theoretical guarantees for recovery under the same modeling assumptions as Arora et al. (2012), with improved error bounds in high SNR regimes.
• Design a scalable, efficient algorithm suitable for large-scale datasets, including multigigabyte matrices, using linear programming and stochastic gradient descent.
• Extend the method to general noise models and eliminate the need for prior knowledge of noise parameters (e.g., ε and α) required by prior algorithms.

Proposed method

• Formulate NMF as a linear program to find a nonnegative matrix C such that X ≈ CX, where C selects rows from the data matrix X to act as basis features.
• Impose linear constraints on C to ensure it selects only a small, representative subset of rows (features), enforcing sparsity and interpretability.
• Use a stochastic gradient descent (SGD) algorithm to solve the linear program efficiently, achieving at least two orders of magnitude speedup over AGKM in Matlab.
• Implement a parallel, multicore-optimized C++ version of the algorithm to scale to matrices with up to 10^5 features and 10^6 examples.
• Introduce a margin constraint to improve robustness and ensure unique, well-defined solutions under the same conditions as AGKM.
• Adapt the framework to other factorization problems such as rank-revealing QR, interpolative decomposition, and dictionary learning via the same data-driven, feature-localizing principle.

Experimental results

Research questions

• RQ1 Can a linear programming formulation of NMF achieve provable recovery guarantees under the same modeling assumptions as the AGKM algorithm?
• RQ2 Does the proposed method improve error bounds compared to AGKM, particularly in high signal-to-noise ratio (SNR) regimes?
• RQ3 Can the algorithm be made scalable and efficient for large-scale datasets without requiring prior knowledge of noise parameters?
• RQ4 To what extent does the method generalize to different noise models and real-world data distributions?
• RQ5 How does the performance of the proposed algorithm compare to AGKM and other baselines in terms of reconstruction error and runtime on synthetic and real datasets?

Key findings

• The Hottopixx algorithm achieves the lowest $(\infty,1)$-norm reconstruction error among tested methods on synthetic data, outperforming AGKM even without prior knowledge of noise parameters.
• The algorithm achieves superlinear speedups (up to 20x) on multicore systems due to hardware prefetching and cache effects, enabling factorization of multigigabyte matrices in minutes.
• An optimized C++ implementation factors a 2.7GB synthetic matrix (1600×64000) in 338 seconds and a 1.14GB RCV1 dataset (47153×781265) in 430 seconds using 12 cores.
• With only 4 epochs, Hottopixx achieves competitive reconstruction error and outperforms AGKM in high-noise regimes (η ≥ 1), where AGKM degrades.
• On the RCV1 dataset, using 1500 Hottopixx-selected topics yields 7% misclassification error in an SVM classifier, compared to 5.5% using all features—demonstrating strong dimensionality reduction capability.
• The method successfully identifies 'hottopics' in the clueweb dataset, with RMSE dropping sharply in the first few hundred topics, indicating utility in NLP applications.