[Paper Review] Non-Negative Matrix Factorization, Convexity and Isometry

Nikolaos Vasiloglou, Alexander Gray+1 more|ArXiv.org|Oct 13, 2008

Sparse and Compressive Sensing Techniques23 references4 citations

TL;DR

This paper introduces isometric NMF (isoNMF), a novel non-negative matrix factorization method that simultaneously preserves non-negativity and enforces isometric embedding to maintain pairwise distances in the low-dimensional space. By leveraging convex optimization theory and efficient algorithms, isoNMF achieves globally optimal solutions with improved interpretability, demonstrated by superior distance preservation and sparser, more compact spectra compared to standard NMF.

ABSTRACT

In this paper we explore avenues for improving the reliability of dimensionality reduction methods such as Non-Negative Matrix Factorization (NMF) as interpretive exploratory data analysis tools. We first explore the difficulties of the optimization problem underlying NMF, showing for the first time that non-trivial NMF solutions always exist and that the optimization problem is actually convex, by using the theory of Completely Positive Factorization. We subsequently explore four novel approaches to finding globally-optimal NMF solutions using various ideas from convex optimization. We then develop a new method, isometric NMF (isoNMF), which preserves non-negativity while also providing an isometric embedding, simultaneously achieving two properties which are helpful for interpretation. Though it results in a more difficult optimization problem, we show experimentally that the resulting method is scalable and even achieves more compact spectra than standard NMF.

Motivation & Objective

• To improve the reliability of NMF as an interpretive exploratory data analysis tool for non-negative data such as images, text, and spectra.
• To address the limitations of standard NMF, including non-convex optimization and poor distance preservation in low-dimensional plots.
• To develop a globally optimal NMF solution using convex optimization frameworks, particularly through Completely Positive Factorization theory.
• To introduce and validate a new method, isoNMF, that simultaneously ensures non-negativity and isometric embedding for enhanced interpretability.
• To demonstrate scalability and practical performance of isoNMF on real-world datasets, showing improved distance preservation and sparsity.

Proposed method

• Formulates the NMF problem using Completely Positive Factorization theory, proving the existence of non-trivial exact NMF solutions and establishing convexity of the underlying optimization problem.
• Explores four convex optimization approaches: convex relaxation using the positive semidefinite cone, approximating the semidefinite cone with smaller cones, convex multi-objective optimization, and generalized geometric programming.
• Develops isoNMF by integrating isometry constraints into the NMF framework, ensuring that pairwise distances in the original space are preserved in the low-dimensional embedding.
• Employs efficient neighborhood computation and optimization techniques to scale isoNMF to large datasets while maintaining numerical stability.
• Uses a spectrum metric defined as $ s_i = \frac{\sum_{l=1}^{N} W_{li}^2}{\sqrt{\sum_{l=1}^{M} H_{il}^2}} $ to compare component energy distributions between standard NMF and isoNMF.
• Applies the nmfpack algorithm for visualization and evaluates performance using reconstruction error, sparsity, and distance error metrics.

Experimental results

Research questions

• RQ1 Does every non-negative matrix admit a non-trivial exact non-negative matrix factorization, and is the NMF optimization problem convex?
• RQ2 Can convex optimization techniques be effectively applied to achieve globally optimal NMF solutions?
• RQ3 Can isometric embedding be integrated into NMF while preserving non-negativity and enabling practical scalability?
• RQ4 How does the proposed isoNMF method compare to standard NMF in terms of reconstruction error, sparsity, and distance preservation?
• RQ5 Does isoNMF yield more compact and interpretable spectral representations than standard NMF?

Key findings

• The NMF optimization problem is convex when formulated via Completely Positive Factorization, and non-trivial exact NMF solutions always exist for any non-negative matrix.
• isoNMF achieves near-perfect distance preservation, with distance error reduced to 0.01% on the ORL faces dataset and 0.03% on the statue dataset, compared to over 90% in standard NMF.
• isoNMF produces significantly sparser solutions, with sparsity increasing from 29.06% in standard NMF to 43.98% on the CBCL face dataset without preprocessing.
• Despite higher reconstruction error in some cases (e.g., 33.34% vs. 10.16% on preprocessed CBCL), isoNMF maintains comparable or better interpretability due to superior distance preservation and compact spectra.
• The spectrum of isoNMF is much more compact than standard NMF, indicating more focused and interpretable components, though the lack of orthogonality limits direct comparison to SVD-type spectra.
• isoNMF scales comparably to standard NMF and shows similar sensitivity to initialization, confirming practical feasibility and efficiency.