[Paper Review] Sparse and Unique Nonnegative Matrix Factorization Through Data Preprocessing

Nicolas Gillis|arXiv (Cornell University)|Apr 11, 2012

Sparse and Compressive Sensing Techniques40 references110 citations

TL;DR

This paper proposes a novel data preprocessing technique to enhance nonnegative matrix factorization (NMF) by improving sparsity and uniqueness. By transforming the input matrix using inverse-positive M-matrices, the method provably yields sparser and more well-posed NMF solutions under separability and rank-three conditions, with empirical validation on image datasets showing competitive performance without hyperparameter tuning.

ABSTRACT

Nonnegative matrix factorization (NMF) has become a very popular technique in machine learning because it automatically extracts meaningful features through a sparse and part-based representation. However, NMF has the drawback of being highly ill-posed, that is, there typically exist many different but equivalent factorizations. In this paper, we introduce a completely new way to obtaining more well-posed NMF problems whose solutions are sparser. Our technique is based on the preprocessing of the nonnegative input data matrix, and relies on the theory of M-matrices and the geometric interpretation of NMF. This approach provably leads to optimal and sparse solutions under the separability assumption of Donoho and Stodden (NIPS, 2003), and, for rank-three matrices, makes the number of exact factorizations finite. We illustrate the effectiveness of our technique on several image datasets.

Motivation & Objective

• To address the ill-posedness and non-uniqueness of standard NMF, which leads to multiple equivalent solutions.
• To improve solution sparsity and well-posedness in NMF through data preprocessing rather than optimization penalties.
• To establish theoretical guarantees for sparsity and finiteness of solutions under separability and low-rank conditions.
• To develop a preprocessing framework that leverages M-matrices and inverse-positive matrices to transform the input data.
• To demonstrate empirical effectiveness on facial and hyperspectral image datasets with competitive performance against state-of-the-art sparse NMF methods.

Proposed method

• The method applies a preprocessing transformation using inverse-positive matrices, specifically M-matrices, to the input nonnegative matrix $ M $.
• It transforms $ M $ into a new matrix $ \mathcal{P}(M) $ such that $ \mathcal{P}(M) = M Q $, where $ Q $ is inverse-positive and $ Q^{-1} $ is an M-matrix.
• The preprocessing ensures that $ \mathcal{P}(M) $ is nonnegative and has reduced column norms, promoting sparsity in subsequent NMF.
• The approach is grounded in the geometric interpretation of NMF via convex hulls and nested polytopes, where $ \theta(M) $ lies within $ \operatorname{conv}(\theta(U)) \subseteq \Delta^m $.
• For rank-three matrices, the preprocessing ensures the set of exact NMF solutions is finite, and generically unique.
• A generalized preprocessing variant allows $ \mathcal{P}(M) $ to be nonnegative only in a relaxed sense, improving robustness to noise and sparsity.

Experimental results

Research questions

• RQ1 Can data preprocessing be used to make NMF solutions sparser and more unique without modifying the optimization objective?
• RQ2 Under what conditions does preprocessing via inverse-positive matrices yield provably optimal and sparse NMF solutions?
• RQ3 Does preprocessing ensure finitely many exact NMF solutions for rank-three matrices?
• RQ4 Can the preprocessing strategy be generalized to handle noisy and sparse data while maintaining sparsity and uniqueness?
• RQ5 How does the preprocessing compare to standard sparse NMF techniques that rely on $ \ell_1 $-penalties in terms of performance and parameter tuning?

Key findings

• Under the separability assumption of Donoho and Stodden, the preprocessing identifies the vertices of the convex hull of $ M $'s columns, yielding optimal and sparse solutions.
• For any rank-two matrix, the preprocessing is optimal due to the inherent satisfaction of the separability condition.
• In the exact rank-three case, the preprocessing ensures the set of exact NMF solutions is finite, with strong evidence suggesting uniqueness under generic conditions.
• The method produces sparser NMF solutions than standard approaches and performs competitively with $ \ell_1 $-penalized sparse NMF, without requiring parameter tuning.
• The computational cost is high, scaling as $ \mathcal{O}(n^{4.5}) $, due to solving $ n $ CLLS problems, but can be reduced via heuristic column subset selection.
• A counterexample shows that not all non-unique NMF problems are resolved by the current M-matrix framework, suggesting potential for broader inverse-positive matrix classes to further improve sparsity.