[Paper Review] Fast Conical Hull Algorithms for Near-separable Non-negative Matrix Factorization

Abhishek Kumar, Vikas Sindhwani+1 more|arXiv (Cornell University)|Oct 3, 2012

Matrix Theory and Algorithms24 references126 citations

TL;DR

This paper proposes Xray, a fast and scalable algorithm for near-separable non-negative matrix factorization (NMF) by reformulating the problem as finding extreme rays of a conical hull. The method uses a greedy, iterative selection process that exactly identifies anchor vectors in r iterations, achieving high scalability and noise robustness, with demonstrated 4.5x speedup on 8 cores for large-scale datasets like IBMT and RCV1.

ABSTRACT

The separability assumption (Donoho & Stodden, 2003; Arora et al., 2012) turns non-negative matrix factorization (NMF) into a tractable problem. Recently, a new class of provably-correct NMF algorithms have emerged under this assumption. In this paper, we reformulate the separable NMF problem as that of finding the extreme rays of the conical hull of a finite set of vectors. From this geometric perspective, we derive new separable NMF algorithms that are highly scalable and empirically noise robust, and have several other favorable properties in relation to existing methods. A parallel implementation of our algorithm demonstrates high scalability on shared- and distributed-memory machines.

Motivation & Objective

• Address the computational inefficiency of existing separable NMF algorithms that rely on solving multiple linear programs or iterative optimization with poor convergence guarantees.
• Develop a scalable and noise-robust algorithm for near-separable NMF that avoids heuristic local optimization and guarantees exact recovery under the separability assumption.
• Enable high-performance computation on large-scale datasets using shared- and distributed-memory parallel architectures.
• Provide a geometric interpretation of NMF as conical hull extreme ray detection to improve algorithmic clarity and robustness.

Proposed method

• Reformulate the separable NMF problem as identifying the extreme rays of the conical hull of the data matrix X, leveraging the geometric insight that anchor vectors correspond to these extreme points.
• Propose a greedy algorithm that iteratively selects the most extreme vector in the current conical hull, ensuring exact recovery in exactly r iterations under the separability assumption.
• Implement a parallel version using PFunc for shared-memory and MPI for distributed-memory systems, enabling efficient scaling on multi-core and cluster architectures.
• Optimize memory access and computational patterns by reorganizing operations around BLAS-3 routines and tuning data layouts to reduce thread contention.
• Use a modified version of the active-set method to efficiently solve the subproblems involved in extreme ray detection, avoiding full linear programming solvers.
• Integrate noise-robustness by incorporating a thresholding mechanism during vector selection, enhancing performance on real-world noisy data.

Experimental results

Research questions

• RQ1 Can the separable NMF problem be solved more efficiently by reformulating it as a conical hull extreme ray detection problem?
• RQ2 Does a greedy, iterative algorithm that identifies extreme rays in exactly r steps achieve exact recovery and scalability on large-scale datasets?
• RQ3 How does the proposed algorithm compare in performance and robustness to state-of-the-art methods like Hottopixx, especially under varying noise and data sparsity levels?
• RQ4 To what extent can parallelization on shared- and distributed-memory systems improve the runtime of conical hull-based NMF algorithms?
• RQ5 Can the algorithm maintain high accuracy and efficiency when applied to real-world text and image datasets with inherent noise and high dimensionality?

Key findings

• The Xray algorithm achieves exact recovery of anchor vectors in exactly r iterations under the separability assumption, ensuring correctness without convergence to local minima.
• On the RCV1 dataset, Xray completes factorization in 409 seconds using 8 cores, achieving a 4.2x speedup over the sequential implementation.
• For the IBMT Twitter dataset, Xray achieves a 4.5x speedup and completes factorization in 9.8 seconds on 8 cores, outperforming Hottopixx even after 5 epochs.
• The algorithm demonstrates strong scalability on both shared-memory (daniel) and distributed-memory systems, with performance improvements attributed to optimized BLAS-3 operations and memory layout.
• Xray shows superior performance on sparse datasets like RCV1 and IBMT compared to Hottopixx, completing factorization faster and with better accuracy, even when Hottopixx is run for more epochs.
• The method is empirically noise-robust, maintaining high accuracy on real-world datasets such as ClueWeb09 and PPL2, where it successfully extracts 100 topics from dense and sparse matrices alike.