[Paper Review] A fast two-stage algorithm for non-negative matrix factorization in streaming data

Ran Gu, Qiang Du+1 more|arXiv (Cornell University)|Jan 21, 2021

Blind Source Separation Techniques32 references4 citations

TL;DR

This paper proposes a fast two-stage algorithm for non-negative matrix factorization (NMF) in streaming data, combining alternating non-negative least squares (ANLS) with an active set method in the first stage and a line search interior point method in the second. The method achieves high-precision solutions up to 10x faster than existing algorithms, particularly effective in the regime $n \gg m \gg k$ with $n \leq 5000$, $m \leq 200$, and $k \leq 10$. The algorithm leverages data continuity for efficient active set updates and accelerates local convergence via Newton-type steps.

ABSTRACT

In this article, we study algorithms for nonnegative matrix factorization (NMF) in various applications involving streaming data. Utilizing the continual nature of the data, we develop a fast two-stage algorithm for highly efficient and accurate NMF. In the first stage, an alternating non-negative least squares (ANLS) framework is used, in combination with active set method with warm-start strategy for the solution of subproblems. In the second stage, an interior point method is adopted to accelerate the local convergence. The convergence of the proposed algorithm is proved. The new algorithm is compared with some existing algorithms in benchmark tests using both real-world data and synthetic data. The results demonstrate the advantage of our algorithm in finding high-precision solutions.

Motivation & Objective

• To address the challenge of efficiently computing high-precision non-negative matrix factorization (NMF) for streaming data with high-dimensional individual measurements ($n \gg k$) and limited time points ($m \gg k$).
• To exploit the continual nature of streaming data—where data evolve smoothly over time—to accelerate convergence via warm-start strategies and active set updates.
• To combine the fast initial descent of ANLS with the rapid local convergence of interior point methods for improved efficiency and accuracy.
• To develop a computationally scalable algorithm with complexity $O(nm^2k^3)$ per iteration, suitable for real-time applications such as materials synthesis monitoring.

Proposed method

• The first stage employs an alternating non-negative least squares (ANLS) framework with an active set method and warm-start strategy, exploiting temporal continuity in streaming data to reduce active set changes.
• The active set method efficiently solves the non-negativity constrained subproblems in ANLS by tracking active and inactive constraints, with updates informed by previous solutions.
• The second stage transitions to a line search interior point method, which provides superlinear local convergence by leveraging Newton-type search directions.
• The interior point method is optimized for streaming data, with computational complexity scaled to $O(nm^2k^3)$ per iteration, enabling fast convergence near optimal solutions.
• A two-stage transition mechanism is implemented, where the ANLS solution is used as an initial guess for the interior point method, with careful parameter tuning to avoid instability.
• Global convergence of the line search interior point method is theoretically proven, ensuring robustness in local optimization.

Experimental results

Research questions

• RQ1 Can a two-stage optimization strategy combining ANLS with active sets and interior point methods significantly improve the speed and accuracy of NMF for streaming data?
• RQ2 How does the use of warm-start and active set updates exploit temporal continuity in streaming data to reduce computational overhead?
• RQ3 To what extent does the interior point method accelerate local convergence in NMF when initialized from ANLS solutions?
• RQ4 What is the computational complexity and scalability of the proposed algorithm in the regime $n \gg m \gg k$?
• RQ5 How does the proposed algorithm compare in precision and speed to existing NMF methods like NeNMF, QRPBB, and ANLS-BPP on synthetic and real-world streaming data?

Key findings

• The proposed two-stage algorithm achieves the highest solution precision among all tested methods, with objective function errors as low as $3.07 \times 10^{-7}$ on synthetic data.
• On synthetic data with dimensions $n=2000$, $m=50$, $k=3$, the algorithm ran in 3.31 seconds on average, outperforming NeNMF (60.01 s), QRPBB (29.36 s), and ANLS-BPP (58.48 s).
• For larger $m=100$, the algorithm maintained high efficiency, solving $n=2000$, $m=100$, $k=6$ problems in 25.54 seconds on average, while NeNMF and QRPBB required nearly 60 seconds.
• The algorithm demonstrated consistent convergence to the same objective value across all test cases, indicating robustness and high-precision solution quality.
• The interior point stage significantly accelerated convergence near the optimum, with the fastest local convergence observed in the final phase of optimization.
• Despite challenges in transition stability, careful parameter selection enabled reliable performance, suggesting potential for further improvement with better initialization techniques.