[Paper Review] Fast and Faster: A Comparison of Two Streamed Matrix Decomposition Algorithms

Radim Řehůřek|arXiv (Cornell University)|Feb 28, 2011

Data Mining Algorithms and Applications4 citations

TL;DR

This paper compares a one-pass distributed matrix decomposition algorithm (P1) with a two-pass stochastic algorithm (P2), introducing a novel hybrid (P12) that combines the speed of P2 with the one-pass accuracy of P1. On the English Wikipedia corpus, P12 achieved a decomposition in 4h20m on a single machine and 1h41m on a 4-node cluster, outperforming P2 in speed while maintaining competitive accuracy when optimized with oversampling and power iterations.

ABSTRACT

With the explosion of the size of digital dataset, the limiting factor for decomposition algorithms is the \\emph{number of passes} over the input, as the input is often stored out-of-core or even off-site. Moreover, we're only interested in algorithms that operate in \\emph{constant memory} w.r.t. to the input size, so that arbitrarily large input can be processed. In this paper, we present a practical comparison of two such algorithms: a distributed method that operates in a single pass over the input vs. a streamed two-pass stochastic algorithm. The experiments track the effect of distributed computing, oversampling and memory trade-offs on the accuracy and performance of the two algorithms. To ensure meaningful results, we choose the input to be a real dataset, namely the whole of the English Wikipedia, in the application settings of Latent Semantic Analysis.

Motivation & Objective

• To evaluate the trade-offs between one-pass and two-pass streamed matrix decomposition algorithms in terms of accuracy, performance, and scalability.
• To investigate the impact of input order, oversampling, and power iterations on decomposition quality in streaming settings.
• To design and implement a hybrid algorithm (P12) that combines the speed of the two-pass method with the one-pass memory efficiency and accuracy of the distributed one-pass approach.
• To assess the effectiveness of distributed computing on the performance and scalability of the algorithms using a real-world, large-scale dataset (English Wikipedia).

Proposed method

• Implement a one-pass distributed algorithm (P1) that processes document chunks in parallel, performs in-core SVD on each chunk using SVDLIBC, and merges the results into a global decomposition.
• Adapt the two-pass stochastic algorithm (P2) from Halko et al. (2009) to operate in constant memory by streaming observations and computing the sample matrix incrementally, avoiding full matrix multiplication.
• Develop a hybrid algorithm (P12) that replaces the in-core SVD in P1 with the faster stochastic decomposition from P2, preserving one-pass processing while improving speed.
• Use oversampling and power iterations in P2 to enhance accuracy at the cost of additional passes, and evaluate their impact on performance and convergence.
• Apply the algorithms to a real-world dataset—3.2 million English Wikipedia documents—for Latent Semantic Analysis (LSA), using TF-IDF vectorization.
• Evaluate performance across varying chunk sizes, input orderings, and cluster configurations (1–4 nodes), measuring both wall-clock time and decomposition accuracy via singular value analysis.

Experimental results

Research questions

• RQ1 How does the accuracy of the one-pass distributed algorithm (P1) compare to the two-pass stochastic algorithm (P2) under identical conditions on a large-scale real-world dataset?
• RQ2 To what extent does input stream ordering affect the quality of the decomposition in one-pass algorithms, particularly in the absence of randomization?
• RQ3 How effective are oversampling and power iterations in improving the accuracy of the two-pass stochastic algorithm (P2), and what is the performance cost?
• RQ4 Can a hybrid algorithm (P12) successfully combine the speed of P2 with the one-pass accuracy of P1, and how does it scale across distributed clusters?
• RQ5 What is the impact of chunk size and cluster size on the runtime and accuracy of the one-pass algorithms (P1 and P12) in a distributed environment?

Key findings

• The hybrid P12 algorithm achieved a decomposition time of 4 hours and 20 minutes on a single 2GHz machine, outperforming the two-pass P2 algorithm (3 hours 6 minutes) when optimized with oversampling and power iterations.
• On a 4-node cluster, the P12 algorithm completed the decomposition in just 1 hour and 41 minutes, demonstrating strong linear scaling with the number of computing nodes.
• The one-pass P1 algorithm showed significant sensitivity to input order: singular values from alphabetically ordered input were less accurate than those from shuffled input, indicating subspace drift issues.
• Oversampling and power iterations in P2 improved accuracy substantially, but at the cost of increased runtime, highlighting the trade-off between accuracy and pass efficiency in streaming settings.
• The two-pass P2 algorithm did not benefit from distribution due to I/O bottlenecks from multiple passes, making it less suitable for networked or distributed environments unless data was already pre-distributed.
• The hybrid P12 algorithm maintained high accuracy comparable to P1 while achieving faster performance than P2, demonstrating the viability of combining stochastic speed with distributed one-pass robustness.