[Paper Review] Improved Distributed Principal Component Analysis

Maria-Florina Balcan, Vandana Kanchanapally+2 more|arXiv (Cornell University)|Aug 25, 2014

Neural Networks and ApplicationsComputer Science6 references65 citations

TL;DR

This paper proposes a communication-efficient distributed PCA algorithm that reduces communication cost from O(skd/ε³) to O(skd/ε²) for k-means clustering and related ℓ₂-fitting problems, while maintaining (1+ε)-approximate solution quality. It introduces a novel close projection property and randomized subspace embedding techniques that enable significant speedups without sacrificing accuracy, validated on real-world datasets with up to 100× faster computation.

ABSTRACT

We study the distributed computing setting in which there are multiple servers, each holding a set of points, who wish to compute functions on the union of their point sets. A key task in this setting is Principal Component Analysis (PCA), in which the servers would like to compute a low dimensional subspace capturing as much of the variance of the union of their point sets as possible. Given a procedure for approximate PCA, one can use it to approximately solve $\ell_2$-error fitting problems such as $k$-means clustering and subspace clustering. The essential properties of an approximate distributed PCA algorithm are its communication cost and computational efficiency for a given desired accuracy in downstream applications. We give new algorithms and analyses for distributed PCA which lead to improved communication and computational costs for $k$-means clustering and related problems. Our empirical study on real world data shows a speedup of orders of magnitude, preserving communication with only a negligible degradation in solution quality. Some of these techniques we develop, such as a general transformation from a constant success probability subspace embedding to a high success probability subspace embedding with a dimension and sparsity independent of the success probability, may be of independent interest.

Motivation & Objective

• Address the high communication and computational costs of existing distributed PCA protocols for large-scale data analysis.
• Reduce communication overhead in distributed k-means clustering and ℓ₂-fitting problems by improving the dimensionality reduction step.
• Develop a practical, scalable algorithm that maintains high solution quality while drastically reducing runtime in distributed settings.
• Introduce a new theoretical property—close projection—that enables efficient approximation of global PCA without full SVD on each server.

Proposed method

• Propose a new distributed PCA protocol that computes O(k/ε²) vectors via Algorithm disPCA with reduced communication cost O(skd/ε²).
• Introduce the close projection property: the projection of global data onto the computed subspace closely approximates the projection onto any k-dimensional subspace.
• Use a general transformation from constant-success-probability subspace embeddings to high-success-probability embeddings with dimension and sparsity independent of success probability.
• Leverage randomized SVD and subspace embedding techniques to accelerate computation while preserving accuracy.
• Design a distributed framework where each server computes local PCA, and the coordinator combines results using the close projection property.
• Implement and evaluate the algorithm on real-world datasets using a star network topology with weighted data partitioning across servers.

Experimental results

Research questions

• RQ1 Can the communication cost of distributed PCA for k-means clustering be reduced below O(skd/ε³) without sacrificing solution quality?
• RQ2 Does the close projection property enable a (1+ε)-approximate solution to k-means clustering using only O(k/ε²) global vectors instead of full SVD?
• RQ3 Can randomized techniques significantly reduce computational time in distributed PCA while maintaining near-optimal accuracy?
• RQ4 How does the proposed method scale on large-scale datasets such as BOWpubmed with 8.2M samples and 141K features?
• RQ5 To what extent do speedup techniques degrade solution quality in low-rank approximation, k-means, and PCR tasks?

Key findings

• The proposed algorithm reduces communication cost from O(skd/ε³) to O(skd/ε²), achieving a quadratic improvement in ε dependence.
• On the NewsGroups dataset, the fast distributed PCA achieves a 10-100× speedup over standard distributed PCA with negligible accuracy loss.
• For the large BOWpubmed dataset (8.2M samples), standard PCA failed to complete within 24 hours, but the fast version produced high-quality results in reasonable time.
• The error ratio for low-rank approximation, k-means clustering, and PCR remained comparable to the baseline SVD solution across all datasets and projection dimensions.
• The use of randomized SVD and subspace embedding reduced runtime significantly while maintaining solution quality within a negligible margin of error.
• The close projection property ensures that any k-dimensional subspace approximation on the projected data yields a (1+ε)-approximate solution to the global problem.