[Paper Review] Dimension Independent Matrix Square using MapReduce

Reza Bosagh Zadeh, Gunnar Carlsson|arXiv (Cornell University)|Apr 4, 2013

Stochastic Gradient Optimization Techniques29 references18 citations

TL;DR

This paper presents DIMSUM, a dimension-independent algorithm for efficiently computing the matrix square $A^T A$ in a distributed MapReduce environment using nonadaptive sampling. It achieves relative error guarantees with shuffle size $O(n^2 / \epsilon^2)$ and reduce-key complexity $O(n / \epsilon^2)$, independent of $m$, enabling scalable SVD computation on massive sparse matrices with $m \gg n$. The method preserves singular values with high probability and is optimized for non-negative matrices, reducing shuffle size to $O(n \log n / s)$ when estimating high-similarity entries, where $s$ is the minimum cosine similarity.

ABSTRACT

We compute the singular values of an $m imes n$ sparse matrix $A$ in a distributed setting, without communication dependence on $m$, which is useful for very large $m$. In particular, we give a simple nonadaptive sampling scheme where the singular values of $A$ are estimated within relative error with constant probability. Our proven bounds focus on the MapReduce framework, which has become the de facto tool for handling such large matrices that cannot be stored or even streamed through a single machine. On the way, we give a general method to compute $A^TA$. We preserve singular values of $A^TA$ with $ε$ relative error with shuffle size $O(n^2/ε^2)$ and reduce-key complexity $O(n/ε^2)$. We further show that if only specific entries of $A^TA$ are required and $A$ has nonnegative entries, then we can reduce the shuffle size to $O(n \log(n) / s)$ and reduce-key complexity to $O(\log(n)/s)$, where $s$ is the minimum cosine similarity for the entries being estimated. All of our bounds are independent of $m$, the larger dimension. We provide open-source implementations in Spark and Scalding, along with experiments in an industrial setting.

Motivation & Objective

• To address the scalability bottleneck in computing $A^T A$ for very large $m \times n$ sparse matrices where $m$ is too large to store or stream on a single machine.
• To eliminate communication and computational dependence on $m$ in distributed SVD computation, enabling efficient singular value estimation in high-dimensional data settings.
• To provide a provably accurate, nonadaptive sampling scheme that maintains relative error bounds for $A^T A$ entries using only $O(n^2 / \epsilon^2)$ shuffle size and $O(n / \epsilon^2)$ reduce-key complexity.
• To optimize for non-negative matrices by reducing shuffle size to $O(n \log n / s)$ and reduce-key complexity to $O(\log n / s)$ when estimating entries with cosine similarity $\geq s$, improving efficiency for high-similarity pairs.

Proposed method

• Proposes a nonadaptive sampling scheme where each row of $A$ is processed via a mapper that emits weighted products of column pairs based on normalized magnitudes and random scaling.
• Uses a randomized estimator for entries of $A^T A$ such that the expected value of the reducer output equals the normalized dot product (cosine similarity) between columns.
• Employs a spectral norm concentration inequality from [23] to prove that the estimated $A^T A$ entries are within $\epsilon$ relative error with constant probability.
• Introduces a leaner variant (LeanDIMSUM) that reduces random number generation per row by using summation reducers, though pairwise independence is relaxed.
• Applies Chernoff bounds to show that the shuffle size is $O(n^2 / \epsilon^2)$ with high probability, ensuring scalability.
• Optimizes for non-negative matrices by thresholding column norms via $\min(1, \sqrt{\gamma}/\|c_j\|)$, reducing redundant emissions for high-magnitude columns.

Experimental results

Research questions

• RQ1 Can $A^T A$ be computed in a distributed MapReduce setting with communication and computation independent of the large dimension $m$?
• RQ2 What is the minimal shuffle size and reduce-key complexity required to estimate $A^T A$ entries with relative error $\epsilon$?
• RQ3 Can the algorithm be optimized further for non-negative matrices by exploiting high cosine similarity among column pairs?
• RQ4 Does nonadaptive sampling with randomized scaling preserve the singular values of $A$ within relative error bounds with constant probability?
• RQ5 What are the theoretical lower bounds on shuffle size for estimating high-similarity entries in $A^T A$, and how does DIMSUM compare?

Key findings

• The shuffle size for estimating $A^T A$ entries with $\epsilon$ relative error is $O(n^2 / \epsilon^2)$, independent of $m$, which is a significant improvement over the naive $O(mL^2)$ approach.
• The reduce-key complexity is $O(n / \epsilon^2)$, which is independent of $m$ and feasible even for $m = 10^{13}$, enabling practical distributed computation.
• For non-negative matrices, the shuffle size can be reduced to $O(n \log n / s)$ and reduce-key complexity to $O(\log n / s)$ when estimating entries with cosine similarity $\geq s$, improving efficiency for high-similarity pairs.
• The algorithm achieves relative error bounds in the spectral norm with constant probability, proven using a novel singular value concentration inequality from [23].
• Experiments at Twitter showed a 40% improvement in performance metrics after deploying DIMSUM in production, validating its real-world scalability and efficiency.
• Open-source implementations in Spark and Scalding have been integrated into major commercial distributions, confirming practical adoption and robustness.