[Paper Review] Efficient Thresholded Correlation using Truncated Singular Value Decomposition

James Baglama, Michael J. Kane+2 more|arXiv (Cornell University)|Dec 22, 2015

Blind Source Separation Techniques8 references3 citations

TL;DR

This paper proposes a novel algorithm using truncated singular value decomposition (SVD) to efficiently compute thresholded subsets of large correlation matrices by leveraging the geometric relationship between correlation and Euclidean distance in low-rank subspaces. The method drastically reduces computation and memory usage by pruning non-viable vector pairs early via SVD-based projections, enabling the discovery of 913,601 highly correlated pairs (r ≥ 0.99) from a 394,014-column matrix in under 8 minutes on a modest cluster, compared to billions of pairs in brute-force approaches.

ABSTRACT

Efficiently computing a subset of a correlation matrix consisting of values above a specified threshold is important to many practical applications. Real-world problems in genomics, machine learning, finance other applications can produce correlation matrices too large to explicitly form and tractably compute. Often, only values corresponding to highly-correlated vectors are of interest, and those values typically make up a small fraction of the overall correlation matrix. We present a method based on the singular value decomposition (SVD) and its relationship to the data covariance structure that can efficiently compute thresholded subsets of very large correlation matrices.

Motivation & Objective

• To address the computational infeasibility of computing all pairwise correlations in large-scale data matrices, especially in genomics, machine learning, and finance.
• To develop a method that efficiently identifies only the subset of vector pairs with correlation values above a user-defined threshold, avoiding full correlation matrix computation.
• To exploit the covariance structure of data via truncated SVD to enable early pruning of non-viable pairs based on projected distances in low-dimensional subspaces.
• To design a parallelizable algorithm that scales efficiently across multi-core and distributed computing environments without code changes.
• To provide a practical, open-source R implementation that supports thresholded correlation and distance computation for real-world data workloads.

Proposed method

• The method uses truncated SVD to project data vectors into a low-rank subspace defined by the top k singular vectors, where k is chosen to preserve dominant covariance structure.
• It leverages Lemma 1.1 to equate correlation to a function of squared Euclidean distance: cor(a_i, a_j) = 1 - ||a_i - a_j||² / 2, enabling distance-based pruning.
• Lemma 1.2 expresses the squared Euclidean distance between any two vectors as a weighted sum of squared projections onto SVD basis vectors, with weights σ_j² decreasing with j.
• The algorithm prunes pairs early if their projected distance in the top-k SVD subspace exceeds the threshold-defined bound, avoiding full correlation computation.
• The filtering step is parallelized using R’s foreach framework with a doRedis back end, enabling distribution across multiple nodes with minimal code changes.
• The method supports both thresholded correlation and thresholded Euclidean distance, with extensions to finding the top-N most correlated pairs.

Experimental results

Research questions

• RQ1 Can truncated SVD be used to efficiently identify only the most highly correlated vector pairs in a large correlation matrix without computing all pairwise correlations?
• RQ2 How can the geometric relationship between correlation and Euclidean distance be exploited to reduce computational and memory costs in high-dimensional data?
• RQ3 To what extent can early pruning based on low-rank SVD projections reduce the number of candidate pairs that require full correlation evaluation?
• RQ4 Can the core algorithm be efficiently parallelized across multi-core and distributed clusters with minimal architectural changes?
• RQ5 How does the performance of the SVD-based thresholded correlation method compare to brute-force computation on real-world genomics datasets?

Key findings

• The algorithm identified 913,601 vector pairs with correlation ≥ 0.99 from a 394,014-column methylation dataset in approximately 8 minutes on a 64-core cluster, reducing the search space from over 77 billion pairs to about 4 million candidates.
• On a standard desktop PC, the same algorithm completed the same task in about 3 hours, demonstrating robustness under memory-constrained conditions.
• The cluster version used 994.3 MB of peak memory, with 16 GB overhead due to data replication across 64 worker processes, indicating room for memory optimization.
• The algorithm’s filtering step was fully parallelized using R’s foreach and doRedis back end, requiring no code changes to scale across 64 cores.
• The method achieved a 95% reduction in computation time compared to brute-force evaluation, which would require evaluating over 77 billion pairwise correlations.
• The approach is extensible to thresholded Euclidean distance and top-N correlation retrieval, with functions included in the open-source R implementation.