[论文解读] Tutorial: Complexity analysis of Singular Value Decomposition and its variants

Xiaocan Li, Shuo Wang+1 more|arXiv (Cornell University)|Jun 28, 2019

Blind Source Separation Techniques参考文献 22被引用 46

一句话总结

本文在时间和空间复杂度方面比较了常规 SVD、截断 SVD、Krylov 方法和 Randomized PCA，结果显示在需要全部特征对时，截断 SVD 最快。文中还探讨了 PCA 与 SVD 的关系，以及在何时有利于使用随机化/ Krylov 方法。

ABSTRACT

We compared the regular Singular Value Decomposition (SVD), truncated SVD, Krylov method and Randomized PCA, in terms of time and space complexity. It is well-known that Krylov method and Randomized PCA only performs well when k << n, i.e. the number of eigenpair needed is far less than that of matrix size. We compared them for calculating all the eigenpairs. We also discussed the relationship between Principal Component Analysis and SVD.

研究动机与目标

• Motivate the study of scalable SVD variants for large matrices.
• Explain the relationship between PCA and SVD and why it matters for dimensionality reduction.
• Provide a detailed complexity analysis (time and space) for Krylov, Randomized PCA, and truncated SVD.
• Empirically compare the methods on synthetic data and MNIST to validate theoretical insights.

提出的方法

• Derive SVD basics and connect to PCA via covariance properties.
• Explain truncated SVD as avoiding large AA^T by using A^T A and reconstructing U.
• Present detailed FLOP-based time complexity derivations for Krylov, Randomized PCA, and truncated SVD.
• Present detailed space complexity derivations for each method, including practical memory estimates.
• Summarize results in a comparative table and corroborate with experiments on synthetic data and MNIST.

实验结果

研究问题

• RQ1 Which SVD variant offers the best time efficiency when all eigenpairs are required?
• RQ2 How do time and space complexities scale for Krylov, Randomized PCA, and truncated SVD as m >> n?
• RQ3 What are the practical memory trade-offs between these methods on large-scale data (e.g., MNIST)?
• RQ4 How do the methods compare empirically in runtime and memory usage on synthetic and real datasets?

主要发现

• Truncated SVD is the fastest when all eigenpairs are needed (k = n).
• Krylov method generally consumes the most memory among the three analyzed approaches.
• Randomized PCA has intermediate memory usage and favorable performance when only a subset k << n is needed.
• For large m >> n, the asymptotic time costs favor truncated SVD over Krylov and Randomized PCA in full-eigenpair computation.
• Experiments on MNIST show truncated SVD outperforms Krylov and Randomized PCA in both runtime and memory for full PCA.
• The study reinforces the equivalence between PCA and SVD in terms of projection bases (U and V) when using orthogonal decompositions.