[Paper Review] Improved Analyses of the Randomized Power Method and Block Lanczos Method

Shusen Wang, Zhihua Zhang+1 more|arXiv (Cornell University)|Aug 26, 2015

Stochastic Gradient Optimization Techniques16 references13 citations

TL;DR

This paper presents improved error bounds for the randomized power method and block Lanczos method in low-rank matrix approximation, using novel analysis techniques based on Chebyshev polynomials and Krylov subspace projections. It establishes the first gap-independent spectral norm error bound for the warm-start block Lanczos method, significantly tightening the convergence guarantees beyond prior work.

ABSTRACT

The power method and block Lanczos method are popular numerical algorithms for computing the truncated singular value decomposition (SVD) and eigenvalue decomposition problems. Especially in the literature of randomized numerical linear algebra, the power method is widely applied to improve the quality of randomized sketching, and relative-error bounds have been well established. Recently, Musco & Musco (2015) proposed a block Krylov subspace method that fully exploits the intermediate results of the power iteration to accelerate convergence. They showed spectral gap-independent bounds which are stronger than the power method by order-of-magnitude. This paper offers novel error analysis techniques and significantly improves the bounds of both the randomized power method and the block Lanczos method. This paper also establishes the first gap-independent bound for the warm-start block Lanczos method.

Motivation & Objective

• To provide tighter, gap-independent error bounds for the randomized power method and block Lanczos method in truncated SVD and eigenvalue problems.
• To address the limitation of existing relative-error bounds that depend on spectral gaps, which can be poor in practice.
• To establish the first gap-independent bound for the warm-start block Lanczos method, improving convergence guarantees.
• To analyze the role of block size and sketching efficiency in large-scale data applications where I/O costs dominate.
• To offer a theoretical foundation for using intermediate power iterates as a Krylov subspace sketch, enhancing convergence without reorthogonalization.

Proposed method

• Uses Krylov subspace structure formed by intermediate power iterates $\mathbf{K} = [\mathbf{M}\mathbf{X}, (\mathbf{M}\mathbf{M}^T)\mathbf{M}\mathbf{X}, \dots, (\mathbf{M}\mathbf{M}^T)^d\mathbf{M}\mathbf{X}]$ as a sketch of $\mathbf{M}$, replacing standard random projections.
• Applies Chebyshev polynomial approximation theory to show that the Krylov subspace captures the top $k$-dimensional singular subspace of $\mathbf{M}$ with high accuracy.
• Introduces a novel matrix perturbation analysis using the tangent of the canonical angle $\tan \theta(\mathbf{V}_1, \mathbf{X}\mathbf{Z})$ to bound residual error.
• Employs a variant of Lemma 3.1 from [2] to bound the error in the projection of $\mathbf{M}_1$ onto the sketch, relating it to the norm of the residual $\mathbf{M}_2 + \mathbf{M}_3$.
• Derives error bounds in both spectral and Frobenius norms, improving the prior $ (1+\epsilon)\sigma_{k+1}^2 $ bound to $ \sigma_{k+1}^2 + \epsilon \sigma_{p+1}^2 $.
• Analyzes the trade-off between block size $p$ and iteration count $d$, showing that larger $p$ reduces iterations and improves I/O efficiency in big data settings.

Experimental results

Research questions

• RQ1 Can the convergence of the randomized power method be improved beyond the standard $ (1+\epsilon)\sigma_{k+1}^2 $ spectral error bound?
• RQ2 Can gap-independent error bounds be established for the block Lanczos method, especially in the warm-start setting?
• RQ3 How does the use of intermediate power iterates as a Krylov subspace sketch affect approximation quality and convergence rate?
• RQ4 What is the impact of block size $p$ on the total computational cost and numerical stability in large-scale low-rank approximation?
• RQ5 Can the theoretical analysis of Krylov subspace projection via Chebyshev polynomials be extended to provide tighter bounds than existing randomized SVD methods?

Key findings

• The paper improves the spectral norm error bound for the randomized power method from $ (1+\epsilon)\sigma_{k+1}^2 $ to $ \sigma_{k+1}^2 + \epsilon \sigma_{p+1}^2 $, which is tighter and gap-independent.
• It establishes the first gap-independent relative-error bound for the warm-start block Lanczos method, showing that $ d = \mathcal{O}(\log n / \sqrt{\epsilon}) $ iterations suffice for $ (1+\epsilon) $-relative error in spectral norm.
• The analysis shows that the Krylov subspace formed by intermediate power iterates captures the top $k$-dimensional subspace of $\mathbf{M}$ with high accuracy, even without reorthogonalization.
• The method achieves pass-efficiency in big data settings by using large block sizes $p \geq k$, reducing the number of passes over data and minimizing I/O costs.
• The error bound depends on $ \|\mathbf{M}_3\|_\xi^2 \tan^2 \theta(\mathbf{V}_1, \mathbf{X}\mathbf{Z}) $, which quantifies the alignment between the sketch and the true subspace, providing a refined error characterization.
• The theoretical framework supports practical deployment in low-precision settings where numerical instability is less of a concern, validating the empirical success of Musco & Musco's approach without additional numerical stabilization.