[Paper Review] The Noisy Power Method: A Meta Algorithm with Applications

Moritz Hardt, Eric Price|arXiv (Cornell University)|Nov 11, 2013

Sparse and Compressive Sensing Techniques18 references102 citations

TL;DR

This paper introduces the noisy power method, a robust meta-algorithm for computing dominant singular vectors of a matrix under significant noise, applicable to streaming PCA, matrix completion, and differentially private PCA. It provides a general convergence analysis that subsumes prior ad-hoc bounds and yields the first nearly-linear time differentially private PCA algorithm with nearly tight error bounds, improving on worst-case and average-case performance via matrix coherence dependence.

ABSTRACT

We provide a new robust convergence analysis of the well-known power method for computing the dominant singular vectors of a matrix that we call the noisy power method. Our result characterizes the convergence behavior of the algorithm when a significant amount noise is introduced after each matrix-vector multiplication. The noisy power method can be seen as a meta-algorithm that has recently found a number of important applications in a broad range of machine learning problems including alternating minimization for matrix completion, streaming principal component analysis (PCA), and privacy-preserving spectral analysis. Our general analysis subsumes several existing ad-hoc convergence bounds and resolves a number of open problems in multiple applications including streaming PCA and privacy-preserving singular vector computation.

Motivation & Objective

• To develop a general convergence analysis for the noisy power method that applies across diverse noisy matrix computation settings.
• To resolve open problems in streaming PCA by providing a simpler, more general analysis beyond the spiked covariance model.
• To design the first nearly-linear time differentially private PCA algorithm with nearly optimal error bounds.
• To show that error dependence on matrix dimension can be replaced by coherence, leading to improved average-case performance.

Proposed method

• The noisy power method iteratively applies matrix-vector multiplication with noise added after each step, maintaining orthonormality via QR factorization.
• The method is analyzed under general noise assumptions, including adversarial and adaptive perturbations after each matrix-vector product.
• A key technique is the use of matrix Chernoff bounds to control concentration of noise in the convergence analysis.
• The analysis leverages sign symmetry and rotational invariance of Gaussian noise to bound the infinity norm of iterates.
• A deterministic function is defined to describe the algorithm's behavior under i.i.d. Gaussian noise, enabling distributional analysis.
• Orthonormal invariance properties are used to show that the algorithm's output distribution is invariant under orthogonal transformations of the input matrix.

Experimental results

Research questions

• RQ1 Can a single robust convergence analysis unify existing ad-hoc analyses of the power method under noise?
• RQ2 Does the noisy power method converge globally for arbitrary initial subspaces and general noise models?
• RQ3 Can the noisy power method be used to achieve nearly-optimal error bounds in differentially private PCA?
• RQ4 Can the dependence of error on matrix dimension be replaced by a tighter dependence on matrix coherence in private PCA?
• RQ5 Does the noisy power method provide provable convergence guarantees in streaming PCA beyond the Gaussian spiked covariance model?

Key findings

• The noisy power method achieves global convergence with high probability when noise is bounded relative to the spectral gap, even with a random initial subspace.
• For streaming PCA, the analysis confirms convergence under arbitrary distributions, not just the spiked covariance model, and improves on prior bounds in natural parameter regimes.
• The paper provides the first nearly-linear time differentially private PCA algorithm with nearly tight worst-case error bounds.
• The error dependence on matrix dimension is replaced by a dependence on coherence, which is often much smaller, leading to strong average-case improvements.
• The convergence rate is O(σₖ / (σₖ − σₖ₊₁)) log(dτ/ε), which matches the optimal rate up to logarithmic factors.
• The analysis shows that the algorithm's output is invariant under orthogonal transformations of the input matrix, enabling distributional symmetry arguments.