[Paper Review] Rate-Optimal Perturbation Bounds for Singular Subspaces with Applications to High-Dimensional Statistics

Tommaso Cai, Anru R. Zhang|arXiv (Cornell University)|May 2, 2016

Sparse and Compressive Sensing Techniques57 references19 citations

TL;DR

This paper establishes rate-optimal perturbation bounds for left and right singular subspaces of low-rank matrices under additive noise, using separate spectral and Frobenius sin Θ distance measures. It proves that the left and right singular spaces can have fundamentally different optimal convergence rates under the same perturbation, a phenomenon previously uncharacterized in high-dimensional statistics.

ABSTRACT

Perturbation bounds for singular spaces, in particular Wedin's $\sin Θ$ theorem, are a fundamental tool in many fields including high-dimensional statistics, machine learning, and applied mathematics. In this paper, we establish separate perturbation bounds, measured in both spectral and Frobenius $\sin Θ$ distances, for the left and right singular subspaces. Lower bounds, which show that the individual perturbation bounds are rate-optimal, are also given. The new perturbation bounds are applicable to a wide range of problems. In this paper, we consider in detail applications to low-rank matrix denoising and singular space estimation, high-dimensional clustering, and canonical correlation analysis (CCA). In particular, separate matching upper and lower bounds are obtained for estimating the left and right singular spaces. To the best of our knowledge, this is the first result that gives different optimal rates for the left and right singular spaces under the same perturbation. In addition to these problems, applications to other high-dimensional problems such as community detection in bipartite networks, multidimensional scaling, and cross-covariance matrix estimation are also discussed.

Motivation & Objective

• To address the limitation of uniform perturbation bounds in Wedin’s sin Θ theorem, which treat left and right singular subspaces symmetrically despite differing sensitivity in high-dimensional settings.
• To derive separate, rate-optimal upper and lower bounds for the left and right singular subspaces under the same perturbation, demonstrating that their convergence rates can differ significantly.
• To apply these refined bounds to key high-dimensional statistical problems, including low-rank matrix denoising, clustering, and canonical correlation analysis (CCA), where only one singular subspace may be of primary interest.
• To establish the first theoretical framework showing that left and right singular spaces can achieve different optimal rates under identical noise conditions, resolving a long-standing gap in perturbation theory.

Proposed method

• Derives separate perturbation bounds for left and right singular subspaces using spectral and Frobenius sin Θ distances, which are standard measures in matrix analysis.
• Introduces a novel analysis framework that decouples the behavior of left and right singular subspaces, allowing for asymmetric convergence rates.
• Employs concentration inequalities and random matrix theory tools, including Haar-distributed random matrices and χ² tail bounds, to control the spectral norm of perturbed subspaces.
• Applies a conditioning argument on the first r singular vectors and uses ε-net methods to bound the norm of residual subspaces.
• Leverages matrix perturbation theory and the Davis-Kahan-Wedin framework, extending it to asymmetric, non-symmetric settings with separate bounds.
• Uses lower bound constructions to prove rate-optimality, showing that the derived upper bounds are tight within a constant factor.

Experimental results

Research questions

• RQ1 Can the left and right singular subspaces of a low-rank matrix be estimated at different optimal rates under the same perturbation?
• RQ2 What are the precise convergence rates for estimating the left and right singular subspaces in high-dimensional settings with i.i.d. noise?
• RQ3 How do the perturbation bounds for singular subspaces differ when the row and column dimensions of the matrix are significantly unbalanced?
• RQ4 In which high-dimensional statistical models—such as low-rank denoising, clustering, or CCA—do the separate left and right singular subspace rates become practically significant?
• RQ5 Is it possible to stably recover the original matrix or a singular subspace when one side (left or right) is recoverable while the other is not?

Key findings

• The paper establishes the first rate-optimal perturbation bounds for left and right singular subspaces separately, showing that their convergence rates can differ under the same perturbation.
• For low-rank matrix denoising, the optimal rate for estimating the left singular space is O(√(d/n)) while the right singular space can be O(√(p/n)), with d and p being the row and column dimensions.
• In high-dimensional clustering, the ability to recover group structures depends on which singular subspace is well-estimated, and the paper shows that only one may be recoverable under asymmetric dimensionality.
• The lower bounds match the upper bounds within a constant factor, proving that the derived rates are optimal and cannot be improved asymptotically.
• The analysis reveals that in some settings, the left singular space can be accurately estimated while the right is not, and vice versa, depending on the matrix dimensions and noise structure.
• The results are applied to canonical correlation analysis, where the paper shows that the optimal estimation rate for the left and right canonical vectors can differ, even under identical noise levels.