[Paper Review] A Statistical Perspective on Randomized Sketching for Ordinary Least-Squares

Garvesh Raskutti, Michael W. Mahoney|arXiv (Cornell University)|Jun 23, 2014

Sparse and Compressive Sensing Techniques26 references18 citations

TL;DR

This paper presents a unified statistical and algorithmic framework for analyzing randomized sketching in ordinary least-squares regression. It establishes that residual efficiency is achievable with $ r \lesssim p \ll n $, while prediction efficiency requires significantly larger sketch sizes, providing tight upper and lower bounds for both metrics across random projections and sampling methods.

ABSTRACT

We consider statistical as well as algorithmic aspects of solving large-scale least-squares (LS) problems using randomized sketching algorithms. For a LS problem with input data $(X, Y) \in \mathbb{R}^{n imes p} imes \mathbb{R}^n$, sketching algorithms use a sketching matrix, $S\in\mathbb{R}^{r imes n}$ with $r \ll n$. Then, rather than solving the LS problem using the full data $(X,Y)$, sketching algorithms solve the LS problem using only the sketched data $(SX, SY)$. Prior work has typically adopted an algorithmic perspective, in that it has made no statistical assumptions on the input $X$ and $Y$, and instead it has been assumed that the data $(X,Y)$ are fixed and worst-case (WC). Prior results show that, when using sketching matrices such as random projections and leverage-score sampling algorithms, with $p < r \ll n$, the WC error is the same as solving the original problem, up to a small constant. From a statistical perspective, we typically consider the mean-squared error performance of randomized sketching algorithms, when data $(X, Y)$ are generated according to a statistical model $Y = X β+ ε$, where $ε$ is a noise process. We provide a rigorous comparison of both perspectives leading to insights on how they differ. To do this, we first develop a framework for assessing algorithmic and statistical aspects of randomized sketching methods. We then consider the statistical prediction efficiency (PE) and the statistical residual efficiency (RE) of the sketched LS estimator; and we use our framework to provide upper bounds for several types of random projection and random sampling sketching algorithms. Among other results, we show that the RE can be upper bounded when $p < r \ll n$ while the PE typically requires the sample size $r$ to be substantially larger. Lower bounds developed in subsequent results show that our upper bounds on PE can not be improved.

Motivation & Objective

• To unify algorithmic and statistical perspectives on randomized sketching for large-scale least-squares problems.
• To analyze the statistical performance of sketching estimators under a linear model $ Y = X\beta + \epsilon $.
• To derive upper bounds on statistical prediction efficiency (PE) and residual efficiency (RE) for random projection and sampling sketching matrices.
• To establish that PE requires substantially larger sketch sizes than RE, and to show these bounds are tight via lower bounds.

Proposed method

• Develops a unified framework to assess both algorithmic and statistical aspects of randomized sketching methods.
• Analyzes the sketched least-squares estimator $ \beta_S $ obtained by solving $ \min_\beta \|SY - SX\beta\|_2^2 $ using a sketching matrix $ S \in \mathbb{R}^{r \times n} $ with $ r \ll n $.
• Introduces and bounds two statistical performance metrics: prediction efficiency (PE) and residual efficiency (RE) of the sketched estimator.
• Uses concentration inequalities and trace-based analysis to bound the expected Frobenius norm of $ U^T S^T S U $, where $ U $ is the right singular vector matrix of $ X $.
• Applies Markov's inequality and existing results on Hadamard-based sketching to derive high-probability bounds on the condition number and error amplification.
• Derives lower bounds showing that the upper bounds on PE cannot be improved, establishing tightness of the analysis.

Experimental results

Research questions

• RQ1 How do algorithmic and statistical perspectives on randomized sketching differ in their performance guarantees for least-squares problems?
• RQ2 Under what sketch size conditions does the sketched least-squares estimator achieve residual efficiency (RE) in a statistical model?
• RQ3 What sketch size is required for the sketched estimator to achieve prediction efficiency (PE), and how does it compare to RE?
• RQ4 Can the upper bounds on PE be improved, or are they tight?
• RQ5 How do different sketching methods—random projections and random sampling—compare in terms of statistical efficiency?

Key findings

• Residual efficiency (RE) can be upper bounded when $ p \lesssim r \ll n $, indicating that sketching preserves residual error well even with small sketches.
• Prediction efficiency (PE) typically requires the sketch size $ r $ to be substantially larger than $ p $, showing a fundamental gap between RE and PE in statistical performance.
• The upper bounds on PE are tight, as shown by matching lower bounds, proving that the derived bounds cannot be improved.
• For Gaussian random projections, the condition number $ \gamma(S) $ of the sketched system is bounded by $ 1 + \frac{n+1}{r} $ with high probability, implying good stability.
• For Hadamard-based sketching, the analysis leverages existing concentration results to bound $ \|U^T S^T S \|_F^2 / p $, showing that $ \gamma(S) \leq 11(1 + \frac{n+1}{r}) $ with probability at least 0.9.
• The analysis confirms that random sampling sketching (SGP) achieves $ \mathbb{E}[\|U^T S^T S U\|_F^2 / p] \leq 1 + \frac{n+1}{r} $, supporting its statistical robustness.