[Paper Review] A spectral algorithm for robust regression with subgaussian rates

Jules Depersin|arXiv (Cornell University)|Jul 12, 2020

Advanced Statistical Methods and Models41 references4 citations

TL;DR

This paper presents a spectral algorithm for robust linear regression that achieves subgaussian error rates under minimal assumptions—only finite fourth moments and the presence of adversarial outliers. By adapting the spectral method for mean estimation to regression and introducing a novel empirical process, the algorithm ensures optimal statistical performance with linear to quadratic time complexity, enabling practical deployment while matching the ideal Gaussian-rate bounds.

ABSTRACT

We study a new linear up to quadratic time algorithm for linear regression in the absence of strong assumptions on the underlying distributions of samples, and in the presence of outliers. The goal is to design a procedure which comes with actual working code that attains the optimal sub-gaussian error bound even though the data have only finite moments (up to $L_4$) and in the presence of possibly adversarial outliers. A polynomial-time solution to this problem has been recently discovered but has high runtime due to its use of Sum-of-Square hierarchy programming. At the core of our algorithm is an adaptation of the spectral method introduced for the mean estimation problem to the linear regression problem. As a by-product we established a connection between the linear regression problem and the furthest hyperplane problem. From a stochastic point of view, in addition to the study of the classical quadratic and multiplier processes we introduce a third empirical process that comes naturally in the study of the statistical properties of the algorithm.

Motivation & Objective

• To develop a computationally efficient regression algorithm that maintains optimal statistical performance under weak moment assumptions and adversarial outliers.
• To achieve subgaussian error rates—matching the ideal i.i.d. Gaussian case—without requiring subgaussian or bounded data.
• To provide a working, implementable algorithm with theoretical guarantees on both statistical accuracy and computational efficiency.
• To establish a theoretical runtime of O(Nd) to O(N²d), making it practical for real-world datasets with heavy-tailed or corrupted distributions.
• To bridge the gap between theoretical robust estimators and practical, code-implementable algorithms with strong finite-sample performance.

Proposed method

• Adapts the spectral method from mean estimation (previously used in [35]) to the linear regression setting via a transformation of the design matrix and response vector.
• Employs block-wise decomposition of data to construct empirical processes and control deviation using concentration inequalities.
• Introduces a third empirical process—beyond classical quadratic and multiplier processes—critical for analyzing the algorithm’s statistical behavior.
• Uses a top singular vector computation as the core computational primitive, enabling linear to quadratic time complexity.
• Applies VC-dimension bounds and concentration results from [13] to control the deviation of the empirical process and ensure subgaussian tail bounds.
• Incorporates pruning and block selection strategies to ensure robustness against outliers while maintaining statistical efficiency.

Experimental results

Research questions

• RQ1 Can a computationally efficient algorithm achieve subgaussian error rates in linear regression under only finite fourth moments and adversarial outliers?
• RQ2 How can spectral methods from mean estimation be generalized to the regression setting to ensure robustness and optimal statistical performance?
• RQ3 What is the minimal computational cost required to achieve optimal subgaussian rates in heavy-tailed regression?
• RQ4 Can a practical, code-implementable algorithm match the theoretical performance of MLE/OLS in the ideal Gaussian setting under weak distributional assumptions?
• RQ5 What new empirical process structure emerges naturally in the analysis of spectral regression algorithms, and how can it be controlled?

Key findings

• The algorithm achieves the optimal subgaussian error rate, matching the deviation bounds of MLE/OLS in the i.i.d. Gaussian framework, even with only L⁴-finite moments.
• With high probability (1−δ), the estimation error is bounded by O(√(log(1/δ)/N)) under weak moment assumptions, achieving the same rate as in the subgaussian case.
• The algorithm’s runtime is O(Nd) in the best case and O(N²d) in the worst case, making it scalable for large datasets.
• The method establishes a novel connection between linear regression and the furthest hyperplane problem, offering new geometric insight into robust estimation.
• The analysis introduces and controls a new empirical process that is essential for proving subgaussian deviation bounds in the spectral regression framework.
• With high probability (≥ 1−1/32), the algorithm’s error is bounded by a constant multiple of the optimal subgaussian rate, even under adversarial contamination.