[Paper Review] Approximate loss minimization with heavy tails.

Daniel Hsu, Sivan Sabato|arXiv (Cornell University)|Jul 7, 2013

Sparse and Compressive Sensing TechniquesEngineering10 citations

TL;DR

This paper introduces a generalized median-of-means estimator for metric spaces that enables exponential concentration under heavy-tailed distributions using only bounded low-order moments. It achieves optimal sample complexity of $\tilde{O}(d\log(1/\delta))$ for approximate minimization of smooth, strongly convex losses—such as least squares regression—without requiring subgaussian or bounded covariates or noise.

ABSTRACT

This work studies applications and generalizations of a simple estimation technique that provides exponential concentration under heavy-tailed distributions, assuming only bounded low-order moments. We show that the technique can be used for approximate minimization of smooth and strongly convex losses, and specifically for least squares linear regression. For instance, our $d$-dimensional estimator requires just $ ilde{O}(d\log(1/\delta))$ random samples to obtain a constant factor approximation to the optimal least squares loss with probability $1-\delta$, without requiring the covariates or noise to be bounded or subgaussian. We provide further applications to sparse linear regression and low-rank covariance matrix estimation with similar allowances on the noise and covariate distributions. The core technique is a generalization of the median-of-means estimator to arbitrary metric spaces.

Motivation & Objective

• To develop a robust estimation technique that maintains strong concentration properties under heavy-tailed distributions with only bounded low-order moments.
• To extend the median-of-means principle beyond real-valued data to arbitrary metric spaces, enabling broader applicability.
• To achieve optimal sample complexity for approximate minimization of smooth and strongly convex losses in high-dimensional settings.
• To apply the method to practical problems such as least squares regression, sparse linear regression, and low-rank covariance estimation under weak distributional assumptions.
• To remove the need for subgaussian or bounded noise and covariates in high-dimensional estimation tasks, improving real-world robustness.

Proposed method

• Generalizes the classical median-of-means estimator to arbitrary metric spaces using a geometric median in the space of empirical risk minimizers.
• Employs a partitioning strategy where data is split into groups, and a risk minimizer is computed per group before taking the geometric median of these estimators.
• Relies on the stability of the loss function and metric structure to ensure concentration even when the underlying distribution has heavy tails.
• Uses boundedness of low-order moments (e.g., second moment) as the minimal assumption, avoiding subgaussian or boundedness requirements.
• Applies the generalized median-of-means to specific problems: least squares regression, sparse regression, and low-rank covariance estimation.
• Establishes theoretical guarantees via concentration inequalities in metric spaces, leveraging the contraction property of geometric medians.

Experimental results

Research questions

• RQ1 Can the median-of-means principle be generalized to metric spaces to maintain exponential concentration under heavy-tailed distributions?
• RQ2 What is the optimal sample complexity for achieving a constant-factor approximation to the optimal loss in high-dimensional regression under weak moment assumptions?
• RQ3 Can the method be applied to sparse linear regression and low-rank covariance estimation without requiring subgaussian or bounded noise?
• RQ4 How does the performance of the generalized median-of-means estimator compare to classical estimators when the data distribution has heavy tails?
• RQ5 What are the minimal moment conditions required to achieve strong concentration in non-subgaussian settings?

Key findings

• The proposed estimator achieves a constant-factor approximation to the optimal least squares loss with probability $1 - \delta$ using only $\tilde{O}(d\log(1/\delta))$ samples in $d$ dimensions.
• The method requires no subgaussian or boundedness assumptions on the covariates or noise, relying only on bounded low-order moments.
• The generalized median-of-means estimator ensures exponential concentration in metric spaces, even under heavy-tailed distributions.
• The technique is applicable to sparse linear regression, achieving robustness under weak moment conditions.
• For low-rank covariance matrix estimation, the method provides robust estimation without requiring bounded or subgaussian entries.
• The theoretical framework demonstrates that the geometric median of empirical risk minimizers provides robustness and optimal sample complexity in high-dimensional settings.