[Paper Review] Loss minimization and parameter estimation with heavy tails

Daniel Hsu, Sivan Sabato|arXiv (Cornell University)|Jan 1, 2016

Sparse and Compressive Sensing Techniques44 references91 citations

TL;DR

This paper introduces a generalized median-of-means estimator for parameter estimation under heavy-tailed distributions, requiring only bounded low-order moments. It achieves exponential concentration with O(d log(1/δ)) samples for d-dimensional least squares regression, enabling robust estimation without subgaussian or bounded noise assumptions.

ABSTRACT

This work studies applications and generalizations of a simple estimation technique that provides exponential concentration under heavy-tailed distributions, assuming only bounded low-order moments. We show that the technique can be used for approximate minimization of smooth and strongly convex losses, and specifically for least squares linear regression. For instance, our d-dimensional estimator requires just O(d log(1/δ)) random samples to obtain a constant factor approximation to the optimal least squares loss with probability 1-δ, without requiring the covariates or noise to be bounded or subgaussian. We provide further applications to sparse linear regression and low-rank covariance matrix estimation with similar allowances on the noise and covariate distributions. The core technique is a generalization of the median-of-means estimator to arbitrary metric spaces.

Motivation & Objective

• Address the challenge of parameter estimation under heavy-tailed distributions where traditional methods fail due to unbounded variance or subgaussian assumptions.
• Develop a generalization of the median-of-means estimator applicable to arbitrary metric spaces to improve robustness in high-dimensional and non-subgaussian settings.
• Enable efficient and accurate estimation of linear regression and low-rank covariance matrices under weak moment conditions, without requiring bounded or subgaussian covariates or noise.
• Provide theoretical guarantees for smooth, strongly convex loss minimization with minimal distributional assumptions on data.
• Extend the applicability of robust estimation techniques to sparse linear regression and covariance estimation under heavy-tailed noise.

Proposed method

• Generalize the classical median-of-means estimator to arbitrary metric spaces, enabling robust estimation beyond Euclidean settings.
• Use a partitioning strategy to divide data into subsets, compute local estimates, and take the median in the metric space to reduce sensitivity to outliers.
• Leverage the stability of the median in metric spaces to achieve exponential concentration of the estimator around the true parameter under only bounded low-order moments.
• Apply the generalized median-of-means to minimize smooth and strongly convex losses, including least squares, with high probability guarantees.
• Derive sample complexity bounds showing O(d log(1/δ)) samples suffice for constant-factor approximation to optimal least squares loss with confidence 1−δ.
• Extend the framework to sparse linear regression and low-rank covariance matrix estimation by adapting the median-of-means approach to structured parameter spaces.

Experimental results

Research questions

• RQ1 Can a robust estimation technique be developed that works under only bounded low-order moments, without requiring subgaussian or bounded noise?
• RQ2 How can the median-of-means principle be generalized to metric spaces beyond the real line to enable robust estimation in complex parameter spaces?
• RQ3 What sample complexity is required to achieve high-probability approximation of the optimal least squares solution under heavy-tailed distributions?
• RQ4 Can the generalized median-of-means be effectively applied to structured estimation problems like sparse regression and low-rank covariance estimation?
• RQ5 What theoretical guarantees on concentration and estimation error can be established for this method under weak moment assumptions?

Key findings

• The generalized median-of-means estimator achieves exponential concentration of the parameter estimate around the true value under only bounded low-order moments.
• For d-dimensional least squares regression, the method requires O(d log(1/δ)) samples to achieve a constant-factor approximation to the optimal loss with probability 1−δ.
• The estimator remains robust even when covariates or noise are heavy-tailed, subgaussian, or unbounded, as long as low-order moments are bounded.
• The framework successfully extends to sparse linear regression, maintaining sample efficiency and robustness under weak moment conditions.
• The method enables accurate low-rank covariance matrix estimation with similar robustness guarantees, without requiring subgaussian or bounded data.
• The theoretical analysis confirms that the median-of-means approach in metric spaces provides strong high-probability error bounds even in the presence of heavy-tailed data.