[Paper Review] A sharp multiplier inequality with applications to heavy-tailed regression problems

Qiyang Han, Jon A. Wellner|arXiv (Cornell University)|Jun 7, 2017

Statistical Methods and Inference5 citations

TL;DR

This paper establishes a sharp rate for the Least Squares Estimator (LSE) in nonparametric regression with heavy-tailed errors (p-th moments, p ≥ 1), showing convergence at rate O_P(n^{-(1/(2+α))} ∨ n^{-(1/2)+(1/(2p))}) under an entropy condition with exponent α ∈ (0,2). The rate matches Gaussian performance when p ≥ 1 + 2/α, but is slower than robust estimators when p < 1 + 2/α, and critically depends on error-covariate independence.

ABSTRACT

We study the performance of the Least Squares Estimator (LSE) in a general nonparametric regression model, when the errors are independent of the covariates but may only have a $p$-th moment ($p\geq 1$). In such a heavy-tailed regression setting, we show that if the model satisfies a standard `entropy condition' with exponent $\alpha \in (0,2)$, then the $L_2$ loss of the LSE converges at a rate \begin{align*} \mathcal{O}_{\mathbf{P}}\big(n^{-\frac{1}{2+\alpha}} \vee n^{-\frac{1}{2}+\frac{1}{2p}}\big). \end{align*} Such a rate cannot be improved under the entropy condition alone. This rate quantifies both some positive and negative aspects of the LSE in a heavy-tailed regression setting. On the positive side, as long as the errors have $p\geq 1+2/\alpha$ moments, the $L_2$ loss of the LSE converges at the same rate as if the errors are Gaussian. On the negative side, if $p<1+2/\alpha$, there are (many) hard models at any entropy level $\alpha$ for which the $L_2$ loss of the LSE converges at a strictly slower rate than other robust estimators. The validity of the above rate relies crucially on the independence of the covariates and the errors. In fact, the $L_2$ loss of the LSE can converge arbitrarily slowly when the independence fails. The key technical ingredient is a new multiplier inequality that gives sharp bounds for the `multiplier empirical process' associated with the LSE. We further give an application to the sparse linear regression model with heavy-tailed covariates and errors to demonstrate the scope of this new inequality.

Motivation & Objective

• To analyze the performance of the Least Squares Estimator (LSE) in nonparametric regression when errors have only p-th moments (p ≥ 1), rather than sub-Gaussian or sub-exponential tails.
• To establish a sharp convergence rate for the L2 loss of the LSE under a standard entropy condition with exponent α ∈ (0,2).
• To clarify the conditions under which the LSE achieves optimal rates comparable to those under Gaussian errors, and when it is outperformed by robust estimators.
• To demonstrate the necessity of error-covariate independence for the derived rate, showing that failure of this assumption can lead to arbitrarily slow convergence.

Proposed method

• Derive a new sharp multiplier inequality to bound the multiplier empirical process associated with the LSE, which is the key technical tool for analyzing the L2 risk under heavy tails.
• Use the entropy condition with exponent α ∈ (0,2) to control the complexity of the function class in the nonparametric regression model.
• Analyze the L2 loss of the LSE by decomposing the risk into bias and variance components, with the variance term controlled via the new multiplier inequality.
• Establish a minimax lower bound to show that the derived rate cannot be improved under the entropy condition alone.
• Apply the main inequality to the sparse linear regression model with heavy-tailed covariates and errors to demonstrate its practical scope and robustness.

Experimental results

Research questions

• RQ1 Under what conditions does the LSE achieve the same convergence rate as under Gaussian errors in a heavy-tailed regression setting?
• RQ2 How does the number of moments p of the error distribution affect the convergence rate of the LSE relative to robust estimators?
• RQ3 What role does the entropy condition with exponent α play in determining the L2 risk rate of the LSE?
• RQ4 How sensitive is the LSE's convergence rate to the assumption of independence between errors and covariates?
• RQ5 Can the new multiplier inequality be effectively applied to high-dimensional or sparse regression models with heavy-tailed noise?

Key findings

• The L2 loss of the LSE converges at rate O_P(n^{-(1/(2+α))} ∨ n^{-(1/2)+(1/(2p))}) under the entropy condition with exponent α ∈ (0,2), which is sharp and cannot be improved under this condition alone.
• When p ≥ 1 + 2/α, the LSE achieves the same convergence rate as under Gaussian errors, indicating robustness to heavy tails in this regime.
• When p < 1 + 2/α, there exist models at any entropy level α for which the LSE converges strictly more slowly than robust estimators, highlighting a fundamental limitation.
• The derived rate is invalid if the independence between errors and covariates fails, as the L2 loss can then converge arbitrarily slowly.
• The new multiplier inequality provides a powerful tool for analyzing the LSE in heavy-tailed settings and is successfully applied to sparse linear regression with heavy-tailed covariates and errors.