[Paper Review] Adaptive Huber Regression: Optimality and Phase Transition

Qiang Sun, Wen‐Xin Zhou+1 more|arXiv (Cornell University)|Jun 21, 2017

Statistical Methods and Inference9 citations

TL;DR

This paper proposes adaptive Huber regression that adjusts the robustification parameter to sample size, dimension, and moment conditions, achieving optimal tradeoff between bias and robustness. It establishes a sharp phase transition: sub-Gaussian-type deviation bounds when δ ≥ 1, and slower rates for 0 < δ < 1, under only (1+δ)-th moment assumptions, enabling robust estimation in heavy-tailed settings.

ABSTRACT

Big data can easily be contaminated by outliers or contain variables with heavy-tailed distributions, which makes many conventional methods inadequate. To address this challenge, we propose the adaptive Huber regression for robust estimation and inference. The key observation is that the robustification parameter should adapt to the sample size, dimension and moments for optimal tradeoff between bias and robustness. Our theoretical framework deals with heavy-tailed distributions with bounded $(1+\delta)$-th moment for any $\delta > 0$. We establish a sharp phase transition for robust estimation of regression parameters in both low and high dimensions: when $\delta \geq 1$, the estimator admits a sub-Gaussian-type deviation bound without sub-Gaussian assumptions on the data, while only a slower rate is available in the regime $0<\delta< 1$. Furthermore, this transition is smooth and optimal. In addition, we extend the methodology to allow both heavy-tailed predictors and observation noise. Simulation studies lend further support to the theory. In a genetic study of cancer cell lines that exhibit heavy-tailedness, the proposed methods are shown to be more robust and predictive.

Motivation & Objective

• To address robust estimation in high-dimensional and heavy-tailed data settings where conventional methods fail.
• To develop a method that adaptively tunes the Huber loss parameter based on sample size, dimension, and moment structure.
• To establish theoretical guarantees for estimation and inference under minimal moment assumptions, specifically bounded (1+δ)-th moments for δ > 0.
• To extend the framework to handle both heavy-tailed predictors and heavy-tailed error distributions.
• To demonstrate empirical superiority through simulations and a real-world genetic study on cancer cell lines.

Proposed method

• Proposes an adaptive Huber regression estimator where the robustification parameter is tuned based on sample size n, dimension p, and the (1+δ)-th moment of the error distribution.
• Introduces a data-driven choice of the Huber loss tuning parameter that balances bias and robustness optimally.
• Establishes theoretical bounds on estimation error using a novel analysis framework that accounts for the interplay between n, p, and δ.
• Derives deviation bounds that achieve sub-Gaussian-type tail behavior when δ ≥ 1, even without sub-Gaussian assumptions.
• Extends the method to handle heavy-tailed design matrices and heavy-tailed errors by modifying the loss function and estimation procedure.
• Employs a robust inference framework that maintains validity under weak moment conditions.

Experimental results

Research questions

• RQ1 What is the optimal tuning strategy for the Huber loss parameter in high-dimensional regression under heavy-tailed errors?
• RQ2 How does the estimation performance of Huber regression depend on the moment structure of the error distribution, particularly the (1+δ)-th moment?
• RQ3 Can sub-Gaussian-type deviation bounds be achieved without assuming sub-Gaussianity, and under what conditions?
• RQ4 How does the method perform when both predictors and errors are heavy-tailed?
• RQ5 What is the phase transition behavior in estimation accuracy as δ varies across the range (0,1) and [1,∞)?

Key findings

• When δ ≥ 1, the adaptive Huber estimator achieves sub-Gaussian-type deviation bounds without requiring sub-Gaussian assumptions on the data.
• For 0 < δ < 1, the estimator achieves a slower but still optimal rate of convergence under the (1+δ)-th moment condition.
• The phase transition between the two regimes is sharp and smooth, indicating a clear change in statistical behavior at δ = 1.
• The method remains robust and effective even when both predictors and error terms are heavy-tailed, as validated in simulation and real data.
• In a genetic study of cancer cell lines with heavy-tailed responses, the proposed method demonstrated superior predictive performance and robustness compared to conventional methods.
• The adaptive tuning of the Huber parameter leads to improved finite-sample performance and better balance between bias and robustness.