[Paper Review] Taming heavy-tailed features by shrinkage

Ziwei Zhu, Wenjing Zhou|arXiv (Cornell University)|Oct 24, 2017

Statistical Methods and Inference23 references4 citations

TL;DR

This paper proposes an $ε$-norm shrinkage method to robustify generalized linear models (GLMs) under heavy-tailed features and responses. By applying $\ell_4$-norm shrinkage in low dimensions and elementwise shrinkage in high dimensions, the method achieves nearly minimax optimal estimation with exponential deviation bounds under bounded fourth moments, significantly improving performance in logistic and linear regression on heavy-tailed data and reducing misclassification by over 30% in digit recognition under noise and mislabeling.

ABSTRACT

In this work, we focus on a variant of the generalized linear model (GLM) called corrupted GLM (CGLM) with heavy-tailed features and responses. To robustify the statistical inference on this model, we propose to apply $\ell_4$-norm shrinkage to the feature vectors in the low-dimensional regime and apply elementwise shrinkage to them in the high-dimensional regime. Under bounded fourth moment assumptions, we show that the maximum likelihood estimator (MLE) based on the shrunk data enjoys nearly the minimax optimal rate with an exponential deviation bound. Our simulations demonstrate that the proposed feature shrinkage significantly enhances the statistical performance in linear regression and logistic regression on heavy-tailed data. Finally, we apply our shrinkage principle to guard against mislabeling and image noise in the human-written digit recognition problem. We add an $\ell_4$-norm shrinkage layer to the original neural net and reduce the testing misclassification rate by more than $30\%$ relatively in the presence of mislabeling and image noise.

Motivation & Objective

• To address the degradation of standard estimators in generalized linear models when features and responses exhibit heavy-tailed distributions.
• To develop a robust statistical framework for GLMs under weak moment assumptions, specifically bounded fourth moments, rather than sub-Gaussian or sub-exponential conditions.
• To improve statistical efficiency and inference robustness in regression problems where heavy-tailed features distort maximum likelihood estimation.
• To extend robust estimation principles from mean estimation and risk minimization to feature-level corruption in GLMs.
• To demonstrate the effectiveness of feature shrinkage in real-world scenarios involving mislabeling and image noise, such as in neural network-based digit recognition.

Proposed method

• Applies $\ell_4$-norm shrinkage to feature vectors in low-dimensional regimes, defined as $\widetilde{\mathbf{x}}_i^s = \min(\|\mathbf{x}_i\|, \tau) \cdot \mathbf{x}_i / \|\mathbf{x}_i\|$ to cap extreme feature magnitudes.
• Uses elementwise shrinkage in high-dimensional regimes to reduce the influence of heavy-tailed features on estimation.
• Employs a modified maximum likelihood estimator (MLE) on shrunk features to achieve sub-Gaussian concentration under bounded fourth moments.
• Derives exponential deviation bounds for the MLE on shrunk data, proving near-minimax optimality under weak moment assumptions.
• Introduces a shrinkage layer in neural networks to guard against adversarial corruption and mislabeling in image classification.
• Uses a thresholding mechanism based on the $\ell_4$-norm to limit feature vector norms, thereby reducing outlier influence on the log-likelihood.

Experimental results

Research questions

• RQ1 Can standard MLEs remain statistically efficient under heavy-tailed features and responses, or do they suffer from poor concentration?
• RQ2 Does feature shrinkage—specifically $\ell_4$-norm or elementwise shrinkage—improve estimation accuracy and robustness in GLMs with heavy-tailed designs?
• RQ3 Can the proposed shrinkage method achieve nearly minimax optimal rates under only bounded fourth moments, without sub-Gaussian assumptions?
• RQ4 How effective is feature shrinkage in mitigating the impact of mislabeling and image noise in deep learning models?
• RQ5 Can the shrinkage principle be extended to neural networks to improve robustness in real-world, corrupted data settings?

Key findings

• The MLE based on $\ell_4$-norm shrunk features achieves nearly minimax optimal estimation rates under bounded fourth moment assumptions.
• The method provides an exponential deviation bound for the MLE on shrunk data, ensuring strong concentration even with heavy-tailed features.
• Simulations show significant performance gains in linear and logistic regression on heavy-tailed data, with improved boundary alignment and reduced misclassification.
• In digit recognition with mislabeling and image noise, adding an $\ell_4$-norm shrinkage layer reduced the testing misclassification rate by over 30%.
• Theoretical analysis confirms that shrinkage trades minimal bias for substantial variance reduction, enabling robust inference under weak moment conditions.
• The method outperforms standard MLEs when features are heavy-tailed, especially in the presence of outliers or mislabeled data.