[Paper Review] Distribution-robust mean estimation via smoothed random perturbations

Matthew J. Holland|arXiv (Cornell University)|Jun 25, 2019

Statistical Methods and Inference13 references4 citations

TL;DR

This paper proposes a new class of distribution-robust mean estimators that apply smoothed random perturbations—additive or multiplicative—to a soft-truncated empirical mean, enabling closed-form computation and sub-Gaussian-like deviation bounds under only finite variance. The key contribution is an estimator that achieves nearly sub-Gaussian performance across diverse distributions, with exponential tail bounds controlled by the second moment.

ABSTRACT

We consider the problem of mean estimation assuming only finite variance. We study a new class of mean estimators constructed by integrating over random noise applied to a soft-truncated empirical mean estimator. For appropriate choices of noise, we show that this can be computed in closed form, and utilizing relative entropy inequalities, these estimators enjoy deviations with exponential tails controlled by the second moment of the underlying distribution. We consider both additive and multiplicative noise, and several noise distribution families in our analysis. Furthermore, we empirically investigate the sensitivity to the mean-standard deviation ratio for numerous concrete manifestations of the estimator class of interest. Our main take-away is that an inexpensive new estimator can achieve nearly sub-Gaussian performance for a wide variety of data distributions.

Motivation & Objective

• To develop a mean estimator that maintains strong deviation guarantees under only finite variance, without assuming sub-Gaussian or light-tailed distributions.
• To construct a class of estimators via integration over random noise applied to a soft-truncated empirical mean, enabling closed-form computation.
• To demonstrate that these estimators achieve exponential tail bounds comparable to sub-Gaussian estimators, even for heavy-tailed distributions.
• To evaluate the sensitivity of the estimator to the mean-to-standard deviation ratio and sample size in finite-sample settings.

Proposed method

• The estimator is defined as the expectation of a soft-truncated empirical mean under a smoothed noise distribution, allowing closed-form computation for specific noise families.
• Relative entropy inequalities are used to derive deviation bounds, linking the tail behavior of the estimator to the second moment of the underlying distribution.
• Additive and multiplicative noise models are analyzed separately, with closed-form expressions derived for normal, Bernoulli, Weibull, and Student-t noise distributions.
• Theoretical bounds are derived using properties of the digamma function and Gamma distributions, particularly for Student-t noise.
• The method leverages the fact that log-transformed chi-squared and Gamma variables yield tractable expectations involving the digamma function Ψ.
• Optimization of the relative entropy upper bound with respect to a scaling parameter s yields tight deviation bounds for both one-sided and two-sided tail probabilities.

Experimental results

Research questions

• RQ1 Can a mean estimator be constructed that achieves sub-Gaussian-like deviation bounds under only finite variance, without assuming sub-Gaussian or light-tailed distributions?
• RQ2 How do different noise distributions—additive or multiplicative—impact the performance and robustness of the resulting estimator?
• RQ3 What is the sensitivity of the estimator to the mean-standard deviation ratio in finite samples across various underlying distributions?
• RQ4 Can the proposed estimator achieve nearly sub-Gaussian performance across a wide range of data-generating mechanisms, including heavy-tailed and skewed distributions?
• RQ5 How does the choice of noise distribution (e.g., normal, Student-t, Bernoulli) affect the tightness of the deviation bounds and the estimator’s finite-sample behavior?

Key findings

• The proposed estimator achieves two-sided deviation bounds with exponential tails, controlled by the second moment of the underlying distribution, even when only finite variance is assumed.
• For Student-t noise with $ q $ degrees of freedom, the relative entropy upper bound is derived as $ \frac{(q+1)}{2}\left(\log\left(1+\frac{\alpha^{2}}{q}\right)+\frac{4|\alpha|\Gamma((q+1)/2)}{\sqrt{q\pi}\Gamma(q/2)(q-1)}\right) $, enabling tight deviation control.
• The estimator achieves nearly sub-Gaussian performance across a wide variety of distributions, including heavy-tailed and skewed ones, as validated empirically.
• The deviation bounds are robust to the mean-SD ratio, with performance stable across different levels of skewness and kurtosis.
• Closed-form computation is achieved for multiple noise families, including normal, Bernoulli, Weibull, and Student-t, enabling practical implementation.
• Empirical results show that the estimator maintains strong finite-sample performance, with deviation distributions closely matching sub-Gaussian behavior even under non-normality.