[Paper Review] Variance Breakdown of Huber (M)-estimators: $n/p ightarrow m \in (1,\infty)$

David L. Donoho, Andrea Montanari|arXiv (Cornell University)|Mar 6, 2015

Statistical Methods and Inference8 references14 citations

TL;DR

This paper analyzes the minimax asymptotic variance of Huber M-estimators in high-dimensional linear regression, where the ratio of sample size to number of parameters $ n/p o m o (1,rown) $. It shows that the variance breaks down when $ m \cdot I(F_{\varepsilon}^*) \leq 1 $, with a critical breakdown point at $ \varepsilon^* = \inf\{\varepsilon : m \cdot I(F_{\varepsilon}^*) \geq 1\} $, contrasting classical results where such breakdown does not occur in the fixed-$ p $, large-$ n $ regime.

ABSTRACT

A half century ago, Huber evaluated the minimax asymptotic variance in scalar location estimation, $ \min_ψ\max_{F \in {\cal F}_ε} V(ψ, F) = \frac{1}{I(F_ε^*)} $, where $V(ψ,F)$ denotes the asymptotic variance of the $(M)$-estimator for location with score function $ψ$, and $I(F_ε^*)$ is the minimal Fisher information $ \min_{{\cal F}_ε} I(F)$ over the class of $ε$-Contaminated Normal distributions. We consider the linear regression model $Y = Xθ_0 + W$, $W_i\sim_{ ext{i.i.d.}}F$, and iid Normal predictors $X_{i,j}$, working in the high-dimensional-limit asymptotic where the number $n$ of observations and $p$ of variables both grow large, while $n/p ightarrow m \in (1,\infty)$; hence $m$ plays the role of `asymptotic number of observations per parameter estimated'. Let $V_m(ψ,F)$ denote the per-coordinate asymptotic variance of the $(M)$-estimator of regression in the $n/p ightarrow m$ regime. Then $V_m eq V$; however $V_m ightarrow V$ as $m ightarrow \infty$. In this paper we evaluate the minimax asymptotic variance of the Huber $(M)$-estimate. The statistician minimizes over the family $(ψ_λ)_{λ> 0}$ of all tunings of Huber $(M)$-estimates of regression, and Nature maximizes over gross-error contaminations $F \in {\cal F}_ε$. Suppose that $I(F_ε^*) \cdot m > 1$. Then $ \min_λ\max_{F \in {\cal F}_ε} V_m(ψ_λ, F) = \frac{1}{I(F_ε^*) - 1/m} $. Strikingly, if $I(F_ε^*) \cdot m \leq 1$, then the minimax asymptotic variance is $+\infty$. The breakdown point is where the Fisher information per parameter equals unity.

Motivation & Objective

• To extend Huber's classical minimax robust estimation framework from scalar location to high-dimensional linear regression under the $ n/p \to m \in (1,\infty) $ asymptotic regime.
• To characterize the minimax asymptotic variance of Huber M-estimators in this high-dimensional regime, accounting for gross-error contamination.
• To identify the critical point at which the variance of the Huber estimator becomes infinite, indicating breakdown under contamination.
• To contrast the high-dimensional breakdown behavior with classical results in the fixed-$ p $, large-$ n $ setting, where no such breakdown occurs.

Proposed method

• Formalizes the high-dimensional asymptotic regime where $ n $ and $ p $ grow with $ n/p \to m \in (1,\infty) $, modeling the design matrix as i.i.d. standard normal.
• Defines the per-coordinate asymptotic variance $ V_m(\psi, F) $ of Huber M-estimators in this regime, distinguishing it from the classical $ V(\psi, F) $.
• Analyzes the game-theoretic minimax problem: the statistician chooses the tuning $ \lambda $ of the Huber $ \psi $-function, while Nature chooses the worst-case contamination distribution $ F \in \mathcal{F}_\varepsilon $.
• Derives the minimax variance as $ \min_\lambda \max_{F \in \mathcal{F}_\varepsilon} V_m(\psi_\lambda, F) = \frac{1}{I(F_\varepsilon^*) - 1/m} $ when $ m \cdot I(F_\varepsilon^*) > 1 $, and $ \infty $ otherwise.
• Uses the implicit function theorem and stability analysis of the fixed-point equations governing the asymptotic variance to prove monotonicity and existence of optimal $ \lambda $.
• Establishes the critical contamination level $ \varepsilon^* = \inf\{\varepsilon : m \cdot I(F_\varepsilon^*) \geq 1\} $, where breakdown occurs.

Experimental results

Research questions

• RQ1 How does the minimax asymptotic variance of Huber M-estimators behave in the high-dimensional regime $ n/p \to m \in (1,\infty) $ under gross-error contamination?
• RQ2 Does the variance of the Huber estimator break down in this regime, and if so, under what conditions?
• RQ3 What is the critical contamination level $ \varepsilon^* $ at which the minimax variance becomes infinite?
• RQ4 How does the high-dimensional breakdown behavior of Huber estimators differ from the classical fixed-$ p $, large-$ n $ setting?
• RQ5 What is the role of the Fisher information $ I(F_\varepsilon^*) $ and the ratio $ m $ in determining the stability of Huber M-estimators?

Key findings

• The minimax asymptotic variance of Huber M-estimators in the $ n/p \to m \in (1,\infty) $ regime is $ \frac{1}{I(F_\varepsilon^*) - 1/m} $ when $ m \cdot I(F_\varepsilon^*) > 1 $, which is strictly larger than the classical minimax variance.
• When $ m \cdot I(F_\varepsilon^*) \leq 1 $, the minimax variance diverges to infinity, indicating a complete breakdown of the Huber estimator under contamination.
• The critical contamination level for breakdown is $ \varepsilon^* = \inf\{\varepsilon : m \cdot I(F_\varepsilon^*) \geq 1\} $, which depends on the asymptotic sample size per parameter $ m $.
• The breakdown occurs precisely when the Fisher information per parameter equals unity, i.e., $ I(F_\varepsilon^*) = 1/m $, marking a phase transition in estimator stability.
• The variance breakdown in the high-dimensional regime is a new phenomenon absent in the classical fixed-$ p $, large-$ n $ setting, where such divergence does not occur.
• The analysis proves that the optimal tuning $ \lambda^* $ of the Huber estimator is strictly increasing in the contamination level $ \varepsilon $, and that the optimal $ \kappa $-parameter in the fixed-point equations is bounded and stable.