[Paper Review] Outlier-robust estimation of a sparse linear model using $\ell_1$-penalized Huber's $M$-estimator

Arnak S. Dalalyan, Philip Thompson|arXiv (Cornell University)|Apr 12, 2019

Advanced Statistical Methods and Models48 references4 citations

TL;DR

This paper proposes an $μat$-penalized Huber's $M$-estimator for robust sparse linear regression under adversarial label contamination. It establishes that this convex optimization approach achieves the minimax-optimal estimation rate of $(s/n)^{1/2} + (o/n)$, up to logarithmic factors, under incoherence and transfer principles, demonstrating that robust, optimal, and computationally tractable estimation is possible with convex methods when only labels are corrupted.

ABSTRACT

We study the problem of estimating a $p$-dimensional $s$-sparse vector in a linear model with Gaussian design and additive noise. In the case where the labels are contaminated by at most $o$ adversarial outliers, we prove that the $\ell_1$-penalized Huber's $M$-estimator based on $n$ samples attains the optimal rate of convergence $(s/n)^{1/2} + (o/n)$, up to a logarithmic factor. For more general design matrices, our results highlight the importance of two properties: the transfer principle and the incoherence property. These properties with suitable constants are shown to yield the optimal rates, up to log-factors, of robust estimation with adversarial contamination.

Motivation & Objective

• To determine whether convex penalized empirical risk minimization (PERM) with Huber loss and $μat$-penalty can achieve optimal rates in sparse linear regression under adversarial label contamination.
• To establish conditions under which the $μat$-penalized Huber estimator attains the minimax-optimal rate of convergence despite adversarial outliers in the response variable.
• To highlight the roles of the transfer principle and incoherence property in enabling optimal robust estimation with convex methods.
• To demonstrate that computationally tractable convex estimators can achieve optimal rates, countering prior pessimistic results on convex methods in robust sparse regression.

Proposed method

• Formulates the problem as a joint minimization over regression coefficients $\boldsymbol{\beta}$ and contamination indicators $\boldsymbol{\theta}$ using a Huber loss function.
• Uses an $\ell_1$-penalized empirical risk minimization (PERM) framework with the loss function $\frac{1}{2n}\|\boldsymbol{Y} - \mathbf{X}\boldsymbol{\beta} - \sqrt{n}\boldsymbol{\theta}\|_2^2 + \lambda_s\|\boldsymbol{\beta}\|_1 + \lambda_o\|\boldsymbol{\theta}\|_1$.
• Establishes theoretical guarantees via Gaussian width and concentration arguments, leveraging the incoherence and transfer principles for the design matrix.
• Derives a lower bound on the norm $\|\mathbf{Z}^{(n)}\boldsymbol{v} + \boldsymbol{u}\|_2$ to control the estimation error, using bounds on Gaussian widths of $\ell_1$-balls.
• Applies a refined bound on the Gaussian width of the intersection of $\ell_1$ and $\ell_2$-balls to reduce logarithmic factors in the final rate.
• Proves that under the incoherence and transfer conditions, the estimator achieves the minimax-optimal rate up to logarithmic factors.

Experimental results

Research questions

• RQ1 Can convex penalized empirical risk minimization with Huber loss and $\ell_1$-penalty achieve the minimax-optimal rate in sparse linear regression under adversarial label contamination?
• RQ2 What structural conditions on the design matrix (e.g., incoherence, transfer principle) are sufficient to ensure optimal robust estimation with convex methods?
• RQ3 Does the $\ell_1$-penalized Huber estimator outperform or match the performance of computationally intractable estimators in the presence of adversarial outliers?
• RQ4 Can logarithmic factors in the estimation rate be reduced or eliminated under mild assumptions on the contamination proportion $o/n$?
• RQ5 Is it possible to achieve consistent support recovery and optimal estimation simultaneously using convex loss and penalty functions under label contamination?

Key findings

• The $\ell_1$-penalized Huber's $M$-estimator achieves the minimax-optimal rate of $\sigma\left(\frac{s\log(p/s)}{n}\right)^{1/2} + \frac{\sigma o}{n}$, up to logarithmic factors, under adversarial label contamination.
• The incoherence property of the design matrix $\mathbf{X}$, particularly when $\boldsymbol{\Sigma}$ has bounded and bounded-away-from-zero diagonal entries, ensures the estimator's optimality.
• The transfer principle and incoherence property are sufficient conditions to achieve the optimal rate, up to logarithmic factors, for general design matrices.
• The estimator remains computationally tractable and achieves optimal rates even when $p > n$, provided $\boldsymbol{\beta}^*$ is $s$-sparse and the contamination $\boldsymbol{\theta}^*$ is $o$-sparse.
• The use of a tighter Gaussian width bound allows the removal of logarithmic terms in the rate when $o/n$ is bounded away from zero or decays slowly.
• The results refute prior pessimistic claims that convex methods cannot achieve optimal rates under adversarial contamination, showing that such methods can be both optimal and computationally efficient.