[Paper Review] Asymptotic behavior of unregularized and ridge-regularized high-dimensional robust regression estimators : rigorous results

Noureddine El Karoui|arXiv (Cornell University)|Nov 11, 2013

Sparse and Compressive Sensing Techniques25 references85 citations

TL;DR

This paper rigorously establishes the asymptotic behavior of high-dimensional robust regression estimators—both unregularized and ridge-regularized—under the regime where $ p/n \to c \in (0, \infty) $. Using tools from random matrix theory, concentration of measure, and convex analysis, it proves that the limiting distribution of the estimator depends on the ratio $ p/n $, and validates earlier probabilistic heuristics with formal mathematical justification, even for non-Gaussian designs.

ABSTRACT

We study the behavior of high-dimensional robust regression estimators in the asymptotic regime where $p/n$ tends to a finite non-zero limit. More specifically, we study ridge-regularized estimators, i.e $\widehatβ= ext{argmin}_{β\in \mathbb{R}^p} \frac{1}{n}\sum_{i=1}^n ρ(\varepsilon_i-X_i' β)+\fracτ{2}\lVertβ Vert^2$. In a recently published paper, we had developed with collaborators probabilistic heuristics to understand the asymptotic behavior of $\widehatβ$. We give here a rigorous proof, properly justifying all the arguments we had given in that paper. Our proof is based on the probabilistic heuristics we had developed, and hence ideas from random matrix theory, measure concentration and convex analysis. While most the work is done for $τ>0$, we show that under some extra assumptions on $ρ$, it is possible to recover the case $τ=0$ as a limiting case. We require that the $X_i$'s be i.i.d with independent entries, but our proof handles the case where these entries are not Gaussian. A 2-week old paper of Donoho and Montanari [arXiv:1310.7320] studied a similar problem by a different method and with a different point of view. At this point, their interesting approach requires Gaussianity of the design matrix.

Motivation & Objective

• To rigorously justify the probabilistic heuristics previously proposed for high-dimensional robust regression estimators in the $ p/n \to c \in (0, \infty) $ regime.
• To establish the asymptotic behavior of ridge-regularized M-estimators under general i.i.d. design matrices with non-Gaussian entries.
• To extend the results to the unregularized case ($ \tau = 0 $) under additional regularity assumptions on the loss function $ \rho $.
• To provide a formal mathematical foundation for the variational problem formulation that emerged from earlier heuristic analyses.
• To demonstrate that the limiting behavior of the estimator depends non-trivially on the dimensionality ratio $ p/n $, contrary to classical fixed-$ p $ asymptotics.

Proposed method

• Employs leave-one-out and martingale techniques to analyze the behavior of the estimator in high-dimensional settings.
• Uses concentration of measure and random matrix theory to control the fluctuations of quadratic forms involving the design matrix.
• Applies the proximal mapping framework to characterize the solution of the optimization problem via the subgradient condition.
• Derives a key variational problem that characterizes the limiting behavior of the estimator, based on the distribution of the error and the loss function $ \rho $.
• Utilizes the Sherman-Morrison-Woodbury formula and block matrix inversion to analyze trace expressions involving regularized precision matrices.
• Establishes existence and uniqueness of the solution to the critical equation $ F(x) = 0 $ that arises in the asymptotic characterization.

Experimental results

Research questions

• RQ1 How does the asymptotic distribution of ridge-regularized robust regression estimators depend on the ratio $ p/n $ in high-dimensional settings?
• RQ2 Can the probabilistic heuristics from prior work be formally justified using rigorous tools from random matrix theory and concentration of measure?
• RQ3 What conditions on the loss function $ \rho $ and error distribution allow the unregularized case ($ \tau = 0 $) to be recovered as a limit of the ridge-regularized case?
• RQ4 To what extent can the results be extended beyond Gaussian design matrices, and what assumptions are necessary for non-Gaussian i.i.d. entries?
• RQ5 How does the limiting behavior of the estimator relate to the proximal mapping of the loss function $ \rho $, and what role does the error distribution play?

Key findings

• The limiting behavior of the ridge-regularized estimator depends on the solution to a variational problem involving the proximal mapping of $ \rho $, which is non-trivially dependent on $ p/n $.
• The paper rigorously proves that the asymptotic distribution of the estimator is characterized by a solution to a deterministic equation involving the cumulative distribution function of the error and the loss function $ \rho $.
• For non-Gaussian i.i.d. design matrices, the limiting behavior is still tractable under mild moment and tail conditions, extending beyond the Gaussian assumption used in prior work.
• The unregularized case ($ \tau = 0 $) can be recovered as a limiting case of the ridge-regularized estimator under additional smoothness and symmetry assumptions on $ \rho $.
• The trace of the regularized precision matrix converges to a deterministic limit that depends on $ p/n $, with explicit bounds derived via matrix inversion identities.
• The solution to the equation $ F(x) = 0 $, which governs the asymptotic behavior, is unique under mild regularity conditions on the error density and the loss function $ \rho $.