[Paper Review] Residual variance and the signal-to-noise ratio in high-dimensional linear models

Lee H. Dicker|arXiv (Cornell University)|Aug 31, 2012

Statistical Methods and Inference19 references3 citations

TL;DR

This paper proposes consistent and asymptotically normal estimators for residual variance, signal strength, and signal-to-noise ratio in high-dimensional linear models with Gaussian predictors and errors, under a non-sparse signal assumption. The method leverages random matrix theory and Stein's lemma to achieve $ n^{-1/2} $-rate estimation even when $ d o ho n $ with $ d > n $, enabling reliable inference without sparsity constraints.

ABSTRACT

Residual variance and the signal-to-noise ratio are important quantities in many statistical models and model fitting procedures. They play an important role in regression diagnostics, in determining the performance limits in estimation and prediction problems, and in shrinkage parameter selection in many popular regularized regression methods for high-dimensional data analysis. We propose new estimators for the residual variance, the l2-signal strength, and the signal-to-noise ratio that are consistent and asymptotically normal in high-dimensional linear models with Gaussian predictors and errors, where the number of predictors d is proportional to the number of observations n. Existing results on residual variance estimation in high-dimensional linear models depend on sparsity in the underlying signal. Our results require no sparsity assumptions and imply that the residual variance may be consistently estimated even when d > n and the underlying signal itself is non-estimable. Basic numerical work suggests that some of the distributional assumptions made for our theoretical results may be relaxed.

Motivation & Objective

• Address the lack of consistent residual variance estimation in high-dimensional linear models when $ d > n $ and the true signal is non-sparse.
• Develop estimators for residual variance $ \sigma^2 $, $ \ell^2 $-signal strength $ \tau^2 $, and signal-to-noise ratio $ \tau^2 / \sigma^2 $ under general conditions.
• Establish asymptotic normality and $ n^{-1/2} $-rate consistency for these estimators in the high-dimensional regime $ d/n \to \rho \in [0, \infty) $.
• Provide theoretical justification for shrinkage parameter selection in regularized regression methods using these estimators.
• Demonstrate that $ \sigma^2 $ can be consistently estimated even when $ \boldsymbol{\beta} $ is non-estimable due to lack of sparsity.

Proposed method

• Propose estimators $ \tilde{\sigma}^2(\mathbf{m}) $ and $ \tilde{\tau}^2(\mathbf{m}) $ based on empirical moments of $ \mathbf{y} $ and $ X^T\mathbf{y} $, using $ \mathbf{m} = (d^{-1}\mathrm{tr}(\Sigma), d^{-1}\mathrm{tr}(\Sigma^2)) $ as plug-in estimates.
• Apply Stein's lemma and Gaussian integration by parts to derive asymptotic normality of the estimators under high-dimensional asymptotics.
• Use concentration inequalities and moment bounds on quadratic forms involving $ X $ and $ \boldsymbol{\epsilon} $ to control the error in the estimators.
• Leverage random matrix theory to analyze the eigenvalue behavior of $ X^T X $, particularly under $ d/n \to \rho $.
• Establish asymptotic normality via a multivariate central limit theorem for nonlinear functions of high-dimensional Gaussian vectors.
• Derive the delta method for the signal-to-noise ratio $ \tilde{\tau}^2 / \tilde{\sigma}^2 $, ensuring its asymptotic normality under the same conditions.

Experimental results

Research questions

• RQ1 Can residual variance $ \sigma^2 $ be consistently estimated in high-dimensional linear models when $ d > n $ and the true regression coefficient $ \boldsymbol{\beta} $ is non-sparse?
• RQ2 What are the asymptotic properties (consistency and normality) of estimators for $ \sigma^2 $, $ \tau^2 $, and $ \tau^2 / \sigma^2 $ under general (non-sparse) signal assumptions?
• RQ3 How does the performance of these estimators depend on the ratio $ d/n \to \rho $, and can they maintain $ n^{-1/2} $-rate consistency?
• RQ4 Can the signal-to-noise ratio be reliably estimated in high-dimensional settings where $ \boldsymbol{\beta} $ is non-estimable?
• RQ5 Are the distributional assumptions (Gaussianity of predictors and errors) necessary, or can they be relaxed in practice?

Key findings

• The proposed estimators for $ \sigma^2 $, $ \tau^2 $, and $ \tau^2 / \sigma^2 $ are consistent and asymptotically normal at rate $ n^{-1/2} $ under the high-dimensional regime $ d/n \to \rho \in [0, \infty) $.
• The estimators remain consistent even when $ d > n $ and $ \boldsymbol{\beta} $ is non-estimable, which contrasts with existing methods requiring sparsity.
• The asymptotic normality of $ \tilde{\sigma}^2(\mathbf{m}) $ and $ \tilde{\tau}^2(\mathbf{m}) $ is established via a multivariate central limit theorem for nonlinear functions of high-dimensional Gaussian vectors.
• The signal-to-noise ratio $ \tilde{\tau}^2 / \tilde{\sigma}^2 $ is asymptotically normal by the delta method, enabling valid inference and confidence intervals.
• Theoretical bounds on the total variation distance between the estimator and a normal distribution are derived, showing convergence at rate $ O(n^{-3/2}) $.
• Numerical experiments suggest that the Gaussianity assumptions may be relaxed in practice, though this is not formally proven in the paper.