[Paper Review] Inference for Low-Rank Models

Victor Chernozhukov, Christian Hansen+2 more|arXiv (Cornell University)|Jul 6, 2021

Sparse and Compressive Sensing Techniques4 citations

TL;DR

This paper proposes a two-stage inference procedure for low-rank matrix models with diverging singular values and incoherent singular vectors, using nuclear-norm penalization followed by ordinary least squares to achieve asymptotically normal estimation and semiparametric efficiency. The method enables valid confidence intervals for linear functionals of the low-rank matrix, even under strong dependence in covariates.

ABSTRACT

This paper studies inference in linear models with a high-dimensional parameter matrix that can be well-approximated by a ``spiked low-rank matrix.'' A spiked low-rank matrix has rank that grows slowly compared to its dimensions and nonzero singular values that diverge to infinity. We show that this framework covers a broad class of models of latent-variables which can accommodate matrix completion problems, factor models, varying coefficient models, and heterogeneous treatment effects. For inference, we apply a procedure that relies on an initial nuclear-norm penalized estimation step followed by two ordinary least squares regressions. We consider the framework of estimating incoherent eigenvectors and use a rotation argument to argue that the eigenspace estimation is asymptotically unbiased. Using this framework we show that our procedure provides asymptotically normal inference and achieves the semiparametric efficiency bound. We illustrate our framework by providing low-level conditions for its application in a treatment effects context where treatment assignment might be strongly dependent.

Motivation & Objective

• To develop a valid inferential framework for high-dimensional linear models where the coefficient matrix is approximately low-rank with diverging singular values.
• To address the challenge of constructing confidence intervals for linear functionals of a low-rank matrix coefficient when the design matrix exhibits strong dependence.
• To establish asymptotic normality and semiparametric efficiency of the proposed estimator under weak regularity conditions.
• To extend existing inference methods beyond sparse models to dense linear combinations of low-rank matrices, particularly in treatment effect estimation with persistent treatment assignment.
• To provide a robust, sample-splitting-based procedure that eliminates regularization bias through a rotation argument on singular vectors.

Proposed method

• Apply nuclear-norm penalized estimation to obtain an initial low-rank approximation of the coefficient matrix Θ.
• Extract the right singular vectors from the initial estimator and treat them as regressors in a second-stage OLS regression.
• Use OLS to estimate the left singular vectors and update the right singular vectors, effectively rotating the initial estimator to remove bias.
• Construct the final estimator as the outer product of the estimated left and right singular vectors, ensuring alignment with the true eigenspace.
• Employ sample splitting to decouple estimation and inference, reducing dependence between estimation error and inference variance.
• Use a plug-in procedure for tuning parameter selection in nuclear-norm regularization based on empirical quantiles of noise-level estimates.

Experimental results

Research questions

• RQ1 Can we construct asymptotically valid confidence intervals for linear functionals of a low-rank matrix coefficient when the matrix has diverging singular values?
• RQ2 Does the proposed two-stage OLS procedure eliminate the bias from nuclear-norm penalization and achieve semiparametric efficiency?
• RQ3 How do the spiked singular value and incoherence conditions affect the finite-sample performance of inference in low-rank models?
• RQ4 Can the method handle strong dependence in the design matrix, such as in matrix completion or time-ordered treatment assignment?
• RQ5 Is the estimator asymptotically normal and efficient under weak conditions, including non-i.i.d. and persistent covariates?

Key findings

• The proposed estimator achieves asymptotic normality for linear functionals of the low-rank matrix under the spiked singular value and incoherence conditions.
• The method attains the semiparametric efficiency bound, meaning it achieves the lowest possible asymptotic variance among all regular estimators.
• Simulation results show 95% confidence interval coverage probabilities between 0.942 and 0.956 across various configurations, indicating good finite-sample performance.
• The estimator remains robust to the choice of rank J, with coverage probabilities stable across J = 1 to 4 in simulations.
• The method allows for inference under strong dependence in X, such as in systematic treatment assignment or matrix completion with persistent missingness.
• Theoretical results confirm that entrywise inference is impossible without the spiked singular value and incoherence conditions, validating their necessity.