[Paper Review] Adaptive Estimation of Multivariate Regression with Hidden Variables

Xin Bing, Yang Ning+1 more|arXiv (Cornell University)|Mar 30, 2020

Statistical Methods and Inference54 references4 citations

TL;DR

This paper proposes HIVE, a novel algorithm for adaptive estimation of multivariate regression coefficients when hidden variables are present but unobserved. By combining group-lasso and ridge penalties in a two-step procedure—first estimating the coefficient matrix and residual structure, then projecting out hidden variable effects—it establishes non-asymptotic error bounds under both homoscedastic and heteroscedastic errors, ensuring identifiability and statistical consistency.

ABSTRACT

This paper studies the estimation of the coefficient matrix $\Ttheta$ in multivariate regression with hidden variables, $Y = (\Ttheta)^TX + (B^*)^TZ + E$, where $Y$ is a $m$-dimensional response vector, $X$ is a $p$-dimensional vector of observable features, $Z$ represents a $K$-dimensional vector of unobserved hidden variables, possibly correlated with $X$, and $E$ is an independent error. The number of hidden variables $K$ is unknown and both $m$ and $p$ are allowed but not required to grow with the sample size $n$. Since only $Y$ and $X$ are observable, we provide necessary conditions for the identifiability of $\Ttheta$. The same set of conditions are shown to be sufficient when the error $E$ is homoscedastic. Our identifiability proof is constructive and leads to a novel and computationally efficient estimation algorithm, called HIVE. The first step of the algorithm is to estimate the best linear prediction of $Y$ given $X$ in which the unknown coefficient matrix exhibits an additive decomposition of $\Ttheta$ and a dense matrix originated from the correlation between $X$ and the hidden variable $Z$. Under the row sparsity assumption on $\Ttheta$, we propose to minimize a penalized least squares loss by regularizing $\Ttheta$ via a group-lasso penalty and regularizing the dense matrix via a multivariate ridge penalty. Non-asymptotic deviation bounds of the in-sample prediction error are established. Our second step is to estimate the row space of $B^*$ by leveraging the covariance structure of the residual vector from the first step. In the last step, we remove the effect of hidden variable by projecting $Y$ onto the complement of the estimated row space of $B^*$. Non-asymptotic error bounds of our final estimator are established. The model identifiability, parameter estimation and statistical guarantees are further extended to the setting with heteroscedastic errors.

Motivation & Objective

• Address the challenge of estimating multivariate regression coefficients when unobserved hidden variables correlate with observed features.
• Establish necessary and sufficient conditions for the identifiability of the coefficient matrix Θ in the presence of hidden variables.
• Develop a computationally efficient estimation algorithm that handles unknown numbers of hidden variables and growing dimensions.
• Provide non-asymptotic prediction and estimation error bounds under both homoscedastic and heteroscedastic error structures.
• Extend the framework to settings where the number of responses (m) and predictors (p) may grow with sample size n.

Proposed method

• Formulate the multivariate regression model as Y = Θ^T X + (B^*)^T Z + E, where Z represents unobserved hidden variables correlated with X.
• Constructively prove identifiability of Θ under a set of conditions that are both necessary and sufficient when errors are homoscedastic.
• Propose the HIVE algorithm: first, estimate the best linear predictor of Y given X, decomposing Θ into sparse and dense components.
• Minimize a penalized least squares loss using group-lasso on Θ and multivariate ridge on the dense component to enforce row sparsity and stability.
• Estimate the row space of B^* by analyzing the covariance structure of residuals from the first step.
• Remove the hidden variable effect by projecting Y onto the orthogonal complement of the estimated row space of B^*, yielding the final estimator.

Experimental results

Research questions

• RQ1 Under what conditions is the coefficient matrix Θ identifiable in multivariate regression with hidden variables?
• RQ2 How can we consistently estimate Θ when the number of hidden variables K is unknown and Z is unobserved?
• RQ3 What is the non-asymptotic prediction error behavior of the proposed estimator under row sparsity and error heteroscedasticity?
• RQ4 Can the estimation procedure be made computationally efficient while maintaining statistical guarantees?
• RQ5 How does the covariance structure of residuals inform the estimation of the hidden variable space?

Key findings

• The proposed identifiability conditions are both necessary and sufficient when errors are homoscedastic, enabling exact recovery of Θ under these constraints.
• The HIVE algorithm achieves non-asymptotic in-sample prediction error bounds through a two-step procedure combining group-lasso and ridge regularization.
• The final estimator of Θ achieves non-asymptotic error bounds after projecting Y onto the complement of the estimated row space of B^*.
• The method remains valid and statistically consistent even when the number of responses m and predictors p grow with sample size n.
• The framework is extended to heteroscedastic errors, maintaining identifiability and providing error bounds under the same structural assumptions.
• The constructive proof of identifiability directly informs the algorithmic design, ensuring that the estimation procedure is both computationally efficient and statistically sound.