[Paper Review] Bayesian Linear Regression for Multivariate Responses Under Group Sparsity

Bo Ning, Seonghyun Jeong+1 more|arXiv (Cornell University)|Jul 10, 2018

Statistical Methods and Inference25 references4 citations

TL;DR

This paper proposes a Bayesian multivariate linear regression model with group sparsity and unknown high-dimensional covariance, using spike-and-slab priors with ℓ₂,₁-regularization and a novel eigendecomposition-based prior on the covariance matrix. It establishes posterior contraction rates under the Euclidean metric and proves frequentist validity via a Bernstein-von Mises theorem, ensuring selection consistency and model dimension recovery.

ABSTRACT

We study frequentist properties of a Bayesian high-dimensional multivariate linear regression model with correlated responses. The predictors are separated into many groups and the group structure is pre-determined. Two features of the model are unique: (i) group sparsity is imposed on the predictors. (ii) the covariance matrix is unknown and its dimensions can also be high. We choose a product of independent spike-and-slab priors on the regression coefficients and a new prior on the covariance matrix based on its eigendecomposition. Each spike-and-slab prior is a mixture of a point mass at zero and a multivariate density involving a $\ell_{2,1}$-norm. We first obtain the posterior contraction rate, the bounds on the effective dimension of the model with high posterior probabilities. We then show that the multivariate regression coefficients can be recovered under certain compatibility conditions. Finally, we quantify the uncertainty for the regression coefficients with frequentist validity through a Bernstein-von Mises type theorem. The result leads to selection consistency for the Bayesian method. We derive the posterior contraction rate using the general theory by constructing a suitable test from the first principle using moment bounds for certain likelihood ratios. This leads to posterior concentration around the truth with respect to the average Rényi divergence of order 1/2. This technique of obtaining the required tests for posterior contraction rate could be useful in many other problems.

Motivation & Objective

• To study frequentist properties of Bayesian high-dimensional multivariate linear regression with group sparsity and unknown covariance matrix.
• To develop a prior structure that induces group sparsity via spike-and-slab priors with ℓ₂,₁-regularization on regression coefficients.
• To construct a novel prior on the covariance matrix using eigendecomposition to handle high-dimensional, unknown covariance.
• To establish posterior contraction rates under the average Rényi divergence of order 1/2 and convert them to Euclidean metric rates.
• To prove frequentist validity of posterior inference through a Bernstein-von Mises type theorem, ensuring selection consistency.

Proposed method

• Uses a product of independent spike-and-slab priors on regression coefficients, where each component is a mixture of a point mass at zero and a multivariate density involving ℓ₂,₁-regularization.
• Imposes group sparsity by applying the ℓ₂,₁-norm across groups, encouraging entire groups of predictors to be selected or excluded together.
• Proposes a new prior on the inverse covariance matrix based on its eigendecomposition, ensuring proper concentration and high-dimensional adaptability.
• Employs a general theory of posterior contraction by constructing tests from first principles using moment bounds on likelihood ratios.
• Breaks the parameter space into small pieces and controls error probabilities via moment bounds, enabling contraction rates under Rényi divergence.
• Converts the Rényi divergence-based contraction rate to the Euclidean metric via a change of measure argument, ensuring valid frequentist inference.

Experimental results

Research questions

• RQ1 Can posterior contraction rates be established for Bayesian multivariate regression with group sparsity and unknown high-dimensional covariance?
• RQ2 Does the proposed spike-and-slab prior with ℓ₂,₁-regularization lead to consistent model selection and recovery of the true group structure?
• RQ3 Can the novel eigendecomposition-based prior on the covariance matrix ensure proper concentration and high-dimensional adaptability?
• RQ4 Is the posterior distribution asymptotically normal in the sense of a Bernstein-von Mises theorem, ensuring frequentist validity of credible intervals?
• RQ5 Can the method achieve selection consistency under compatibility conditions and high-dimensional scaling?

Key findings

• The posterior contraction rate is established with respect to the average Rényi divergence of order 1/2, and this rate is converted to the Euclidean metric on the regression parameter.
• The method achieves posterior concentration around the true regression coefficients, with effective model dimension bounded with high posterior probability.
• Model dimension recovery is valid under compatibility conditions, with the number of non-zero groups estimated consistently.
• The prior concentration on the covariance matrix is controlled, with the prior probability of the neighborhood around the true inverse covariance bounded below by a constant multiple of −d³ log n.
• The sieve-based entropy and prior mass estimates are controlled, ensuring log N* ≲ nεₙ², which is essential for posterior contraction.
• Selection consistency is achieved, and the Bernstein-von Mises theorem holds, validating Bayesian credible sets as frequentist confidence sets in the high-dimensional regime.