[Paper Review] Efficient Bayesian reduced rank regression using Langevin Monte Carlo approach

Tien Mai|arXiv (Cornell University)|Feb 15, 2021

Sparse and Compressive Sensing Techniques32 references4 citations

TL;DR

This paper proposes a novel Langevin Monte Carlo (LMC) approach for Bayesian reduced rank regression (BRRR), using a spectral-scaled Student prior to induce low-rank structure without prespecifying rank. The method avoids costly matrix inversions and SVDs, enabling faster sampling than Gibbs sampling in high-dimensional settings while achieving predictive performance comparable to cross-validated frequentist methods.

ABSTRACT

The problem of Bayesian reduced rank regression is considered in this paper. We propose, for the first time, to use Langevin Monte Carlo method in this problem. A spectral scaled Student prior distrbution is used to exploit the underlying low-rank structure of the coefficient matrix. We show that our algorithms are significantly faster than the Gibbs sampler in high-dimensional setting. Simulation results show that our proposed algorithms for Bayesian reduced rank regression are comparable to the state-of-the-art method where the rank is chosen by cross validation.

Motivation & Objective

• To address the computational inefficiency of Gibbs sampling in high-dimensional Bayesian reduced rank regression (BRRR), which requires repeated matrix inversions and SVDs.
• To develop a scalable Bayesian inference method for BRRR that avoids low-rank matrix factorization and does not require prespecifying the rank.
• To leverage the spectral-scaled Student prior to promote low-rank structure in the coefficient matrix naturally.
• To demonstrate that Langevin Monte Carlo (LMC) and Metropolis-adjusted LMC (MALA) outperform Gibbs sampling in speed while maintaining predictive accuracy.

Proposed method

• Proposes a spectral-scaled Student prior on the coefficient matrix B, defined as π(B) ∝ det(λ²Iₚ + BBᵀ)⁻⁽ᵖ⁺ᵐ⁺²⁾ᐟ², which encourages low-rank structure without rank prespecification.
• Develops an unadjusted Langevin Monte Carlo (LMC) algorithm for posterior sampling in BRRR, using gradient of the log-posterior without matrix inversion.
• Implements a Metropolis-Hastings correction in MALA to improve sampling accuracy, maintaining detailed balance.
• Uses a step-size h chosen to achieve ~50% acceptance rate, following Roberts and Rosenthal (1998), to balance exploration and stability.
• Avoids explicit matrix inversion and SVD computation by relying on gradient-based updates, enabling scalability to large datasets.
• Employs a fixed σ² in the likelihood, with posterior estimation via LMC and MALA for B, while σ² is treated as known.

Experimental results

Research questions

• RQ1 Can Langevin Monte Carlo sampling be effectively applied to Bayesian reduced rank regression with a non-factorized prior?
• RQ2 Does the proposed LMC/MALA approach achieve faster convergence than Gibbs sampling in high-dimensional BRRR settings?
• RQ3 How does the spectral-scaled Student prior compare to traditional low-rank factorization priors in promoting low-rank structure without prespecifying rank?
• RQ4 Can LMC-based inference match the predictive performance of cross-validated frequentist RRR methods?
• RQ5 What is the impact of step-size selection on the convergence and accuracy of LMC and MALA in BRRR?

Key findings

• The proposed LMC and MALA algorithms are significantly faster than Gibbs sampling in high-dimensional settings, with running times scaling more favorably with increasing p.
• MALA outperforms LMC in terms of predictive accuracy and stability, achieving lower mean squared prediction error on real data.
• On the breast cancer dataset, MALA achieved a mean squared prediction error of 0.049 (±0.008), outperforming LMC (0.052 ± 0.009) and approaching the cross-validated RRR method (0.030 ± 0.008).
• The estimated rank was 1.03 (±0.17) with MALA, close to the true underlying rank, indicating effective rank estimation.
• The LMC and MALA methods achieved comparable predictive performance to the state-of-the-art frequentist RRR method with cross-validated rank selection.
• The spectral-scaled Student prior successfully induced low-rank structure without requiring prespecification of rank, enabling flexible and efficient Bayesian inference.