[Paper Review] Prediction risk for the horseshoe regression

Anindya Bhadra, Jyotishka Datta+3 more|arXiv (Cornell University)|May 16, 2016

Gene expression and cancer classification3 citations

TL;DR

This paper establishes theoretically that horseshoe regression, a global-local shrinkage method with component-specific local shrinkage, reduces prediction risk in high-dimensional linear models by overcoming two key limitations of global shrinkage methods: monotonic shrinkage relative to singular values and dependence on a single tuning parameter. The authors prove that under certain conditions, horseshoe regression achieves lower finite-sample predictive risk than ridge or principal components regression.

ABSTRACT

We show that prediction performance for global-local shrinkage regression can overcome two major difficulties of global shrinkage regression: (i) the amount of relative shrinkage is monotone in the singular values of the design matrix and (ii) the shrinkage is determined by a single tuning parameter. Specifically, we show that the horseshoe regression, with heavy-tailed component-specific local shrinkage parameters, in conjunction with a global parameter providing shrinkage towards zero, alleviates both these difficulties and consequently, results in an improved risk for prediction. Numerical demonstrations of improved prediction over competing approaches in simulations and in a pharmacogenomics data set confirm our theoretical findings.

Motivation & Objective

• To address the theoretical limitations of global shrinkage regression methods like ridge and principal components regression in high-dimensional prediction.
• To investigate why global-local shrinkage priors, such as the horseshoe, outperform global methods in practice despite lacking formal finite-sample risk comparisons.
• To establish conditions under which horseshoe regression achieves lower predictive risk than global shrinkage estimators.
• To develop a theoretical framework based on Stein's unbiased risk estimate for comparing finite-sample prediction risk across shrinkage methods.

Proposed method

• The authors use Stein's unbiased risk estimate (SURE) to derive finite-sample prediction risk expressions for shrinkage estimators.
• They analyze the risk behavior of global-local priors, particularly the horseshoe, by modeling component-specific local shrinkage parameters with heavy-tailed distributions.
• The method involves deriving analytical expressions for the expectation of functions of truncated gamma-type random variables under a generalized beta distribution framework.
• Theoretical risk comparisons are conducted using moment-generating functions and integral representations of normalized expectations under the CCH (confluent hypergeometric) distribution.
• The analysis focuses on the sensitivity of risk to the design matrix's singular values and the tuning parameters in the prior hierarchy.
• Theoretical results are validated through numerical simulations and a pharmacogenomics data application.

Experimental results

Research questions

• RQ1 Does horseshoe regression achieve lower prediction risk than global shrinkage methods like ridge and principal components regression in finite samples?
• RQ2 How do component-specific local shrinkage parameters affect the relative shrinkage pattern across singular values of the design matrix?
• RQ3 Under what conditions does the horseshoe prior reduce predictive risk compared to global shrinkage estimators?
• RQ4 Can the theoretical risk of global-local shrinkage methods be bounded or minimized using SURE-based analysis?
• RQ5 How does the structure of the design matrix influence the risk performance of horseshoe versus global shrinkage estimators?

Key findings

• The horseshoe regression achieves lower prediction risk than global shrinkage methods such as ridge and principal components regression in finite-sample settings.
• The horseshoe method avoids the monotonic shrinkage pattern relative to singular values that plagues global shrinkage estimators.
• The use of component-specific local shrinkage parameters allows the horseshoe to better adapt to the signal structure in the data, reducing bias in large signals.
• Theoretical analysis confirms that the horseshoe prior reduces risk when the true regression coefficients are sparse and signals vary in magnitude.
• Numerical results in simulations and a pharmacogenomics dataset confirm that horseshoe regression outperforms competing methods in terms of prediction accuracy.
• The study establishes that global-local priors can achieve lower risk than global priors by escaping the constraints of single-tuning-parameter dependence and singular value monotonicity.