[Paper Review] Think before you shrink: Alternatives to default shrinkage methods can improve prediction accuracy, calibration and coverage

Mark A. van de Wiel, Gwenaël G. R. Leday+4 more|arXiv (Cornell University)|Jan 24, 2023

Data Analysis with R4 citations

TL;DR

This paper proposes advanced shrinkage methods—differential ridge penalties and hierarchical Bayesian shrinkage—as alternatives to standard lasso and ridge regression, demonstrating that they significantly improve prediction accuracy, calibration, and coverage of confidence intervals in low-dimensional regression settings. The key contribution is that these methods outperform default shrinkage techniques by reducing bias while maintaining low variance, with implementations available via mgcv and r-stan for reproducibility and uncertainty quantification.

ABSTRACT

While shrinkage is essential in high-dimensional settings, its use for low-dimensional regression-based prediction has been debated. It reduces variance, often leading to improved prediction accuracy. However, it also inevitably introduces bias, which may harm two other measures of predictive performance: calibration and coverage of confidence intervals. Much of the criticism stems from the usage of standard shrinkage methods, such as lasso and ridge with a single, cross-validated penalty. Our aim is to show that readily available alternatives can strongly improve predictive performance, in terms of accuracy, calibration or coverage. For linear regression, we use small sample splits of a large, fairly typical epidemiological data set to illustrate this. We show that usage of differential ridge penalties for covariate groups may enhance prediction accuracy, while calibration and coverage benefit from additional shrinkage of the penalties. In the logistic setting, we apply an external simulation to demonstrate that local shrinkage improves calibration with respect to global shrinkage, while providing better prediction accuracy than other solutions, like Firth's correction. The benefits of the alternative shrinkage methods are easily accessible via example implementations using exttt{mgcv} and exttt{r-stan}, including the estimation of multiple penalties. A synthetic copy of the large data set is shared for reproducibility.

Motivation & Objective

• To address the trade-off between prediction accuracy, calibration, and confidence interval coverage in low-dimensional regression models.
• To challenge the default use of single-penalty lasso and ridge methods, which introduce bias and instability.
• To demonstrate that differential and local shrinkage methods can enhance predictive performance without sacrificing interpretability.
• To provide accessible, reproducible implementations using mgcv and r-stan for applied researchers in epidemiology and biostatistics.
• To advocate for hierarchical shrinkage and penalty uncertainty modeling to improve model reliability and uncertainty quantification.

Proposed method

• Application of differential ridge penalties tailored to groups of covariates, allowing variable-specific shrinkage without prior variable selection.
• Use of hierarchical Bayesian priors with hyperpriors on penalty variances to enable local shrinkage and stabilize estimates.
• Implementation of multiple penalty estimation using mgcv (frequentist) and r-stan (Bayesian) for automatic tuning and uncertainty propagation.
• Comparison of global vs. local shrinkage via simulation in logistic regression, including Firth’s correction and empirical Bayes approaches.
• Use of synthetic data and small sample splits to validate performance across varying n:p ratios.
• Incorporation of penalty uncertainty in frequentist intervals via modified variance estimation, improving coverage.

Experimental results

Research questions

• RQ1 Can differential ridge penalties improve prediction accuracy and calibration compared to standard ridge and lasso in low-dimensional settings?
• RQ2 How does local shrinkage via hierarchical priors affect calibration and coverage compared to global shrinkage?
• RQ3 To what extent do Bayesian hierarchical models with hyperpriors improve stability and uncertainty quantification of penalty parameters?
• RQ4 Can alternative shrinkage methods outperform Firth’s correction in logistic regression while maintaining better calibration and prediction accuracy?
• RQ5 How do the performance gains of advanced shrinkage methods compare to ordinary least squares in terms of bias, variance, and coverage?

Key findings

• Differential ridge penalties significantly improved prediction accuracy in linear regression by allowing variable group-specific shrinkage, outperforming standard ridge and lasso.
• Local shrinkage via hierarchical priors enhanced calibration and coverage of confidence intervals, reducing bias compared to global shrinkage methods.
• Bayesian hierarchical models with hyperpriors on penalty variances provided better uncertainty quantification and more stable penalty estimates than fixed or cross-validated penalties.
• The proposed methods achieved better calibration than Firth’s correction while maintaining or improving prediction accuracy in logistic regression.
• Implementations using mgcv and r-stan enabled automatic estimation of multiple penalties and provided reliable coverage, even in small samples.
• The synthetic data set and example code ensure reproducibility and facilitate adoption by applied researchers in epidemiology and biostatistics.