[Paper Review] Projection predictive variable selection using Stan+R

Juho Piironen, Aki Vehtari|arXiv (Cornell University)|Aug 11, 2015

Advanced Statistical Methods and ModelsMathematics11 references19 citations

TL;DR

This paper presents a practical implementation of projection predictive variable selection using Stan for Bayesian model fitting and R for variable selection in linear Gaussian models. It demonstrates that with hierarchical shrinkage priors (e.g., horseshoe with $ν=3$), the method achieves near-full-model predictive performance using only ~5 to 20 variables, while ensuring stable MCMC sampling and robust variable selection via cross-validation.

ABSTRACT

This document is additional material to our previous study comparing several strategies for variable subset selection. Our recommended approach was to fit the full model with all the candidate variables and best possible prior information, and perform the variable selection using the projection predictive framework. Here we give an example of performing such an analysis, using Stan for fitting the model, and R for the variable selection.

Motivation & Objective

• To demonstrate a scalable, robust workflow for Bayesian variable selection using Stan and R.
• To address the instability of NUTS sampling with heavy-tailed priors like horseshoe by adjusting degrees of freedom ($ν$) in hierarchical shrinkage priors.
• To show that projection predictive selection reliably identifies minimal subsets of variables that preserve predictive performance of the full model.
• To provide a practical, reproducible example using the UCI Communities and Crime dataset with cross-validation and test set evaluation.

Proposed method

• Fit a full linear Gaussian regression model using Stan with hierarchical shrinkage priors (HS-$t_\nu$) on regression weights, including weakly informative priors for intercept, global scale $\tau$, and noise variance $\sigma^2$.
• Use half-Student-$t$ priors on local scale parameters $\lambda_i$ and global scale $\tau$, with $\nu=3$ to reduce divergent transitions while maintaining sparsity.
• Apply projection predictive variable selection: sequentially add variables to a submodel by minimizing the Kullback-Leibler divergence between the submodel and the full model's predictive distribution.
• Use forward search heuristic: at each step, select the variable that most reduces KL divergence to the full model's predictive distribution.
• Evaluate performance via 10-fold cross-validation and test set evaluation using mean log predictive density (MLPD) and mean squared error (MSE).
• Use R for post-processing, variable ranking, and performance comparison between submodels and the full model.

Experimental results

Research questions

• RQ1 Can hierarchical shrinkage priors with moderate tail heaviness ($ν=3$) stabilize MCMC sampling in Stan while preserving the sparsity-inducing properties of the horseshoe prior?
• RQ2 How well does projection predictive variable selection recover the predictive performance of the full model using only a subset of variables?
• RQ3 What is the minimal number of variables required to achieve predictive performance indistinguishable from the full model?
• RQ4 How do cross-validated performance estimates compare to test set performance in variable selection?

Key findings

• Using $\nu=3$ in the HS-$t_\nu$ prior reduced divergent transitions to 0.0% and 0.1% for HS-$t_3$ and HS-$t_3$+, respectively, compared to 3.4% and 5.2% with $\nu=1$, without compromising predictive performance.
• The full model's predictive ability is captured with only about 5 variables, as shown by minimal difference in mean log predictive density (MLPD) and mean squared error (MSE).
• By 20 variables, the submodel's predictive performance is indistinguishable from the full model in practical terms, with MLPD and MSE differences near zero.
• Cross-validated performance closely matched test set performance, validating the reliability of cross-validation for model selection in this framework.
• The forward search heuristic effectively identified the most relevant variables, with consistent results across cross-validation folds and test data.