[Paper Review] Extended Comparisons of Best Subset Selection, Forward Stepwise Selection, and the Lasso

Trevor Hastie, Robert Tibshirani+1 more|arXiv (Cornell University)|Jul 27, 2017

Statistical Methods and Inference20 references163 citations

TL;DR

Extends simulations comparing best subset selection, forward stepwise, and the lasso; finds no uniform dominance among methods and highlights the relaxed lasso as a strong performer, with subset selection being most accurate at high SNR and lasso performing better at low SNR.

ABSTRACT

In exciting new work, Bertsimas et al. (2016) showed that the classical best subset selection problem in regression modeling can be formulated as a mixed integer optimization (MIO) problem. Using recent advances in MIO algorithms, they demonstrated that best subset selection can now be solved at much larger problem sizes that what was thought possible in the statistics community. They presented empirical comparisons of best subset selection with other popular variable selection procedures, in particular, the lasso and forward stepwise selection. Surprisingly (to us), their simulations suggested that best subset selection consistently outperformed both methods in terms of prediction accuracy. Here we present an expanded set of simulations to shed more light on these comparisons. The summary is roughly as follows: (a) neither best subset selection nor the lasso uniformly dominate the other, with best subset selection generally performing better in high signal-to-noise (SNR) ratio regimes, and the lasso better in low SNR regimes; (b) best subset selection and forward stepwise perform quite similarly throughout; (c) the relaxed lasso (actually, a simplified version of the original relaxed estimator defined in Meinshausen, 2007) is the overall winner, performing just about as well as the lasso in low SNR scenarios, and as well as best subset selection in high SNR scenarios.

Motivation & Objective

• Assess when best subset selection, forward stepwise, and the lasso outperform each other across varied SNR, predictor correlations, and sparsity patterns.
• Quantify predictive accuracy, sparsity, and computational costs of the three methods under broader simulation settings.
• Introduce and evaluate a relaxed lasso variant and compare its performance to standard lasso and subset-based methods.

Proposed method

• Formulates best subset selection as a mixed integer optimization (MIO) problem to leverage advanced solvers (e.g., Gurobi).
• Conducts extensive simulations across varying n, p, sparsity s, SNR ν, and predictor correlation ρ, comparing lasso, relaxed lasso, forward stepwise, and best subset selection.
• Uses external validation to tune hyperparameters for all methods and reports predictive performance metrics.
• Implements an R package (bestsubset) providing access to the MIO-based best subset solver and steps for reproducibility.
• Defines a relaxed lasso estimator hat-beta^{relax}(λ, γ) = γ hat-beta^{lasso}(λ) + (1−γ) hat-beta^{LS}}(λ), analyzing its degrees of freedom and aggressiveness relative to other methods.
• Evaluates computational costs with time budgets for each method, including a 3-minute per k limit for the MIO solver in the subset path.

Experimental results

Research questions

• RQ1 How do best subset selection, forward stepwise selection, and the lasso compare in prediction accuracy across different SNR regimes and predictor correlations?
• RQ2 Does the relaxed lasso provide a robust middle ground between shrinkage and sparsity, and how does its complexity compare to other methods?
• RQ3 What are the computational trade-offs of using MIO-based best subset selection versus fast convex/greedy methods in practical settings?
• RQ4 Under which problem settings do the nonconvex best subset and forward stepwise methods outperform the lasso, and where does the lasso dominate?
• RQ5 How realistic are the SNR levels used in prior work, and how do results change with broader SNR ranges?

Key findings

• Best subset selection and the lasso do not uniformly dominate each other; best subset tends to win at high SNR, while the lasso excels at low SNR.
• Forward stepwise and best subset selection perform quite similarly across many settings.
• The relaxed lasso often performs best overall, matching lasso in low-SNR regimes and approaching best subset in high-SNR regimes.
• The aggressive nature (higher degrees of freedom) of best subset and forward stepwise can lead to higher variance at low k, whereas the lasso provides shrinkage that stabilizes predictions.
• A realistic assessment of SNR suggests typical PVEs are modest in many practical scenarios, influencing method choice toward less aggressive shrinkage.