[Paper Review] When is best subset selection the "best"?

Jianqing Fan, Yongyi Guo+1 more|arXiv (Cornell University)|Jul 3, 2020

Statistical Methods and Inference27 references4 citations

TL;DR

This paper identifies conditions under which best subset selection (BSS) achieves optimal model recovery—specifically, when it yields the oracle estimator or achieves the sure screening property. It shows that iterative hard thresholding (IHT) can efficiently find a near-optimal sparse model, and a second BSS step within this set recovers the true model, outperforming LASSO, SCAD, and SIS in simulation and real data with lower false discovery and higher true positive rates.

ABSTRACT

Best subset selection (BSS) is fundamental in statistics and machine learning. Despite the intensive studies of it, the fundamental question of when BSS is truly the best, namely yielding the oracle estimator, remains partially answered. In this paper, we address this important issue by giving a weak sufficient condition and a strong necessary condition for BSS to exactly recover the true model. We also give a weak sufficient condition for BSS to achieve the sure screening property. On the optimization aspect, we find that the exact combinatorial minimizer for BSS is unnecessary: all the established statistical properties for the best subset carry over to any sparse model whose residual sum of squares is close enough to that of the best subset. In particular, we show that an iterative hard thresholding (IHT) algorithm can find a sparse subset with the sure screening property within logarithmic steps; another round of BSS within this set can recover the true model. The simulation studies and real data examples show that IHT yields lower false discovery rates and higher true positive rates than the competing approaches including LASSO, SCAD and SIS.

Motivation & Objective

• To determine when best subset selection (BSS) exactly recovers the true model, i.e., achieves the oracle property.
• To establish weak sufficient and strong necessary conditions for BSS to yield the true model.
• To identify conditions under which BSS achieves the sure screening property.
• To show that the exact combinatorial minimizer in BSS is not required for statistical optimality.
• To propose and validate an efficient two-step method combining IHT and BSS for improved model selection performance.

Proposed method

• Derive a weak sufficient condition and a strong necessary condition for BSS to exactly recover the true model.
• Introduce the concept that any sparse model with a residual sum of squares (RSS) close to the best subset's RSS inherits the same statistical properties.
• Propose an iterative hard thresholding (IHT) algorithm that converges to a sparse model with the sure screening property in logarithmic steps.
• Apply a second BSS step on the IHT-selected model to recover the true model.
• Use simulation studies and real data examples to compare IHT-BSS with LASSO, SCAD, and SIS in terms of false discovery and true positive rates.
• Demonstrate that the statistical properties of BSS are preserved as long as the RSS is sufficiently close to the optimal value.

Experimental results

Research questions

• RQ1 Under what conditions does best subset selection exactly recover the true model?
• RQ2 What conditions ensure that best subset selection achieves the sure screening property?
• RQ3 Is the exact combinatorial minimizer of BSS necessary for optimal statistical performance?
• RQ4 Can iterative hard thresholding (IHT) efficiently identify a model subset that enables subsequent BSS to recover the true model?
• RQ5 How does the IHT-BSS two-step approach compare to LASSO, SCAD, and SIS in terms of false discovery and true positive rates?

Key findings

• Best subset selection achieves the oracle estimator when a weak sufficient condition on the design matrix and signal strength is satisfied.
• A strong necessary condition for exact model recovery is derived, providing a theoretical boundary for BSS performance.
• A weak sufficient condition for the sure screening property is established, ensuring the true model is included in the selected subset.
• The exact combinatorial minimizer in BSS is not required; any sparse model with RSS close to the best subset retains the same statistical properties.
• Iterative hard thresholding (IHT) finds a sparse model with the sure screening property in logarithmic number of steps, enabling efficient model search.
• The IHT-BSS two-step method yields lower false discovery rates and higher true positive rates than LASSO, SCAD, and SIS in both simulations and real data applications.