[Paper Review] A review and recommendations on variable selection methods in regression models for binary data

Souvik Bag, Kapil Gupta+1 more|arXiv (Cornell University)|Jan 16, 2022

Gene expression and cancer classification4 citations

TL;DR

This paper provides a comprehensive review and empirical comparison of 16 frequentist variable selection methods in logistic regression for binary data, categorized into test-based, penalty-based, screening-based, and tree-based approaches. It evaluates their performance across low, moderate, and high-dimensional settings with varying correlation structures, concluding that SCAD and MCP excel in high-dimensional sparse models, while Boruta and varSelRF are optimal for gene expression data due to superior prediction accuracy and variable selection consistency.

ABSTRACT

The selection of essential variables in logistic regression is vital because of its extensive use in medical studies, finance, economics and related fields. In this paper, we explore four main typologies (test-based, penalty-based, screening-based, and tree-based) of frequentist variable selection methods in logistic regression setup. Primary objective of this work is to give a comprehensive overview of the existing literature for practitioners. Underlying assumptions and theory, along with the specifics of their implementations, are detailed as well. Next, we conduct a thorough simulation study to explore the performances of fifteen different methods in terms of variable selection, estimation of coefficients, prediction accuracy as well as time complexity under various settings. We take low, moderate and high dimensional setups and consider different correlation structures for the covariates. A real-life application, using a high-dimensional gene expression data, is also included in this study to further understand the efficacy and consistency of the methods. Finally, based on our findings in the simulated data and in the real data, we provide recommendations for practitioners on the choice of variable selection methods under various contexts.

Motivation & Objective

• To provide a systematic review of classical frequentist variable selection methods in logistic regression for binary outcomes.
• To evaluate the performance of 16 variable selection methods across diverse simulation settings with varying dimensionality and correlation structures.
• To assess both inferential accuracy (variable selection, coefficient estimation) and predictive performance (Brier score, prediction accuracy) of the methods.
• To offer evidence-based recommendations for practitioners based on simulation results and a real high-dimensional gene expression dataset.
• To identify limitations of existing methods and suggest future research directions, including group structures and ultra-high-dimensional settings.

Proposed method

• Classifies variable selection methods into four typologies: test-based (e.g., SIVS, VSURF), penalty-based (e.g., LASSO, SCAD, MCP, ElasticNet), screening-based (e.g., SIS, null screen), and tree-based (e.g., Boruta, varSelRF).
• Conducts a simulation study across 18 distinct setups, varying sample size (n), number of predictors (m), correlation structure (uncorrelated, equi-correlated, block-correlated), and sparsity (number of true predictors).
• Employs standard performance metrics: variable selection accuracy (e.g., true positive rate, false discovery rate), coefficient estimation bias, prediction accuracy (Brier score), and computational time.
• Validates findings using a real high-dimensional gene expression dataset from statistical genetics, focusing on identifying important genes for disease classification.
• Uses parallelized implementations for computationally intensive methods (e.g., SIVS, PIMP) and evaluates time complexity across different n and m values.
• Compares both inferential and predictive performance, emphasizing the trade-off between model sparsity, accuracy, and computational feasibility.

Experimental results

Research questions

• RQ1 How do different variable selection methods perform in terms of variable selection accuracy, coefficient estimation, and prediction under low, moderate, and high-dimensional settings?
• RQ2 How does correlation structure among predictors affect the performance of each method, particularly in high-dimensional scenarios?
• RQ3 Which methods offer the best balance between model sparsity, prediction accuracy (Brier score), and computational efficiency in high-dimensional data?
• RQ4 How do tree-based methods like Boruta and varSelRF compare to penalty-based and screening-based methods in real-world gene expression data with high dimensionality?
• RQ5 In what contexts should practitioners prefer methods like ElasticNet or NTA over LASSO, especially when an exhaustive list of potentially relevant variables is required?

Key findings

• In low-dimensional settings (n ≫ m), SIVS, Best Subset, and MCP outperform other methods, while LASSO and ElasticNet tend to select more irrelevant variables.
• In moderate-dimensional settings (n ≈ m), Best Subset and MCP maintain strong performance, and SCAD offers fast, accurate out-of-sample predictions with slightly higher false positive rates.
• In high-dimensional settings (m > n), screening-based methods select very few variables but maintain acceptable prediction accuracy, making them suitable for sparse model construction.
• For high-dimensional sparse models, SCAD and MCP achieve the lowest Brier scores and select the most appropriate number of true predictors, outperforming LASSO and ElasticNet.
• Boruta and varSelRF demonstrate the highest prediction accuracy and select a consistent number of important genes in real gene expression data, making them ideal for biological applications.
• SIVS, VSURF, and PIMP are computationally expensive and should be avoided in ultra-high-dimensional settings despite their theoretical appeal.