[Paper Review] Fully Bayesian Classification with Heavy-tailed Priors for Selection in High-dimensional Features with Grouping Structure

Lai Jiang, Longhai Li+1 more|arXiv (Cornell University)|Jul 1, 2016

Statistical Methods and Inference44 references3 citations

TL;DR

This paper proposes FBRHT, a fully Bayesian Robit regression method with heavy-tailed priors for high-dimensional feature selection that automatically identifies sparse, predictive feature subsets while handling grouped, correlated features without requiring pre-specified group structures. It outperforms LASSO, group LASSO, and other methods by achieving comparable or better predictive performance with significantly sparser feature sets, especially in real-world gene expression data.

ABSTRACT

Feature selection is demanded in many modern scientific research problems that use high-dimensional data. A typical example is to find the most useful genes that are related to a certain disease (eg, cancer) from high-dimensional gene expressions. The expressions of genes have grouping structures, for example, a group of co-regulated genes that have similar biological functions tend to have similar expressions. Many statistical methods have been proposed to take the grouping structure into consideration in feature selection, including group LASSO, supervised group LASSO, and regression on group representatives. In this paper, we propose a fully Bayesian Robit regression method with heavy-tailed (sparsity) priors (shortened by FBRHT) for selecting features with grouping structure. The main features of FBRHT include that it discards more aggressively unrelated features than LASSO, and it can make feature selection within groups automatically without a pre-specified grouping structure. In this paper, we use simulated and real datasets to demonstrate that the predictive power of the sparse feature subsets selected by FBRHT are comparable with other much larger feature subsets selected by LASSO, group LASSO, supervised group LASSO, penalized logistic regression and random forest, and that the succinct feature subsets selected by FBRHT have significantly better predictive power than the feature subsets of the same size taken from the top features selected by the aforementioned methods.

Motivation & Objective

• To address the challenge of selecting sparse, predictive feature subsets in high-dimensional data with inherent grouping or correlation structures.
• To overcome the limitations of non-convex penalized methods, such as unstable optimization and poor scalability, by using a fully Bayesian framework with MCMC sampling.
• To enable automatic within-group feature selection without requiring pre-specified groupings, leveraging the correlation structure naturally.
• To improve predictive performance over existing methods like LASSO, group LASSO, and random forest while achieving significantly sparser feature subsets.
• To develop a robust, stable method that combines the sparsity-inducing properties of heavy-tailed priors with the interpretability of Bayesian inference.

Proposed method

• FBRHT employs a fully Bayesian Robit regression model with heavy-tailed (e.g., Laplace, horseshoe-like) priors to induce sparsity in coefficient estimates.
• It uses Markov Chain Monte Carlo (MCMC) sampling to explore the posterior distribution of coefficients, enabling uncertainty quantification and robust feature selection.
• The method automatically performs within-group selection by leveraging correlation among features, even without pre-defined group structures.
• Feature selection is performed by identifying modes in the MCMC output, with a thresholding rule (e.g., relative coefficient magnitude > 0.1) to filter out noise.
• A two-stage approach is used: first, MCMC is run on the full feature set to identify promising groups; second, MCMC is re-run on a reduced set of candidate features for refinement.
• The method is extended to handle various heavy-tailed priors and can be adapted to other models such as linear and graphical models.

Experimental results

Research questions

• RQ1 Can a fully Bayesian approach with heavy-tailed priors outperform frequentist penalized methods in selecting sparse, predictive feature subsets in high-dimensional data with grouping structure?
• RQ2 Does FBRHT automatically perform within-group feature selection without requiring pre-specified groupings?
• RQ3 Can FBRHT achieve comparable or better predictive performance than LASSO, group LASSO, and random forest while using significantly fewer features?
• RQ4 How does the predictive performance of FBRHT compare to other methods when feature subsets are of equal size?
• RQ5 Can MCMC-based Bayesian inference with heavy-tailed priors stabilize the selection process and reduce the instability common in non-convex optimization?

Key findings

• FBRHT achieves predictive performance comparable to much larger feature subsets selected by LASSO, group LASSO, supervised group LASSO, random forest, and penalized logistic regression, even with far fewer features.
• On the Leukemia dataset, FBRHT selected an average of 1.00 gene in the top subset (FBRHTtop), while LASSO selected 26.43 features on average, yet FBRHT maintained an AUC of 1.00.
• On the Breast Cancer dataset, FBRHTtop used only 1.00 gene, FBRHTopt used 1.95, and both achieved AUC of 1.00, while LASSO used 26.43 genes with AUC 1.00 and ER × 121 of 1.
• The sparse feature subsets selected by FBRHT had significantly better predictive power than feature subsets of the same size taken from the top features selected by LASSO, group LASSO, and other methods.
• FBRHT demonstrated superior sparsity without sacrificing predictive accuracy, with the most sparse subsets (1–2 genes) achieving near-perfect AUC values of 1.00 on both real datasets.
• The method effectively handles within-group selection and noise reduction, discarding unrelated features more aggressively than LASSO while preserving signal.