[Paper Review] Stability Approach to Regularization Selection (StARS) for High Dimensional Graphical Models

Han Liu, Kathryn Roeder+1 more|arXiv (Cornell University)|Jun 16, 2010

Gene expression and cancer classification172 citations

TL;DR

This paper proposes StARS (Stability Approach to Regularization Selection), a novel method for selecting the regularization parameter in high-dimensional graphical models by balancing sparsity and stability across random subsamples. It outperforms traditional methods like K-fold CV, AIC, and BIC in both synthetic and real microarray data, achieving high partial sparsistency and producing more accurate, interpretable graphs even in high-dimensional settings.

ABSTRACT

A challenging problem in estimating high-dimensional graphical models is to choose the regularization parameter in a data-dependent way. The standard techniques include $K$-fold cross-validation ($K$-CV), Akaike information criterion (AIC), and Bayesian information criterion (BIC). Though these methods work well for low-dimensional problems, they are not suitable in high dimensional settings. In this paper, we present StARS: a new stability-based method for choosing the regularization parameter in high dimensional inference for undirected graphs. The method has a clear interpretation: we use the least amount of regularization that simultaneously makes a graph sparse and replicable under random sampling. This interpretation requires essentially no conditions. Under mild conditions, we show that StARS is partially sparsistent in terms of graph estimation: i.e. with high probability, all the true edges will be included in the selected model even when the graph size diverges with the sample size. Empirically, the performance of StARS is compared with the state-of-the-art model selection procedures, including $K$-CV, AIC, and BIC, on both synthetic data and a real microarray dataset. StARS outperforms all these competing procedures.

Motivation & Objective

• To address the critical challenge of regularization parameter selection in high-dimensional undirected graphical models, where standard methods like K-CV, AIC, and BIC fail due to high dimensionality.
• To develop a data-driven method that selects the minimal regularization ensuring both graph sparsity and replicability across random subsamples.
• To provide a theoretically grounded yet interpretable approach that ensures inclusion of true edges even as graph size diverges with sample size.
• To empirically demonstrate superior performance over existing model selection procedures on synthetic and real-world high-dimensional data.

Proposed method

• StARS uses overlapping random subsamples of the data to assess edge stability across different regularization levels.
• It starts with high regularization (sparse, stable graph) and gradually reduces it until a threshold of acceptable variability across subsamples is reached.
• The selected regularization parameter corresponds to the least amount of regularization that maintains stable edge patterns across subsamples.
• The method is applied to the graphical lasso framework, where regularization controls the sparsity of the inverse covariance matrix estimate.
• StARS does not rely on cross-validation or information criteria; instead, it directly optimizes for stability and sparsity in the selected graph.
• The procedure is generalizable to other high-dimensional structure estimation tasks such as regression, clustering, and dimensionality reduction.

Experimental results

Research questions

• RQ1 Can a stability-based approach outperform traditional regularization selection methods like K-CV, AIC, and BIC in high-dimensional graphical models?
• RQ2 Does StARS achieve partial sparsistency, ensuring that all true edges are included with high probability as the graph size grows?
• RQ3 Can StARS effectively balance sparsity and replicability in the selected graph without requiring strong distributional assumptions?
• RQ4 How does StARS perform on real-world high-dimensional data, such as gene expression microarray datasets?

Key findings

• On synthetic data with high-dimensional hub and neighborhood graphs, StARS achieved an F1-score of 0.6274, significantly outperforming K-CV (0.3769) and AIC (0.3951).
• In low-dimensional settings, BIC performed competitively, but in high-dimensional settings, StARS clearly outperformed all competing methods, including BIC and AIC.
• On the microarray dataset (n=294, p=324), StARS produced a sparse, informative graph with cliques and hub genes, while BIC generated a dense graph that obscured meaningful associations.
• The StARS graph was nearly as accurate as the oracle graph (estimated on a subsample), while K-CV, BIC, and AIC graphs were overly dense and less interpretable.
• StARS demonstrated partial sparsistency under mild conditions, meaning it includes all true edges with high probability even as the number of variables increases with sample size.
• The method’s performance is robust and interpretable: it selects the minimal regularization that ensures both sparsity and stability, making it well-suited for scientific discovery in high-dimensional settings.