[Paper Review] Performance Assessment of High-dimensional Variable Identification

Yanjia Yu, Yi Yang+1 more|arXiv (Cornell University)|Apr 28, 2017

Gene expression and cancer classification13 references3 citations

TL;DR

This paper proposes a data-driven method, PAVI (Performance Assessment of Variable Identification), to estimate F- and G-measures for high-dimensional variable selection when the true model is unknown. By combining candidate models with adaptive weighting, PAVI provides uniformly consistent estimates of F- and G-measures, enabling reliable, data-based evaluation of variable selection performance in regression and classification settings, with strong finite-sample performance and real-world validation on gene expression data.

ABSTRACT

Since model selection is ubiquitous in data analysis, reproducibility of statistical results demands a serious evaluation of reliability of the employed model selection method, no matter what label it may have in terms of good properties. Instability measures have been proposed for evaluating model selection uncertainty. However, low instability does not necessarily indicate that the selected model is trustworthy, since low instability can also arise when a certain method tends to select an overly parsimonious model. F- and G-measures have become increasingly popular for assessing variable selection performance in theoretical studies and simulation results. However, they are not computable in practice. In this work, we propose an estimation method for F- and G-measures and prove their desirable properties of uniform consistency. This gives the data analyst a valuable tool to compare different variable selection methods based on the data at hand. Extensive simulations are conducted to show the very good finite sample performance of our approach. We further demonstrate the application of our methods using several micro-array gene expression data sets, with intriguing findings.

Motivation & Objective

• To address the lack of computable, data-driven performance measures for variable selection in high-dimensional settings where the true model is unknown.
• To overcome the limitations of instability measures, which can be low even when a method selects an overly parsimonious model.
• To provide a practical, consistent estimation of F- and G-measures—harmonic and geometric means of precision and recall—for evaluating overall variable selection accuracy.
• To enable direct comparison of different variable selection methods based on their estimated F- and G-measures using real data.
• To validate the method’s reliability through extensive simulations and real-world analysis of microarray gene expression datasets.

Proposed method

• Proposes a model combination approach using candidate models from penalized methods (e.g., Lasso, SCAD, MCP, adaptive Lasso) to estimate F- and G-measures.
• Employs two weighting schemes: adaptive regression by mixing (Yang, 2001) and information criterion-based weighting (e.g., AIC, BIC) for model combination.
• Derives uniformly consistent estimators for F- and G-measures under weak consistency of the weighting scheme, ensuring reliable estimation across model sets.
• Applies the method to both regression and classification problems, with specific implementation details for logistic regression and SVM in the real data analysis.
• Uses resampling and cross-validation to assess finite-sample performance and validate estimator stability.
• Integrates multiple perspectives (F/G-estimates, AIC/BIC, deviance) to cross-validate model reliability in real data applications.

Experimental results

Research questions

• RQ1 Can F- and G-measures be consistently estimated in high-dimensional settings when the true model is unknown?
• RQ2 How does the proposed PAVI method compare to traditional instability measures in assessing the reliability of variable selection outcomes?
• RQ3 To what extent do estimated F- and G-measures reflect the true performance of variable selection methods on real data?
• RQ4 Can the PAVI framework detect and flag unreliable variable selection results, such as those with near-zero F- and G-measures?
• RQ5 How do different weighting schemes (adaptive mixing vs. information criteria) affect the consistency and accuracy of F- and G-estimates?

Key findings

• The proposed PAVI method produces uniformly consistent estimates of F- and G-measures under weakly consistent weighting, providing theoretical grounding for practical use.
• In simulations, the estimated F- and G-measures accurately reflect the true performance of variable selection methods, even in high-dimensional, low-sample-size settings.
• On three real microarray datasets (Colon, Leukemia, Prostate), models with near-zero estimated F- and G-measures were consistently associated with high AIC, BIC, and deviance values, indicating poor fit and unreliable selection.
• The Lasso and SCAD methods on the Colon dataset yielded near-zero F- and G-estimates (e.g., F ≈ 0.0), which were corroborated by high AIC (26.0) and BIC (53.6) values, suggesting poor model reliability.
• The PAVI approach successfully flagged models with low performance across multiple evaluation perspectives, including model fit (AIC/BIC) and classification performance (deviance), reinforcing its diagnostic power.
• The method demonstrates strong utility in real-world applications, enabling data analysts to assess the reliability and reproducibility of variable selection outcomes without access to the true model.