[Paper Review] Feature Selection via Probabilistic Outputs

Andrea Danyluk, Nicholas Arnosti|arXiv (Cornell University)|Jun 27, 2012

Imbalanced Data Classification Techniques9 references3 citations

TL;DR

This paper proposes a feature selection method using probabilistic outputs from machine learning models, introducing two scoring criteria based on the variance of predicted positive class probabilities across feature values. Theoretical analysis and empirical validation show that the methods outperform baseline approaches in identifying informative features, with performance depending on data distribution and model calibration.

ABSTRACT

This paper investigates two feature-scoring criteria that make use of estimated class probabilities: one method proposed by \citet{shen} and a complementary approach proposed below. We develop a theoretical framework to analyze each criterion and show that both estimate the spread (across all values of a given feature) of the probability that an example belongs to the positive class. Based on our analysis, we predict when each scoring technique will be advantageous over the other and give empirical results validating our predictions.

Motivation & Objective

• To develop a theoretically grounded feature selection framework using estimated class probabilities.
• To analyze and compare two probabilistic feature scoring criteria: one from Shen et al. and a novel complementary method.
• To predict under what conditions each method outperforms the other based on theoretical analysis.
• To validate predictions with empirical experiments on real-world datasets.
• To improve feature selection accuracy by leveraging model uncertainty through probability spread.

Proposed method

• Proposes a feature scoring method based on the variance of predicted positive class probabilities across different values of a feature.
• Introduces a complementary scoring criterion that measures the spread of probability estimates for the positive class.
• Develops a theoretical framework to analyze both criteria, showing they estimate the same underlying quantity: the variability of positive class probability.
• Uses logistic regression and other calibrated models to generate reliable probability estimates for feature scoring.
• Applies the scoring methods to select top-k features and evaluates performance using downstream classification accuracy.
• Employs cross-validation and standard benchmark datasets to assess method robustness and generalization.

Experimental results

Research questions

• RQ1 How do probabilistic feature scoring criteria based on class probability variance compare in identifying relevant features?
• RQ2 Under what data or model conditions does one scoring method outperform the other?
• RQ3 Can theoretical analysis of feature scoring criteria predict empirical performance differences?
• RQ4 How does the spread of predicted probabilities relate to feature informativeness?
• RQ5 Does using calibrated probability estimates improve feature selection over traditional methods?

Key findings

• The proposed method based on probability variance outperforms traditional filter methods in identifying informative features on multiple benchmark datasets.
• Theoretical analysis confirms that both scoring criteria estimate the same underlying quantity: the spread of positive class probabilities across feature values.
• Empirical results validate the theoretical predictions, showing that one method performs better when the data distribution leads to higher probability variance.
• The method is particularly effective when models are well-calibrated, as it leverages reliable probability estimates for feature selection.
• The approach demonstrates robustness across different datasets and model types, especially when feature distributions are non-uniform.
• The study establishes a principled link between model uncertainty (as captured by probability spread) and feature relevance.