[Paper Review] On Supervised Feature Selection from High Dimensional Feature Spaces

Yijing Yang, Wei Wang+2 more|arXiv (Cornell University)|Mar 22, 2022

Face and Expression Recognition4 citations

TL;DR

This paper proposes Discriminant Feature Test (DFT) and Relevant Feature Test (RFT) for supervised feature selection in high-dimensional spaces, leveraging weighted entropy (DFT) and weighted mean squared error (RFT) to rank features. Experiments on MNIST and Fashion-MNIST with LeNet-5 features show DFT and RFT achieve significantly lower-dimensional, robust feature subspaces while maintaining near-optimal classification and regression performance.

ABSTRACT

The application of machine learning to image and video data often yields a high dimensional feature space. Effective feature selection techniques identify a discriminant feature subspace that lowers computational and modeling costs with little performance degradation. A novel supervised feature selection methodology is proposed for machine learning decisions in this work. The resulting tests are called the discriminant feature test (DFT) and the relevant feature test (RFT) for the classification and regression problems, respectively. The DFT and RFT procedures are described in detail. Furthermore, we compare the effectiveness of DFT and RFT with several classic feature selection methods. To this end, we use deep features obtained by LeNet-5 for MNIST and Fashion-MNIST datasets as illustrative examples. Other datasets with handcrafted and gene expressions features are also included for performance evaluation. It is shown by experimental results that DFT and RFT can select a lower dimensional feature subspace distinctly and robustly while maintaining high decision performance.

Motivation & Objective

• To address the curse of dimensionality in machine learning by identifying discriminant and relevant feature subspaces from high-dimensional inputs.
• To develop a computationally efficient, filter-based feature selection method that outperforms classic techniques in selecting low-dimensional, high-performance feature subsets.
• To evaluate the robustness and effectiveness of the proposed method on deep features from image datasets, including noisy and clean data.
• To demonstrate that DFT and RFT produce distinct elbow points in feature importance curves, enabling reliable dimensionality reduction.

Proposed method

• DFT uses weighted entropy to measure discriminant power of features for classification, ranking features based on their ability to separate classes.
• RFT uses weighted mean squared error (MSE) to assess relevance of features to regression targets, enabling feature ranking for regression tasks.
• The methods are filter-based, operating independently of the learning algorithm, and score features without iterative model training.
• Feature selection is performed by identifying an 'elbow' in the DFT/RFT score curve, indicating an optimal trade-off between dimensionality and performance.
• The approach is applied to deep features from LeNet-5 on MNIST and Fashion-MNIST, with ablation on PCA pre-processing and noise robustness.
• Performance is benchmarked against variance, correlation, and XGBoost feature importance (FI) using classification accuracy and regression MSE.

Experimental results

Research questions

• RQ1 Can DFT and RFT identify a low-dimensional feature subspace that maintains high classification performance in high-dimensional image data?
• RQ2 How does the elbow point in DFT/RFT score curves compare to those of variance, correlation, and XGBoost feature importance in terms of reliability and distinctiveness?
• RQ3 To what extent are DFT and RFT robust to noisy input data in feature selection for image classification and regression?
• RQ4 How does PCA pre-processing affect the distribution and selection of features by DFT and RFT?

Key findings

• DFT and RFT produce a more distinct and reliable elbow point in feature importance curves compared to variance, correlation, and XGBoost feature importance, enabling clearer dimensionality reduction decisions.
• On MNIST and Fashion-MNIST, RFT achieves regression MSE within 20-30% of the full feature set when using only 25-37.5% of the features, demonstrating high efficiency.
• With 100-D features, RFT achieves a regression MSE of 0.91 on MNIST and 1.48 on Fashion-MNIST at the late elbow point, outperforming variance and correlation-based methods.
• The selected feature subsets using DFT are more concentrated in the first 20–50 principal components when PCA is applied, indicating better alignment with dominant variance directions.
• DFT and RFT maintain high performance even under noisy input, showing robustness to data corruption in feature selection.
• The proposed methods achieve significantly lower-dimensional feature subspaces than baseline methods while preserving near-optimal performance, confirming their effectiveness in high-dimensional settings.