[Paper Review] Feature Selection Using Classifier in High Dimensional Data

Vijendra Singh, Shivani Pathak|arXiv (Cornell University)|Jan 5, 2014

Gene expression and cancer classification6 references3 citations

TL;DR

This paper proposes a hybrid feature selection approach combining filter and wrapper methods using QDA and LDA classifiers to reduce high-dimensional data while maintaining classification performance. It demonstrates that the filter method, based on univariate criteria, achieves lower misclassification error rates than the wrapper approach with sequential feature selection, particularly on bioinformatics datasets.

ABSTRACT

Feature selection is frequently used as a pre-processing step to machine learning. It is a process of choosing a subset of original features so that the feature space is optimally reduced according to a certain evaluation criterion. The central objective of this paper is to reduce the dimension of the data by finding a small set of important features which can give good classification performance. We have applied filter and wrapper approach with different classifiers QDA and LDA respectively. A widely-used filter method is used for bioinformatics data i.e. a univariate criterion separately on each feature, assuming that there is no interaction between features and then applied Sequential Feature Selection method. Experimental results show that filter approach gives better performance in respect of Misclassification Error Rate.

Motivation & Objective

• To address the challenge of high-dimensional data in machine learning by reducing feature space while preserving classification accuracy.
• To compare the effectiveness of filter and wrapper feature selection methods in high-dimensional settings.
• To evaluate the performance of QDA and LDA classifiers in guiding feature selection for dimensionality reduction.
• To determine whether univariate filter methods outperform sequential wrapper methods in terms of misclassification error.
• To assess the utility of feature selection in improving classification performance on bioinformatics datasets.

Proposed method

• The filter method applies a univariate criterion independently to each feature, assuming no feature interactions, to rank and select features based on individual relevance.
• The wrapper method uses Sequential Forward Selection (SFS) with QDA and LDA as the evaluation criterion to iteratively select feature subsets.
• Feature selection is performed using LDA and QDA classifiers as the underlying models to assess subset performance.
• The filter approach evaluates features based on a statistical criterion without training a classifier, while the wrapper approach uses classifier accuracy as the evaluation metric.
• The study compares both methods on bioinformatics datasets to assess their impact on misclassification error rate.
• Performance is evaluated using misclassification error rate as the primary metric across multiple datasets.

Experimental results

Research questions

• RQ1 Does the filter method outperform the wrapper method in terms of misclassification error rate on high-dimensional bioinformatics data?
• RQ2 How effective is a univariate filter criterion in selecting relevant features when feature interactions are ignored?
• RQ3 Can QDA and LDA classifiers serve as effective evaluators in wrapper-based feature selection for high-dimensional datasets?
• RQ4 To what extent does dimensionality reduction via feature selection improve classification performance?
• RQ5 Is the assumption of feature independence in filter methods justified in practice for high-dimensional data?

Key findings

• The filter method achieved lower misclassification error rates compared to the wrapper method across the evaluated datasets.
• The univariate filter criterion effectively identified relevant features without considering feature interactions, yielding strong performance.
• Sequential Feature Selection with QDA and LDA as the wrapper classifier did not outperform the simpler filter approach.
• The use of LDA and QDA as classifiers in the wrapper framework did not significantly improve classification accuracy over the filter method.
• The results suggest that the filter approach is more efficient and effective for high-dimensional data, particularly in bioinformatics contexts.
• The study confirms that feature selection significantly reduces misclassification error, with the filter method providing the best trade-off between performance and computational cost.