[Paper Review] Feature Selection and Feature Extraction in Pattern Analysis: A Literature Review

Benyamin Ghojogh, Maria N. Samad+5 more|arXiv (Cornell University)|May 7, 2019

Neural Networks and Applications68 citations

TL;DR

A comprehensive survey of theory, motivation, and method classes for feature selection and feature extraction in pattern analysis, with applications and comparisons.

ABSTRACT

Pattern analysis often requires a pre-processing stage for extracting or selecting features in order to help the classification, prediction, or clustering stage discriminate or represent the data in a better way. The reason for this requirement is that the raw data are complex and difficult to process without extracting or selecting appropriate features beforehand. This paper reviews theory and motivation of different common methods of feature selection and extraction and introduces some of their applications. Some numerical implementations are also shown for these methods. Finally, the methods in feature selection and extraction are compared.

Motivation & Objective

• Define the role of pre-processing in pattern analysis to improve discrimination or representation.
• Categorize and explain feature selection and feature extraction methods and their criteria.
• Survey common filter and wrapper feature selection methods and their practical considerations.
• Survey dimensionality reduction and feature extraction techniques, including linear and nonlinear approaches, with discussions of applications.

Proposed method

• Introduce the problem setting and formal notation for features, samples, and targets.
• Describe feature selection as reducing dimensionality by selecting a subset of original features and discuss filter and wrapper paradigms.
• Detail major filter methods: correlation criteria, mutual information, chi-square, Markov blanket, consistency-based filters, fast correlation-based filter, interact, mRMR.
• Outline wrapper methods: sequential feature selection (SFS/SBS, SFFS/SFBS) and metaheuristics (PSO, GA).
• Discuss feature extraction as learning a new representation (p < d), including PCA (and dual PCA), kernel PCA, MDS, Isomap, and LLE, with notes on supervised vs unsupervised and linear vs nonlinear.

Experimental results

Research questions

• RQ1 What are the main criteria and trade-offs used to select informative features?
• RQ2 How do filter and wrapper approaches compare in terms of performance and computational complexity?
• RQ3 What are the common dimensionality-reduction and feature-extraction techniques, and when should each be used?
• RQ4 How do nonlinear manifold methods (Isomap, LLE, kernel PCA) relate to linear methods in practice?

Key findings

• The literature divides feature selection into filter and wrapper methods with various criteria (relevance and redundancy).
• Several explicit criteria are surveyed, including correlation, mutual information, chi-square, and Markov blanket-based approaches.
• Wrapper methods couple feature search with model evaluation, including sequential and metaheuristic strategies.
• Feature extraction reduces to learning a new lower-dimensional representation, with PCA, dual PCA, kernel PCA, MDS, Isomap, and LLE discussed as core techniques.
• Nonlinear manifold methods (Isomap, LLE, kernel PCA) address limitations of linear methods in capturing data geometry.
• The paper connects different dimensionality-reduction approaches under a unified framing and discusses out-of-sample considerations.