[Paper Review] Feature Selection Techniques for Classification: A widely applicable code library

Giorgio Roffo|arXiv (Cornell University)|Jul 5, 2016

Machine Learning and Data Classification3 citations

TL;DR

This paper introduces FSLib, a versatile MATLAB library implementing filter, embedded, and wrapper feature selection methods to improve classification performance by reducing dimensionality. By selecting relevant features, FSLib mitigates the curse of dimensionality, accelerates learning, and enhances model interpretability across diverse machine learning applications.

ABSTRACT

Feature Selection Library (FSLib) is a widely applicable MATLAB library for Feature Selection (FS). FS is an essential component of machine learning and data mining which has been studied for many years under many different conditions and in diverse scenarios. These algorithms aim at ranking and selecting a subset of relevant features according to their degrees of relevance, preference, or importance as defined in a specific application. Because feature selection can reduce the amount of features used for training classification models, it alleviates the effect of the curse of dimensionality, speeds up the learning process, improves model's performance, and enhances data understanding. This short report provides an overview of the feature selection algorithms included in the FSLib MATLAB toolbox among filter, embedded, and wrappers methods.

Motivation & Objective

• To develop a widely applicable MATLAB toolbox for feature selection across diverse machine learning and data mining applications.
• To address the curse of dimensionality by identifying and selecting the most relevant features for classification models.
• To improve model performance, training speed, and data interpretability through systematic feature subset selection.
• To provide researchers and practitioners with a unified, accessible implementation of multiple feature selection techniques.

Proposed method

• The FSLib toolbox implements filter methods that evaluate features based on statistical measures independent of learning algorithms.
• Embedded methods are integrated into the learning process, where feature selection is performed as part of model training, such as in L1-regularized models.
• Wrapper methods use a predictive model to evaluate subsets of features through iterative search and performance assessment.
• The library supports various feature ranking and subset selection strategies, enabling flexibility across different data types and application needs.
• Algorithms are designed for compatibility with classification tasks, with configurable parameters for relevance and importance thresholds.
• The toolbox is structured for extensibility, allowing users to integrate new feature selection techniques easily.

Experimental results

Research questions

• RQ1 How can a unified MATLAB library effectively support diverse feature selection techniques across classification tasks?
• RQ2 What impact do different feature selection methods—filter, embedded, and wrapper—have on model performance and training efficiency?
• RQ3 To what extent can feature selection reduce dimensionality while preserving or improving classification accuracy?
• RQ4 How can a modular and extensible library enhance reproducibility and usability in machine learning research?

Key findings

• The FSLib toolbox successfully implements a comprehensive set of feature selection algorithms, including filter, embedded, and wrapper methods, within a single MATLAB environment.
• Feature selection using FSLib reduces the number of input features, thereby alleviating the curse of dimensionality in high-dimensional datasets.
• By selecting relevant features, the toolbox improves training speed and enhances model performance across classification tasks.
• The library supports improved data understanding by highlighting the most informative features based on application-specific relevance criteria.
• The modular design enables researchers to extend and customize feature selection workflows for specific use cases.
• The toolbox provides a practical, accessible solution for both researchers and practitioners seeking to apply feature selection in classification problems.